Results and lessons

All numbers on this page come from the saved result files in ../exams/results/. The tables are regenerated by python3 exams/inject_tables.py, so they cannot drift from the data. How the exams work, and how they could be wrong, is in 05.

The answer in five sentences

1. Yes, a general coding agent inside the Databricks shell can do most Genie-Code-style work: with a strong model it passed essentially every exam.
2. The model matters more than the tooling. The same setup scored 39% with one model and 86% with another.
3. For a weaker model, engineering closes most of the gap: scoped tools, a supervisor loop and written rules took it from 19% to 94%.
4. Databricks' official CLI and skills are good enough to be the foundation, but they hide a Genie shortcut and give no write fence.
5. Most of what we tuned did not matter for a strong model: on three never-tuned tasks, even our very first setup scored 100% with Gemini.

Summary: who scores what

Each cell is the total score over all tasks in the set. - means not run.

Development set (M1-M8), total score

(* = not every task was run)

Held-out set (X1-X3), total score

(* = not every task was run)

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#E8F0FE','primaryTextColor':'#1F2937','primaryBorderColor':'#A8C0E8','lineColor':'#94A3B8','xyChart':{'plotColorPalette':'#A8C0E8, #A8D5B5, #F5C29C'}}}}%%
xychart-beta
    title "Development set, gpt-oss-120b: score by setup"
    x-axis ["V0 bare", "V1 first", "V2 loop", "V3 cells", "V4 skills", "V5 stock", "V6 hybrid"]
    y-axis "Total score (%)" 0 --> 100
    bar [19, 20, 82, 94, 94, 39, 88]

The variant ladder

We changed one family of things at a time. Each step is a fix for a failure we saw in the step before.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#E8F0FE','primaryTextColor':'#1F2937','primaryBorderColor':'#A8C0E8','lineColor':'#94A3B8'}}}%%
flowchart TB
    V0["V0 bare shell: 19%<br/>stops after 0 to 3 tool calls,<br/>types SQL into the chat instead of running it"]:::pink
    V1["V1 MCP server and rules: 20%<br/>still stops early, 6 of 8 answers empty"]:::orange
    V2["V2 supervisor, run-and-verify tools,<br/>scoped writes: 82%<br/>about 16 tool calls per task"]:::yellow
    V3["V3 cells writer, no local file tools,<br/>report template: 94%<br/>no empty answers"]:::green
    V4["V4 native list params, skills: 94%<br/>same total as V3, different weak spots"]:::teal
    V0 -->|"Fix: supervisor, working agreement,<br/>tools that run and report real errors"| V2
    V1 --> V2
    V2 -->|"Fix: remove misleading tools,<br/>force a structured final report"| V3
    V3 -->|"Fix: accept real lists,<br/>give exact formats on demand"| V4
    classDef pink fill:#FDE7EF,stroke:#F2B5CB,color:#1F2937
    classDef orange fill:#FFEDE0,stroke:#F5C29C,color:#1F2937
    classDef yellow fill:#FFF8E1,stroke:#F2D98D,color:#1F2937
    classDef green fill:#E6F4EA,stroke:#A8D5B5,color:#1F2937
    classDef teal fill:#E0F5F3,stroke:#9AD5CE,color:#1F2937

What each step bought (gpt-oss-120b, development set, one run):

The biggest single jump (V1 to V2, +62 points) came from making the agent keep going and prove its work, not from any clever Databricks knowledge. V4's "improvements" did not raise the total at all.

Full tables

Development set (M1-M8): used while tuning V2-V4

Held-out set (X1-X3): never used for tuning

Token counts show the cost shape: a strong model reads and writes a lot more (5 to 9 million tokens per eight tasks) than the weak one (2 to 3 million), because it checks its work more. We have not turned tokens into money; see 07.

Model versus tooling

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#E8F0FE','primaryTextColor':'#1F2937','primaryBorderColor':'#A8C0E8','lineColor':'#94A3B8','xyChart':{'plotColorPalette':'#A8C0E8, #F5C29C'}}}}%%
xychart-beta
    title "Development set: the same three setups with two models"
    x-axis ["V4 custom MCP", "V5 stock CLI + skills", "V6 hybrid"]
    y-axis "Total score (%)" 0 --> 100
    bar [100, 86, 99]
    bar [94, 39, 88]

(Mermaid draws the two series over each other: the taller bar behind is Gemini 3.8 Flash, the shorter bar in front is gpt-oss-120b. On this site Gemini is blue and gpt-oss orange.)

Why the stock setup fails with the weak model

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#E8F0FE','primaryTextColor':'#1F2937','primaryBorderColor':'#A8C0E8','lineColor':'#94A3B8','actorBkg':'#E8F0FE','actorBorder':'#A8C0E8','noteBkgColor':'#FDE7EF','noteBorderColor':'#F2B5CB'}}}%%
sequenceDiagram
    participant Ag as gpt-oss + stock setup
    participant Sk as Official skills
    participant CLI as databricks CLI
    Ag->>Sk: load databricks-core
    Sk-->>Ag: pages of instructions
    Ag->>Sk: load databricks-jobs
    Sk-->>Ag: more pages
    Ag->>CLI: databricks jobs create ...
    CLI-->>Ag: error: unknown flag
    Ag->>CLI: databricks workspace import ...
    CLI-->>Ag: error: zip archive contains no items
    Note over Ag: after 8 to 16 calls the model stops<br/>and returns an empty answer, five times in a row

Three causes, in order of weight: the model's own tendency to stop (the "post-tool stall"), long skill text pushing it further, and CLI flags that are easy to get wrong. A stronger model reads the error and corrects itself.

The M6 asterisk

V5 with Gemini scored 25% on M6 (the Lakebase and web-app task) and 100% on everything else. Part of that is our grader, not the agent: M6 checks the older Lakebase "database instances" API, while the official databricks-lakebase skill steers agents to the newer "projects" API. The agent built a working database through the newer route, and the grader looked in the older place. We did not change the grader mid-study. Read V5's M6 number as "partly a mismatch".

GitHub Copilot CLI

GitHub Copilot CLI, development set

GitHub Copilot CLI, held-out set

What we first measured was the adapter, not the agent. Our first Copilot batch scored 54% on the development set and 51% with the same setup, and the reason was not the agent: Copilot CLI adds a private field (__copilotMessageSource) to its requests, the Databricks gateway rejects the request with HTTP 400, and Copilot's next model call after its first tool call dies. On M2, M4, M5 and M7 that produced empty answers with 5 API errors each. We kept those runs, renamed with a _ prefix so they stay out of the tables, as evidence of the failure.

The fix is a 40-line proxy, experiments/copilot_field_shim.py, that deletes every key starting with __copilot and forwards the rest untouched. With it in front of the gateway, the same setup went from 54% to 70%, and from 51% to 92% on the held-out tasks:

What this says about choosing a client: the agent you pick matters less than whether it can talk to your gateway at all. Both clients, given the same tools and rules, handled the well-specified tasks. Both struggled with the same two: M6 (Lakebase plus an app) and M7 (writing documentation into the catalogue). Copilot was more willing to run CLI-style commands and less willing to finish when a tool returned an error.

One task type remains hard for every agent: on the debugging task (X2) Copilot fixed the notebook but, like OpenCode, never re-ran the job, so the job's latest run is still a failure. Fixing the code is not the same as proving the fix works.

Adding the CLI and the official skills (V6) did not help Copilot either: 71% against 70%. It fixed the data-investigation task (27% to 93%) and the Lakebase work (61% to 89%), and lost the SQL-debugging task entirely (100% to 12%). Swapping one tool layer for another moved the score around, it did not raise it.

Two caveats. Copilot runs report no token counts (its output format does not expose them), so we cannot compare cost. And its M2 score is 27% versus 80% for OpenCode with the same model: one run each, and M2 is the task with the widest natural spread.

Held-out tasks: did the improvements generalise?

This is the check against fooling ourselves (05).

So the gains are real for a weak model and invisible for a strong one. That is useful: you can choose how much to engineer based on the model you can afford.

How much is chance?

Models are random. A single run of a setup can land high or low.

Repeatability: V4 / gpt-oss-120b, the same setup run several times

Reading the table:

• The total is stable. Three identical runs scored 94.1%, 96.7% and 94.7%: a spread of about 2.6 points. So a gap of one or two points between two setups (for example V3 and V4, both 94% in one run each) is not a real difference.
• Single tasks are not stable. M8 went from 79% to 100% with no change at all. A one-run result on one task says very little.
• Some weaknesses are real. M2 (the data investigation) was 67% to 80% every time: the agent found the cause (EUR amounts stored in cents) but gave the wrong start date and missed the distractor. That is a repeatable gap, not bad luck.
• Big jumps are real. The 62-point jump from V1 to V2 is about twenty times the noise we measured here.

What we did not repeat: the other setups (V0 to V3, V5, V6), the Gemini runs and the held-out tasks were run once. Treat their numbers as one draw from a spread like the one above.

Failure catalogue: what went wrong, and what fixed it

Lessons about tools

1. Make the agent finish. A supervisor that checks the transcript and pushes on was worth more than any other change.
2. Demand evidence, not claims. "Name the tool call that proves each requirement" turns a confident summary into a checked one.
3. Return real errors. "Failed" teaches nothing; the line and message let the agent fix it.
4. Accept messy input, answer with what you did. Cell counts and "no exit value" notes caught silent failures.
5. Remove tools that mislead. A tool that does something plausible and useless is worse than no tool.
6. Match the setup to the model. A long skill library helps a strong model and can sink a weak one.
7. Test on tasks you did not tune on. Otherwise you will fool yourself.

Validity incidents we found in our own work

Measurement is easy to get wrong. These are the mistakes we made and how each was handled. None of them is hidden: the affected batches were thrown away or re-run.

What we still do not know

• Prices. We measured tokens, not money, and we did not model Genie Code's own charges. See 07.
• Real-world mess. Our data is small and clean compared with a production lakehouse.
• Whether the web terminal works as the shell. The notebook route does.
• Least-privilege tokens. We recommend a scoped service principal but did not test it.
• Scheduled, unattended agents, memory across sessions, and Genie Migrate-style SQL conversion were not measured.

Next: 07 Decision guide.