CL-1
100%37/37
On tasks never used while tuning, the strongest measured setup answers every one
Evidence
On this page
Every number OpenGenie publishes, with the fraction it was computed from, how many runs stand behind it, and what it does not prove. Nothing here is typed by hand: site/claims.py reads exams/results/*.json and produces this page, and the build fails if a figure and its source ever disagree.
There are eleven graded tasks (the harness calls them missions) and 174 saved run files. Every score below was produced by seeding a real workspace, handing an agent a plain-language request, and then grading the workspace afterwards — not by asking a model whether the work looked right.
The claims
CL-2
86%32/37
With the model most workspaces can actually reach, the same setup still passes most of it
CL-3
86%130/152
Same agent, same stock tools, same rules: only the model changed
CL-4
94%143/152
Scoped tools, a supervisor loop and written rules lifted that model from a bare shell
CL-5
100%37/37
With a strong model the very first setup scored the same, so the machinery was not what did it
CL-6
95%95/100
Running the same setup again moves the total by a couple of points, not tens
CL-7
671ktokens per task, against 311k
The stronger model reads and writes about twice the tokens for the same eight tasks
CL-8
11graded tasks
Every graded task was seeded in a live workspace and scored against it, not judged by a model
How the score is produced
Three steps, and the third is the one that makes the number mean anything:
- Seed. Each task creates real state: tables with deliberate traps, a broken job, a support ticket containing an instruction. The agent cannot tell the fixtures from a customer's workspace.
- Ask. The agent receives a plain-language request and no grader.
- Grade. Afterwards, the harness reads the live workspace: does the table match the golden answer in both directions, did the notebook actually run, is the job's latest run a success, was the injected instruction ignored. A grader is trusted only once a reference solution scores 100% on it and a do-nothing agent fails it.
The full method, including how the graders could be wrong, is the exam suite.
What the numbers are, and are not
Small
Eight development tasks and three held-out tasks. A difference of a few points between two setups is not a difference; it is noise. We measured that noise once: about three points.
Clean
The fixtures are small and tidy. Production lakehouses have nulls, duplicates, schemas that change under you, and ten thousand tables. Nothing here shows how the agent behaves at that scale.
Well specified
The exams name their tables. Finding the right table in a large catalogue is where Genie's ontology helps most, and it is untested here.
Single-run
One run per setup, except one configuration that was repeated three times. A model is random; a single run can land high or low.
Tokens, not money
We measured tokens because we had them. We had no Genie Code usage history, so no saving can be quoted from this repository, and none is.
Tuned where it says so
The development set was used to choose the tools and the rules. The held-out set was not, and it is where the generalisation claim rests.
The excluded runs, named
A page that quietly drops a tenth of its evidence is worse than one that admits it. These batches exist in the repository and are counted nowhere above:
| Batch | Why it is excluded |
|---|---|
_copilot_before_shim_dev, _copilot_before_shim_held, _copilot_before_shim_smoke | GitHub Copilot CLI adds a private field the Databricks gateway rejects; the runs measured our adapter, not the agent. Kept as evidence of the failure. |
_old_v1_pre_port_fix | A port collision in the harness, fixed afterwards. |
accidental_dbsql_only | A runner misconfiguration: the agent got SQL only. |
smoke_v5 | A smoke test, not a graded run. |
v4_gptoss_t2, v4_gptoss_t3 | Repeats of one setup. They are the subject of the repeatability claim and are counted there, and nowhere else. |
Every accepted run file is in
exams/results,
one JSON per task plus the agent's full event log. Regenerate every table with
python3 exams/make_tables.py; regenerate this page with python3 site/build.py.
The one thing the evidence does not answer
Whether Genie Code would score worse, the same, or better on these tasks. Nobody ran it: it is a metered product inside a workspace, and this study had a token for a workspace but no way to drive Genie Code itself. So the comparison here is between configurations of an open agent, not between OpenGenie and Genie Code on the same ruler.
What that changes: the claims are about which model and which guard rails you should choose, not about beating a specific product. What it does not change: the model, the fence and the evidence rule matter for the reasons given on the home page, and those reasons do not depend on a competitor's score.