How the benchmark works

This is a summary of the benchmark design, version 1.0 draft of 20 Oct 2026. The design is waiting for approval, and no model has been run on a test item yet. When the first run is published, this page will link the preregistration that fixed every rule below before any private item existed.

What is tested

Each test item is one real decision: the state a person saw when they had to decide, the options they had, and what they actually decided, recorded in the normal course of their work before this benchmark existed. The model sees only the state as it was at the start. There are five core tracks, each its own dataset with 500 decisions per test set:

Agent action approval (yes/no): was a coding agent's pull request merged or closed by a maintainer?
Ticket routing (choice): which area label did a maintainer give a GitHub issue?
Content moderation (choice of six): did a new Stack Exchange question stay open, and if not, why was it closed?
Edit review (yes/no): was a Wikipedia edit by a new or unregistered editor reverted within 48 hours?
Quality scoring (score, five levels): what class did WikiProject reviewers give a new article?

Drill-down tracks add photos (Wikimedia Commons quality image reviews), screenshots in issues, and edit review in German, Spanish, French and Japanese.

How the answers are checked

Code extracts each outcome from the public record using written rules: who counts, the window the outcome must happen in, what happens when a decision is reversed, and which items are dropped because the decision depended on something the model can't see. A second extractor, written separately from the rules alone, runs on every item, and every disagreement is settled by hand. Two auditors independently check 300 items per track and test set against the source, and a person re-checks a sample of their work. We publish the agreement between auditors, the label error rate with its interval, and how often an outcome couldn't be explained from the state.

Public and private test sets

Every track has a public set, from decisions made between May and August 2026, which anyone can download and rerun, and a private set of decisions made after the newest tested model was released, so no model can have seen how they turned out. Before any model sees a private item we publish a salted hash of it. The private set rotates at least every 8 weeks or whenever a new model version is added, every model is re-scored on the new rotation, and retired items are published so anyone can check the hashes.

Both sets are drawn by the same code with the same quotas and checked for balance before they are frozen. Every model's public and private scores are shown side by side, next to simple baselines that can't be contaminated, so a model that does much better on the public set than drift explains stands out.

Confidence you can act on

Every decision model returns a probability for each option. An agent uses that by setting a threshold: act when the model is at least that sure, hand off the rest. So for each model we report, at thresholds 0.5, 0.7, 0.8, 0.9 and 0.95, how many decisions it takes and how often it is right on those, plus whether its 90% really means 90%.

The headline index

Deciding Index = 100 × the average net value over the five core tracks and over thresholds 0.5, 0.8 and 0.9, on the private set. At threshold t, a decision the model hands off is worth 0, a right decision it takes is worth +1, and a wrong one it takes costs t ÷ (1 − t): 1 at 0.5, 4 at 0.8, 9 at 0.9. A model that always hands off scores 0 and a perfect model scores 100. An overconfident model can score below 0. Every track counts equally. The index is small on purpose: every other measurement has its own page.

The measurements

Accuracy by question type and track; confidence you can trust; order and wording (reordered options, reworded questions, 2 to 20 options); distractions and long states; "None of these"; scores on a scale; consistency on identical requests; safety against text that tries to push the decision; images; languages; and speed and cost. Each page says how many items it uses and what size of difference that can show.

Intervals and winners

Every number has a 95% interval from 10,000 bootstrap resamples, clustered by source and stratified by track. Comparisons between two models use the same resamples for both. A model is called better than another only when the interval on their difference excludes zero. Otherwise the page says the two can't be told apart.

How the results are checked

Every request and response is stored. Responses for the public set are published.
Two scorers, written separately in two languages, must agree on every metric before anything is published, and both must pass hand-computed test cases.
500 decisions per test set are sent again, in the same run and a day later, to check answers don't change.
Raw responses are traced to scores by hand for a sample of every model and provider, and every response is checked for options we didn't send.
An independent reviewer checks the scoring code and a sample of traces before each publication.

Providers, speed and cost

Each model is tested on every provider that serves it. Latency is timed from one server, network included, with retries, rate limits and errors shown next to every number. Cost per 1,000 decisions is what the provider billed. Small open models we run ourselves are added later, reported separately with memory use, and never ranked against hosted providers on speed or cost.

Rerun it yourself

The public test set, the harness and both scorers will be in the public repository when results are published. One command runs a model on the public set with your own API key and compares your numbers with ours.