Frequently asked questions
What is a decision model?
A decision model takes a state (text, JSON and sometimes images) and one or more typed questions, and returns a probability for every option the caller listed, in one pass, instead of writing free text. It can only answer with options you gave it. TypeSafe's Jev, Cloudflare's Clef, Perplexity's pplx-decider and OpenAI's Decisions API are examples.
What is the best decision model?
As of 20 Oct 2026 there is no independent answer that checks decision models against real recorded outcomes. decidingbench will rank Jev, Clef, Clef-Flash, pplx-decider, OpenAI Decisions, Microsoft-Decision-1 and Kev on how often each is right when it is confident enough to act, with 95% intervals, and will only call one model better than another when the intervals support it.
Jev vs Clef: which is better?
Not measured independently yet. Cloudflare's own leaderboard from 1 Oct 2026 puts Clef ahead of Jev, but Cloudflare built and ran it. decidingbench will test both on the same real decisions, on the same private set, and report accuracy, calibration, speed and cost side by side.
How accurate is TypeSafe Jev?
TypeSafe publishes no benchmark table for Jev; its blog says it chose not to. decidingbench will publish Jev 1.13 and 1.14 accuracy on five kinds of real decisions, how often Jev is right when it is at least 90% sure, and whether its stated confidence matches how often it is right.
Is Jev calibrated?
TypeSafe describes Jev as calibrated and says its score levels are weak in numerical calibration. decidingbench measures it directly: at a 0.9 threshold, how many decisions Jev takes and how often it is right on those, plus calibration error and a reliability diagram.
Are there open-source alternatives to Jev?
Yes. Cloudflare's Clef and Clef-Flash have open weights under Apache 2.0. Others include Kev (Apache 2.0, served on OpenRouter as Kev-4B), Von, Laya, H2O-Lightning-4B, CLM and community fine-tunes such as Chon-1.0-12B. Most have no hosted API yet, so decidingbench lists them as not tested until one exists.
Which decision model is best for gating agent tool calls?
That is what decidingbench's agent action approval track measures: whether a coding agent's pull request was merged or closed by a maintainer, decided from the pull request as it was opened. Results are not published yet.
How is a decision model different from asking an LLM to pick an option?
An LLM writes its answer token by token and can be asked to name an option. A decision model returns a probability for every listed option in one forward pass, so it cannot answer outside the list and its confidence is part of the output. Decision models charge for input tokens only. decidingbench ranks decision models only, not general LLMs.
Where do the right answers come from?
From what actually happened. Each item is a real decision someone made for their own reasons before this benchmark existed: a maintainer merged or closed a pull request, labelled an issue, a community closed a question, an editor reverted an edit, a reviewer rated an article. Code extracts the outcome, a second extractor checks every item, and two independent auditors check a sample. Error rates are published.
Why is part of the test set private?
Public test sets end up in training data. Every private item is a decision made after the newest tested model was released, so no model can have seen its outcome. We publish a salted hash of every private item before any model sees it, rotate the private set at least every 8 weeks, and publish retired items so anyone can check the hashes.
Why test the same model on several providers?
The same model can differ in speed, price, limits and sometimes answers from one host to another. decidingbench measures every model on every provider that serves it, with latency, throughput, cold start and billed cost, and never ranks a model we run ourselves against hosted providers on speed or cost.
Can I rerun the benchmark myself?
Yes, the public set. One command runs a model on it with your own API key and scores it with the same code we use, then compares your numbers with ours.
Who runs decidingbench, and who pays for it?
Leo runs it, independently. No model vendor or sponsor has any say in the method or the results. If sponsors ever appear, they will be named and will have no influence on rankings.
Can you add my model?
If it is a decision model (options in, a probability per option out) and it is served by a public API, yes. Adding one is a small adapter. Contact Leo on X, or open an issue once the repository is public.