# decidingbench

> An independent benchmark of decision models such as TypeSafe Jev, Cloudflare Clef, Perplexity pplx-decider and OpenAI Decisions: accuracy on real recorded decisions, selective accuracy at confidence thresholds, calibration, robustness, speed and cost per provider, with 95% intervals.

As of 20 Oct 2026, decidingbench has not published results. The benchmark design is public and the first run is planned. Published figures for these models so far come from vendors (Cloudflare's leaderboard) or from boards that use written or academic items (JevBench, the Artificial Analysis Decision Index). As of this date we found no independent test against real recorded outcomes.

## Models

- Jev 1.14 (TypeSafe): not run yet. TypeSafe's newest decision model, available since 14 Oct 2026. It answers choice, score and yes/no questions about a text or JSON state in one pass and returns a probability for every option. Image input is available behind an opt-in header. It costs $0.042 per million input tokens. https://www.decidingbench.com/models/jev-1-14
- Jev 1.13 (TypeSafe): not run yet. The Jev version TypeSafe's jev-latest alias pointed to on 20 Oct 2026. Text and JSON states, choice, score and yes/no questions, a probability for every option, $0.042 per million input tokens. TypeSafe launched Jev, the first decision model, on 15 Sep 2026. https://www.decidingbench.com/models/jev-1-13
- Clef (Cloudflare): not run yet. Cloudflare's larger decision model, released on 1 Oct 2026 with open weights under Apache 2.0, post-trained from Qwen3.8-27B. It follows the same request format as Jev, takes text and up to four images, has a 65,536-token context and costs $0.24 per million input tokens on Workers AI. No other host serves it as of 20 Oct 2026. https://www.decidingbench.com/models/clef
- Clef-Flash (Cloudflare): not run yet. Cloudflare's small decision model, a post-train of Qwen3.5-9B with a head that scores all options together, released with open weights under Apache 2.0 on 1 Oct 2026. Text and up to four images, a 24,576-token context, $0.038 per million input tokens on Workers AI. https://www.decidingbench.com/models/clef-flash
- pplx-decider v1.2 (Perplexity): not run yet. Perplexity's 27B decision model. It takes yes/no, choice (up to 255 options) and score questions with up to 8 images per question, a 256k context, and costs $0.02 per million input tokens. https://www.decidingbench.com/models/pplx-decider
- pplx-decider-flash v1.2 (Perplexity): not run yet. Perplexity's 4B decision model, on the same API as pplx-decider, at $0.01 per million input tokens. https://www.decidingbench.com/models/pplx-decider-flash
- OpenAI Decisions (OpenAI): not run yet. OpenAI's Decisions API, in public beta as of 20 Oct 2026. It answers predicate (yes/no), choice and score questions about text and images and returns a probability for every option, at $0.10 per million input tokens. https://www.decidingbench.com/models/openai-decisions
- Microsoft-Decision-1 (Microsoft): not run yet. Microsoft's decision model on Microsoft Foundry. It uses the System One request format with yes/no, choice and score questions over a text or JSON state. It needs an Azure subscription and a deployment. https://www.decidingbench.com/models/microsoft-decision-1
- Kev-4B (Jared Palmer): not run yet. An open decision model from Jared Palmer, Qwen3.5-4B plus an adapter, Apache 2.0, that serves the same /v1/systemone request format as Jev. Text only. Hosted on OpenRouter at $0.042 per million input tokens. https://www.decidingbench.com/models/kev-4b

## Measurements

- Accuracy: How often does it match what really happened? https://www.decidingbench.com/measurements/accuracy
- Confidence you can trust: When it says 90% sure, is it right 90% of the time, and how much can it handle at that level? https://www.decidingbench.com/measurements/confidence
- Order and wording: Does the answer change when the options are shuffled, the question is reworded, or more options are added? https://www.decidingbench.com/measurements/order-and-wording
- Distractions and long states: Does it hold up when the state is long or full of irrelevant text? https://www.decidingbench.com/measurements/distractions
- None of these: What does it do when none of the listed options is right? https://www.decidingbench.com/measurements/none-of-these
- Scores on a scale: When it rates something on a scale, how close does it land? https://www.decidingbench.com/measurements/scores-on-a-scale
- Consistency: Does the same request give the same answer and the same probabilities? https://www.decidingbench.com/measurements/consistency
- Safety: Can text inside the state push the decision? https://www.decidingbench.com/measurements/safety
- Images: How good is it when the state includes a photo or screenshot? https://www.decidingbench.com/measurements/images
- Languages: How good is it on decisions that aren't in English? https://www.decidingbench.com/measurements/languages
- Speed and cost: How fast is it, and what does it really cost, on each provider? https://www.decidingbench.com/measurements/speed-and-cost

## Method

- Deciding Index: How much value a model adds when an agent lets it act only above a confidence threshold, with wrong actions costing more at stricter thresholds. Averaged over thresholds 0.5, 0.8 and 0.9 and over five tracks, on the private set. A model that always hands off scores 0; a perfect one scores 100.
- Five tracks of real decisions (agent pull request approval, issue routing, Stack Exchange moderation, Wikipedia edit review, article quality scoring), each with a public test set anyone can rerun and a private set of decisions made after every tested model was released. Labels are recorded outcomes, checked by a second extractor and two auditors, with published error rates.
- 95% intervals from a bootstrap clustered by source. A model is called better only when the interval on the difference excludes zero.
- Full method: https://www.decidingbench.com/methodology

## Data

- https://www.decidingbench.com/results.json
- https://www.decidingbench.com/results.csv

## Pages

- https://www.decidingbench.com
- https://www.decidingbench.com/what-is-a-decision-model
- https://www.decidingbench.com/models
- https://www.decidingbench.com/models/jev-1-14
- https://www.decidingbench.com/models/jev-1-13
- https://www.decidingbench.com/models/clef
- https://www.decidingbench.com/models/clef-flash
- https://www.decidingbench.com/models/pplx-decider
- https://www.decidingbench.com/models/pplx-decider-flash
- https://www.decidingbench.com/models/openai-decisions
- https://www.decidingbench.com/models/microsoft-decision-1
- https://www.decidingbench.com/models/kev-4b
- https://www.decidingbench.com/measurements
- https://www.decidingbench.com/measurements/accuracy
- https://www.decidingbench.com/measurements/confidence
- https://www.decidingbench.com/measurements/order-and-wording
- https://www.decidingbench.com/measurements/distractions
- https://www.decidingbench.com/measurements/none-of-these
- https://www.decidingbench.com/measurements/scores-on-a-scale
- https://www.decidingbench.com/measurements/consistency
- https://www.decidingbench.com/measurements/safety
- https://www.decidingbench.com/measurements/images
- https://www.decidingbench.com/measurements/languages
- https://www.decidingbench.com/measurements/speed-and-cost
- https://www.decidingbench.com/methodology
- https://www.decidingbench.com/faq
- https://www.decidingbench.com/about