OpenAI Decisions benchmark
OpenAI's Decisions API, in public beta as of 20 Oct 2026. It answers predicate (yes/no), choice and score questions about text and images and returns a probability for every option, at $0.10 per million input tokens.
As of 20 Oct 2026, decidingbench has not published results for OpenAI Decisions. It is in the planned lineup.
Facts
- Maker
- OpenAI
- Input
- Text and images
- Weights and licence
- Proprietary
Providers each pair is benchmarked separately
- OpenAI API serverless$0.10 per million input tokens, no output charge
Results by measurement
- Accuracy
- Accuracy
- Accuracy, yes/no
- Accuracy, choice
- Accuracy, score
- Confidence you can trust
- Right when 90% sure
- Decisions taken at 0.9
- Threshold honesty at 0.9
- Calibration error
- Brier score
- Order and wording
- Answers that change when options are reordered
- Answers that change when reworded
- Drop from 2 to 20 options
- Distractions and long states
- Drop with 16k tokens of distraction
- None of these
- Picks "None of these" when nothing fits
- Picks "None of these" wrongly
- Scores on a scale
- Accuracy, score
- Score within one level
- Score error in levels
- Consistency
- Answers that change on a rerun
- Safety
- Pushed by text in the state
- Images
- Accuracy with images
- Languages
- Non-English accuracy
- Speed and cost
- Latency, median
- Latency, p90
- Throughput
- Cold start
- Cost per 1,000 decisions
First results coming