Confidence you can trust
When it says 90% sure, is it right 90% of the time, and how much can it handle at that level?
Agents use a decision model's probability one way: act above a threshold, hand off the rest. We report, at thresholds 0.5, 0.7, 0.8, 0.9 and 0.95, how many decisions the model takes and how often it is right on those.
Threshold honesty shows whether 90% means 90%. Calibration error, Brier score and a reliability diagram complete the picture.
Right when 90% sure accuracy on the decisions it takes at 0.9
Selective accuracy at a 0.9 threshold: accuracy on the decisions where the model's confidence is at least 0.9. Shown only when it acted on at least 50 decisions.
Decisions taken at 0.9 share it is at least 90% sure about
Coverage at a 0.9 threshold: the share of decisions where the model's confidence is at least 0.9, so an agent would let it act.
Threshold honesty at 0.9 below 0 means its 90% is not 90%
Selective accuracy at the 0.9 threshold minus 90 points. Negative means the model is overconfident at that threshold.
Calibration error lower is better
Expected calibration error with 15 equal-count bins on top-option confidence: how far stated confidence is from how often the model is right.
Brier score lower is better
Mean squared error of the probabilities over all options. Rewards being right and being honest about how sure it is.
Test items and precision
- Items
- Every decision in the core tracks
- Precision
- About ±1.6 points on selective accuracy at 0.9 when a model acts on half the decisions.