11 measurements, one small headline index
The Deciding Index combines only how often a model is right on the decisions it is confident enough to take, at three thresholds, with wrong actions costing more at stricter ones. Everything else is a separate measurement with its own method, test items and stated precision.
- AccuracyHow often does it match what really happened?Planned. First results coming →
- Confidence you can trustWhen it says 90% sure, is it right 90% of the time, and how much can it handle at that level?Planned. First results coming →
- Order and wordingDoes the answer change when the options are shuffled, the question is reworded, or more options are added?Planned. First results coming →
- Distractions and long statesDoes it hold up when the state is long or full of irrelevant text?Planned. First results coming →
- None of theseWhat does it do when none of the listed options is right?Planned. First results coming →
- Scores on a scaleWhen it rates something on a scale, how close does it land?Planned. First results coming →
- ConsistencyDoes the same request give the same answer and the same probabilities?Planned. First results coming →
- SafetyCan text inside the state push the decision?Planned. First results coming →
- ImagesHow good is it when the state includes a photo or screenshot?Planned. First results coming →
- LanguagesHow good is it on decisions that aren't in English?Planned. First results coming →
- Speed and costHow fast is it, and what does it really cost, on each provider?Planned. First results coming →
- Deciding IndexHow the one headline number is built, and why it is small.Read the formula →