Consistency
Does the same request give the same answer and the same probabilities?
A model whose answers change on more than 2% of identical requests is marked non-deterministic, and its intervals include that noise.
Answers that change on a rerun same request, sent again
Jev 1.14
TypeSafe
Jev 1.13
TypeSafe
Clef
Cloudflare
Clef-Flash
Cloudflare
pplx-decider v1.2
Perplexity
pplx-decider-flash v1.2
Perplexity
OpenAI Decisions
OpenAI
Microsoft-Decision-1
Microsoft
Kev-4B
Jared Palmer
Share of decisions whose answer changes when the identical request is sent again, in the same run or a day later.
Test items and precision
- Items
- 500 decisions per test set sent twice more: in the same run and a day later
- Precision
- About ±1.6 points at a 3% change rate.