Order and wording
Does the answer change when the options are shuffled, the question is reworded, or more options are added?
The meaning stays the same, so the answer should too. Paraphrases are audited before use.
Option-count sets are nested, so only the number of options changes.
Answers that change when options are reordered same options, different order
Jev 1.14
TypeSafe
Jev 1.13
TypeSafe
Clef
Cloudflare
Clef-Flash
Cloudflare
pplx-decider v1.2
Perplexity
pplx-decider-flash v1.2
Perplexity
OpenAI Decisions
OpenAI
Microsoft-Decision-1
Microsoft
Kev-4B
Jared Palmer
Share of choice decisions whose answer changes when the options are shifted, and of yes/no decisions that flip the wrong way when the question is negated.
Answers that change when reworded same meaning, different words
Jev 1.14
TypeSafe
Jev 1.13
TypeSafe
Clef
Cloudflare
Clef-Flash
Cloudflare
pplx-decider v1.2
Perplexity
pplx-decider-flash v1.2
Perplexity
OpenAI Decisions
OpenAI
Microsoft-Decision-1
Microsoft
Kev-4B
Jared Palmer
Share of decisions whose answer changes across three audited paraphrases of the question and option descriptions.
Drop from 2 to 20 options accuracy points lost
Jev 1.14
TypeSafe
Jev 1.13
TypeSafe
Clef
Cloudflare
Clef-Flash
Cloudflare
pplx-decider v1.2
Perplexity
pplx-decider-flash v1.2
Perplexity
OpenAI Decisions
OpenAI
Microsoft-Decision-1
Microsoft
Kev-4B
Jared Palmer
Accuracy with 2 options minus accuracy with 20, on the same routing decisions with nested sets of real labels.
Test items and precision
- Items
- 1,000 reordered choice decisions and 500 negated yes/no decisions; 500 decisions in three paraphrases; 300 routing decisions at 2, 5, 10 and 20 options
- Precision
- About ±1.5 to ±2.4 points on changed answers; about ±6 on the option-count drop, enough for drops of 8 points or more.