Distractions and long states
Does it hold up when the state is long or full of irrelevant text?
The padding is real text of the same kind from unrelated sources, checked so it contains no option names or clues.
Lengths above a model's context limit show not supported.
Drop with 16k tokens of distraction accuracy points lost
Jev 1.14
TypeSafe
Jev 1.13
TypeSafe
Clef
Cloudflare
Clef-Flash
Cloudflare
pplx-decider v1.2
Perplexity
pplx-decider-flash v1.2
Perplexity
OpenAI Decisions
OpenAI
Microsoft-Decision-1
Microsoft
Kev-4B
Jared Palmer
Accuracy on the original decisions minus accuracy on the same decisions with 16,000 tokens of irrelevant text added to the state.
Test items and precision
- Items
- 300 decisions with 1k, 4k, 16k and 32k tokens of irrelevant text, up to each model's limit
- Precision
- About ±5 points per length; enough for drops of 5 points or more.