Accuracy
How often does it match what really happened?
Every answer is checked against an outcome someone recorded for their own reasons: a pull request merged or closed, an issue labelled by a maintainer, a question closed by the community, an edit reverted, an article assessed.
Broken down by question type (yes/no, choice, score) and by track. Per-track numbers can only separate large gaps.
Accuracy share of decisions that match what happened
Share of decisions where the model's top option matches the recorded outcome. Every track counts equally.
Accuracy, yes/no agent action approval and edit review
Accuracy on yes/no decisions: whether a coding agent's pull request was merged and whether a Wikipedia edit stood.
Accuracy, choice ticket routing and moderation
Accuracy on choice decisions: which area a maintainer gave an issue, and what happened to a Stack Exchange question.
Accuracy, score exact level on a five-level scale
Share of article quality ratings where the model's top level matches the WikiProject assessment exactly.
Test items and precision
- Items
- 2,500 decisions per test set: 500 in each of five tracks
- Precision
- About ±3 points by question type, about ±4 within one track.