Kodiva-Extended: why public benchmarks overstate AI models
Public benchmark scores mix real capability with exposure to the test. Kodiva-Extended adds fresh, private expert questions to tell the two apart.
Top AI models now score around 90% on GPQA Diamond, a set of 198 graduate-level science questions written to be impossible to look up. A few years ago that number would have been astonishing. Today it raises a different question: how much of that score is reasoning, and how much is familiarity?
The problem with famous tests
A benchmark becomes useful the moment it becomes popular, and starts losing value the moment it becomes popular too. Once questions are public, they get discussed, quoted and copied across the internet. Benchmark authors work hard to prevent leakage (GPQA, for example, asks users never to post its questions online), but no one can fully control what ends up in training data.
The result is that a high score on a well-known benchmark is a mix of two things: real capability, and exposure to the test. You can’t separate them by looking at the public score alone.
What a Kodiva-Extended benchmark does
We keep everything about a public benchmark the same, except who wrote the questions.
- Same format. Same question style, same answer format, same difficulty bar.
- Same scoring. We use the benchmark’s own scoring code, unchanged.
- Fresh questions. Specialists from our network write new questions that have never been published. They stay private.
- Both scores, side by side. Every model is tested on the original set and on ours. We publish both.
If a model scores about the same on both, its public score is trustworthy. If it drops sharply on fresh questions, the public score was flattering it.
How we check every question
A question only counts if it passes three checks:
- Expert check. A second specialist in the same field answers it and gets it right. This proves the question and its answer are correct.
- Search check. A non-expert with internet access and 30 minutes gets it wrong. This proves it can’t be looked up.
- Model check. Most of a panel of current models get it wrong. This proves it’s hard enough to measure progress.
Questions that fail any check go back to the writer or get dropped. We also reject questions written with AI tools.
Why we’re doing this
Kodiva runs a network of vetted domain experts: PhDs, physicians, lawyers, engineers and native speakers. Writing hard, original questions is exactly what labs hire them for. Publishing Kodiva-Extended results is how we show that work in the open, so labs can judge its quality before they buy anything.
What’s first
We’re starting with GPQA Diamond (physics, chemistry and biology), followed by MILU (India-specific knowledge across 11 languages) and an Indian law extension of LegalBench. Results will be published on this page together with the full method. We won’t publish any numbers before the runs are checked.
Want your model included in the first leaderboard? Talk to us.