Indian law in Indian languages: the gap no benchmark covers
Legal benchmarks are mostly English and US law; Indian-language benchmarks test general knowledge. Nothing yet tests Indian law in Indian languages.
Millions of Indians will ask AI tools legal questions in Hindi, Tamil, Bengali and other languages long before they ever speak to a lawyer. Questions like whether a rental agreement is valid, what a notice from a bank means, or how to respond to a consumer dispute. Today there is no good way to measure how well models answer them.
What exists today
- Legal benchmarks are mostly in English and built around US law. LegalBench, from Stanford, is the best-known: a large collection of legal reasoning tasks, almost all grounded in American legal concepts.
- Indian-language benchmarks test general knowledge. MILU, from AI4Bharat, covers India-specific knowledge across 11 languages and 41 subjects. It’s an important step, but it isn’t built to test legal reasoning.
- Nothing combines the two. There is no widely used benchmark that asks models to reason about Indian law, in Indian languages, the way real people would ask.
Why it matters
Legal answers are high-stakes. A model that translates English legal reasoning into Hindi can still be wrong about Indian law: the wrong act, an outdated rule, a concept that doesn’t exist here. And a model that’s accurate in English may fall apart when the same question is asked in Marathi with everyday phrasing and mixed-in English words.
Without a benchmark, model builders can’t see these failures, and users can’t tell which tools to trust.
What we’re building
An Indian-law benchmark, written by practising Indian lawyers and native speakers:
- Real situations, not textbook questions: tenancy, consumer complaints, employment, contracts, family and property matters, written the way people actually describe them.
- Multiple languages, starting with Hindi and English, then expanding to more of India’s major languages.
- Expert answers and reasoning, citing the relevant law, checked by a second lawyer.
- The same quality checks we use for every Kodiva benchmark: a second-expert review, a search test and a model test.
It follows our first benchmarks, GPQA Diamond and MILU (both Kodiva-Extended), and results will be published on this page with the full method.
This is not legal advice
The benchmark measures models. It doesn’t give advice to anyone, and neither should any AI tool without proper safeguards.
Building legal or Indian-language AI? Get notified when results go live, or talk to us about evaluating your model.