When a question changes in translation, its original answer key may no longer fit.
In an October 2026 release, AI4Bharat at IIT Madras and NVIDIA describe an offline evaluation suite for Indian-language models. Their explanation identifies a specific problem: translated questions can lose conditions or change task constraints while retaining the original expected answer.
The Breakdown
Consider a hypothetical arithmetic test, not an observed error in any named system. The original asks for the cost of three notebooks at ₹20 each, after a ₹10 discount. The answer is ₹50. If the translated question omits the discount, ₹60 correctly answers the altered question. Keeping ₹50 in the answer key would penalise that correct reasoning.
The mistake sits in the test, not necessarily in the model. A valid comparison must preserve the information needed to solve the problem, rather than merely producing a fluent sentence.
Checking the test itself
The team describes task-specific translation instructions, automated quality checks, corrections and native-speaker review of sampled items. Language coverage and scoring vary by benchmark; sampled review is not a claim that every item received human approval.
Training material answers another question
A separate October 11 preprint, BharatSetu, describes nearly a million English sentences machine-translated into 22 Indian languages. Its authors explicitly report no translation-quality results and do not present the translations as human-verified reference answers.
That distinction gives dataset size its proper meaning. A large collection can supply material for experiments without establishing how accurately a model performs. Training examples and an independently checked test serve different purposes; one cannot silently substitute for the other.
Why This Matters
For an Indian business comparing language tools, the practical question is specific: does the test preserve the task customers will actually face? A product-description translation, a calculation and a request with formatting constraints require different checks. These are possible evaluation choices, not claims about deployments of the research described here.
A useful comparison records the language, task, expected answer and scoring rule together. It also keeps corrections visible: a changed score after repairing a question need not mean that the model itself improved.
The Crunch
The suite covers offline benchmarking, not proof of reliability in a live customer service. Each score describes the test performed.