NVIDIA and IIT Madras unveil 2.8 trillion-token Indic pretraining dataset

NVIDIA and AI4Bharat, the Indian-language research lab at IIT Madras, announced on 16 October 2026 a new pretraining corpus called Nemotron-pretraining-Indic-v1. The dataset holds roughly 2.8 trillion tokens across the 22 languages listed in India's Constitution. The partners said it will be released soon, together with the GPU-accelerated generation pipeline used to build it, including prompts, LLM workflows and quality checks.

Those 22 languages are the first languages of 97% of Indians, close to 1.2 billion people according to the 2011 Census. AI4Bharat head Mitesh Khapra has argued that Indian language technology lags English because "we do not have enough data for Indian languages." The lab previously released Sangraha, an open pretraining corpus for the same 22 languages.

What the corpus contains

Measured with the Gemma-4 tokenizer, the corpus contains 2,601.0 billion native-script tokens and 224.6 billion romanized tokens. Twelve languages, among them Bengali, Hindi, Tamil, Telugu and Urdu, appear in both native and Latin script, while ten others, including Sanskrit, Kashmiri and Santali, are native-script only, for 34 language-script configurations in total. The team stresses that native-script text is still translated from English, and that romanized records are Indic text written in Latin letters rather than English.

Source material comes from 15 subsets of Nemotron-CC-v2, covering web text, question-answer content, mathematics and code. Translation, scoring and repair run on NeMo Data Designer, NVIDIA's open-source synthetic data framework, and each record keeps its source identifier so outputs can be traced to the original document.

Translate, score, repair

For the twelve dual-script languages, whole documents are translated with Gemma-4-31B-it, preserving code, equations, URLs and formatting. The same model then grades each output against its English source using a rubric inspired by Multidimensional Quality Metrics (MQM), covering meaning, fluency, terminology, style and formatting. Only outputs scoring at least 75 out of 100 are kept; weaker ones are repaired using the feedback and rescored, a step the team says recovered 438.4 billion tokens. The remaining ten languages go through a separate translation workflow.

Part of the data is already in use. Continued pretraining of Nemotron 3.5 Super VL added 10 billion Indic tokens drawn from a twelve-language subset of the corpus and from natural Indic text, and NVIDIA says future Nemotron models will build on it. The authors also note that synthetic translations complement, rather than replace, naturally written Indic text, which remains essential for local knowledge and cultural context.