Read the full analysis: Scaling Indic Pretraining: A Guide To 2.8 Trillion Tokens And Open Data on ThorstenMeyerAI.com
TL;DR
NVIDIA and AI4Bharat at IIT Madras say they plan to release Nemotron-pretraining-Indic-v1, a corpus with about 2.8 trillion retained tokens across 22 Indian languages, plus a GPU-accelerated data-generation pipeline. The release date, licensing terms and detailed quality results have not been specified.
NVIDIA and AI4Bharat at IIT Madras plan to release Nemotron-pretraining-Indic-v1, a corpus containing about 2.8 trillion retained tokens across 22 Indian languages, alongside the data-generation pipeline detailed in the original analysis. The planned resource is intended to give teams more data for training Indic-language models, but a release date and full technical details have not been announced.
The reported inventory contains 2,601.0 billion native-script tokens covering all 22 languages and 224.6 billion romanized tokens across 12 languages. Romanized text represents Indic-language content written in Latin letters. The project counted the tokens using the Gemma-4 tokenizer; the figures describe the corpus under that tokenizer, not the number of unique source documents.
The workflow begins with English material from 15 Nemotron-CC-v2 subsets, including web text, question-and-answer material, mathematics and code. Using NVIDIA’s NeMo Data Designer framework, it translates documents, evaluates translation quality, repairs outputs that score poorly and converts some accepted translations into Latin script. The announcement says records retain source identifiers and checks compare outputs with inputs, including checks intended to preserve equations, tables and code.
The release is described as planned rather than available now. The teams say developers will be able to use the corpus directly or adapt the prompts, workflows and quality checks for other source material. The announcement does not provide a release date, complete filtering rules or acceptance thresholds.
What More Indic Training Data Could Enable
A large corpus could give model developers a substantial additional source of material for languages that have had fewer readily available training resources than English. Its reported subject mix includes technical content and question-and-answer text, potentially useful for systems that answer questions or explain mathematics and programming in Indic languages. Those are possible applications, not demonstrated outcomes: the announcement provides no independent evaluation showing that the corpus improves model quality or performance.
Publishing the generation pipeline as well as the text could let researchers inspect how the data was produced and adapt the workflow to other sources. Source identifiers and input-output checks may aid traceability, but their practical usefulness will depend on the released code, documentation, licensing and quality evidence. The project also says naturally authored Indic data remains essential; translated English material cannot by itself represent the full range of local knowledge, expression and cultural context.
From Sangraha to Translated Data
AI4Bharat is an Indian-language research lab at IIT Madras that works on open datasets and models. The announcement identifies Sangraha, its open pretraining corpus for 22 Indian languages, as part of the foundation for Indic-language model research. Nemotron-pretraining-Indic-v1 would add a large volume of translated material and a described process for generating more from selected English sources.
The project says its 22 languages are those listed in India’s Constitution and describes them as first languages for 97% of Indians, citing the 2011 Census and a population of nearly 1.2 billion. That is a figure tied to the cited census, not a current population estimate. The announcement also says continued pretraining for Nemotron 3.5 Super VL used 10 billion Indic tokens from a 12-language subset alongside naturally authored Indic text. That training use is separate from the reported 2.8 trillion-token corpus inventory.
“We do not have enough data for Indian languages.”
— Mitesh Khapra, head of AI4Bharat
Release Terms and Quality Still Unknown
The announcement does not specify when or where the corpus and pipeline will be published, or the terms for using, adapting or redistributing them. It also does not give per-language token totals, detailed translation-quality scores, filtering criteria or acceptance thresholds. Those details will affect how researchers assess the corpus and whether its contents suit particular training uses.
The reported total includes translations and script variants, and the project cautions that records may represent shared source documents. It is not a count of unique English documents or proof that a model was trained on the entire corpus. The announcement gives no comparative results showing the corpus’s effect on downstream models, and does not fully specify how synthetic translations would be mixed with naturally authored Indic text in future training.
Publication and Independent Review
The next milestone is the planned publication of the corpus and its generation pipeline. Once released, researchers and developers will be able to inspect the data, documentation, code and terms of use, then assess the translation quality and determine whether the workflow can be adapted to other sources.
Further details from the teams could clarify the release schedule, language-by-language inventory, filtering process and evaluation results. Until those materials are available, the stated scale and workflow are project-reported figures and descriptions; the corpus’s practical effect on model performance remains unmeasured in the material provided.
Key Questions
What is Nemotron-pretraining-Indic-v1?
It is a planned corpus of about 2.8 trillion retained tokens across 22 Indian languages, which NVIDIA and AI4Bharat at IIT Madras say they intend to release with its data-generation pipeline.
How many languages and scripts are included?
The corpus has native-script text for 22 languages and romanized text in Latin letters for 12. The announcement describes these as 34 language-script configurations.
Is the corpus available now?
No release date is given. The announcement describes the corpus and pipeline as planned for release, not as already available.
Does the announcement show that the corpus improves model performance?
No. It reports corpus size and describes the generation workflow, but provides no independent evaluation or performance comparison demonstrating an effect on downstream models.
Are the 2.8 trillion tokens unique documents?
No. The count is a tokenizer-based corpus total, and records can include translations and script variants of shared source documents. The announcement says the total does not represent unique English documents.
Primary source: Hugging Face · via ThorstenMeyerAI.com