Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
NVIDIA and AI4Bharat are releasing 2.8 trillion tokens across all 22 constitutional Indian languages — native-script for 22, romanized for 12 — plus the full GPU-accelerated generation pipeline including prompts, LLM workflows, and quality checks, with every synthetic record traced and validated against a curated source document. If you're fine-tuning or pretraining for Indic markets, this is the largest open grounded resource yet, and the released recipe means you can regenerate domain-specific data (math, code, QA) from your own sources rather than scraping or paying for proprietary translation.
2.8 trillion Indic pretraining tokens are being released across 22 Indian languages, with native-script data for all 22 and romanized data for 12, plus the GPU-accelerated pipeline that generated them. This makes continued pretraining or domain adaptation for Indian-language LLMs materially easier: teams can start from a large grounded synthetic corpus and reuse the translation, QA, quality-check, and repair workflow on their own English source material instead of building an Indic data factory from scratch.