Developing a Model to Generate More Digital Data of Indian Languages for Multilingual Applications

Abstract

Most of India’s scheduled languages remain critically under-served by language technology because parallel (translated) text — the raw material that modern multilingual systems depend on — is extremely scarce. Back-translation can synthesise such data automatically, but its quality varies enormously, and unfiltered synthetic data can be worse than no data at all. This dissertation develops a framework that generates synthetic parallel data for four low-resource Indian languages spanning three language families and four scripts — Assamese (Indo-Aryan, Bengali script), Bodo (Tibeto-Burman, Devanagari), Manipuri (Tibeto-Burman, Bengali script) and Santali (Austroasiatic, Ol Chiki)—and introduces CASCADE, a learned multi-signal quality gate that scores each synthetic pair from four cheap signals: semantic similarity, round-trip consistency, length ratio, and language-identification confidence. CASCADE attains a held-out ROC–AUC of 0.954 and 91.7% accuracy at separating well-aligned pairs from misaligned ones, outperforming every single-signal filter (best single signal: round-trip chrF++, AUC 0.932). An ablation shows that round-trip consistency is the dominant signal, while language-identification confidence carries no quality information (AUC 0.534). Three algorithms are contributed on top of the gate: a cost-aware staged cascade that reduces filtering compute by 37.6%, a quality-diversity selector that preserves the data diversity that naive top-K filtering destroys, and an adaptive operating-point selector. Finally, a single multilingual generator of 52.5M parameters is trained from scratch on the curated, tagged data; steered by a target-language token, it produces text in all four languages in their correct scripts, and the low-resource languages benefit measurably from joint training. An honest analysis characterises when quality filtering improves downstream translation and when generation quality, rather than selection, is the binding constraint.