On the Faithfulness of Language Model-based Data-to-Text Generation

Abstract

Data-to-Text Generation (D2T)—the task of transforming structured or semi-structured data such as tables, graphs, and meaning representations into readable and faithful natural language—has advanced substantially with the advent of language modeling, particularly with Large Language Models (LLMs). Despite remarkable improvements in language model-based D2T systems in terms of readability, maintaining faithfulness---defined in terms of informativeness and factual consistency---remains a persistent challenge. This thesis presents a systematic exploration of faithfulness in language model-based D2T, focusing on both its evaluation and enhancement, with particular attention to influential factors such as source–reference divergence. The thesis first introduces a structure-aware encoder-decoder framework, based on pre-LLM architectures such as Long Short-Term Memory (LSTM), which encodes input data according to its record-field hierarchy and enhances informativeness through selective gating and hierarchical encoding. It then transitions to LLM-based D2T, conducting a large-scale evaluation across multiple D2T benchmarks and LLM families. The study identifies source-reference divergence---the semantic mismatch between input data and reference text---as a primary factor influencing readability, informativeness, and overall faithfulness. To address this issue, a divergence-weighted training approach is proposed, employing an adaptive loss function that significantly improves model performance. Focusing specifically on factual consistency, the thesis presents a detailed evaluation of widely used LLMs using both automatic and human assessments, with improvements quantified through the Average Rate of Change. The findings demonstrate that source-reference divergence is a major contributor to factual inconsistency. To mitigate this effect, a divergence-aware selective generation framework is introduced, enabling models to abstain from producing factually inconsistent outputs under conditions of high divergence, thereby improving factual consistency without substantial coverage loss. Finally, empirical and mathematical analyses of scaling behavior reveal that factual inconsistency in D2T follows an exponential---rather than a power-law---scaling with respect to model size, compute, and data volume, an observation validated through a statistical framework, \textit{VaCScal}. Collectively, the findings of this research provide both conceptual and empirical insights into faithfulness in language model-based D2T, laying a foundation for the development of future models that are not only readable but also factually faithful.