Dissertations - M Tech (CS)

Permanent URI for this collectionhttps://dspace.isical.ac.in/handle/10263/2147

These Dissertations were submitted in partial fulfilment of the requirements for the award of M TECH (Computer Science) Degree of Indian Statistical Institute

Browse

Filters

20262

Settings

Has files: YesSubject: search.filters.subject.Small Language Models

Search Results

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Efficiency Improvement of RAG based SLM for Edge Devices
    (Indian Statistical Institute, 2026-06-16) Avanigadda, Pavan Prashanth
    The increasing need to deploy language models on constrained devices has given rise to efficiency issues in retrieval-augmented generation (RAG) approaches. Although RAGs boost answers’ quality by retrieving knowledge from external sources, current methods utilize static retrieval mechanisms, resulting in unnecessary computation, higher latencies, and inefficiency in resource usage. In this work, an efficient RAG approach based on small language models (SLMs) is presented, which uses a efficient and adaptive retrieval scheme. This method dynamically changes the retrieval depth and context constrution based on the complexity of the query, using a trained MLP router whose routing decisions are learned from Adaptive-RAG-style oracle labels rather than hand-written rules, leading to a compromise between performance and efficiency. A full pipeline is provided, including dataset preprocessing, corpus generation, embedding construction, vector indexation, retrieval, and answer generation processes. Experiments were performed on HotpotQA bench mark and SQuAD 2.0 datasets, comparing the presented approach with the baseline RAG approach using static retrieval scheme. Experimental results show that the proposed approach lowers the computation cost while providing similar answers’ quality. By adaptively controlling retrieval and context size, the framework provides an effective solution for deploying RAG systems in constrained environments.
  • Thumbnail Image
    Item
    A Study of Prompt Tuning on Small Language Models(SLMs): A Controlled Benchmark and a Lightweight Instance-Aware Method
    (2026-06-16) Sahith, Narkadamilli
    Parameter-efficient fine-tuning (PEFT) adapts a frozen pre-trained language model by training only a small number of additional parameters. Among PEFT approaches, prompt tuning prepends trainable continuous vectors (soft prompts) to the input. A recurring finding in the literature is that prompt tuning is strongly scale dependent: it rivals full fine-tuning on very large models but lags on smaller ones. This dissertation studies prompt tuning specifically in the small-language-model (SLM) regime. We (i) re-implement a representative set of prompt-tuning methods—Prompt Tuning, P-Tuning v2, LoPT, DPT, DePT, ACCEPT, Residual Prompt Tuning, and PARA—within a single controlled harness, enabling a fair head-to-head comparison against full fine-tuning; (ii) propose IA-DePT, a lightweight instance-aware extension of Decomposed Prompt Tuning that conditions the short soft prompt on each input through a small, zero-initialised gate; and (iii) extend the benchmark beyond a single backbone and task, evaluating the full method suite on six backbone/task settings that span encoder–decoder (t5-small), encoder-only (BERT-base, RoBERTa-base, ELECTRA-small), and decoder-only (DistilGPT-2) architectures across the GLUE/SuperGLUE tasks RTE, WSC, CB, COPA, WiC, and MRPC. On RTE with t5-small, IA-DePT is the strongest parameter-efficient method in our benchmark (55.6% single-seed accuracy) and improves over its own base, DePT, by 6.5 points (53.6% vs. 47.1%, mean over three seeds) while adding only ≈16.9k parameters—a total trainable footprint of 0.05% of the backbone. Because the gate degrades exactly to DePT at initialisation, the comparison is a clean single-variable ablation. The cross-architecture study shows that the instance gate improves on DePT in five of the six settings on each setting’s primary metric (it ties or marginally regresses only on WiC, where every PEFT method sits at chance), so the benefit is broad but not universal. Our analysis characterises the accuracy/parameter trade-offs across method families, the strong effect of task difficulty on the small-model regime, and the role of instance-conditioning, including an honest discussion of why many prompt-tuning methods remain close to the chance baseline at this scale.