Efficient XGLM adaptation for Ayacucho Quechua

Overview

Multilingual language models cover many languages at once, but rarely evenly. A model can be fluent in the languages that dominated its pretraining and close to useless in those that barely appeared. This study follows one such model from that uneven starting point through a targeted adaptation to a language it handles poorly, asking a practical question along the way: can the weak language be improved without retraining the model in full, and without eroding the languages it already knows?

The work proceeds in three stages that build on one another. The first measures how well the pretrained model fits each language and establishes where it falls short. The second looks inside the network to see whether language identity is encoded in the hidden states, and how that structure shifts with depth. The third adapts the model under a tight parameter budget while watching for cross-language forgetting. Together they treat adaptation as something to analyze rather than only to perform. The study was carried out for Neural Networks: Theory and Implementation at Saarland University.

The model under study is XGLM-564M, a multilingual autoregressive transformer.^[01]01Lin et al. (2022), “Few-shot Learning with Multilingual Generative Language Models,” EMNLP 2022. Evaluation draws on FLORES, a human-translated benchmark of aligned sentences across many languages.^[02]02Costa-jussà et al. (2022), “No Language Left Behind: Scaling Human-Centered Machine Translation.” Alignment matters here: a difference in loss should reflect the model and the language as much as possible, rather than unrelated topics drawn from separate corpora.

Ayacucho Quechua (quy_Latn) became the adaptation target once the first stage exposed a large modeling gap. The question that followed was concrete: can Quechua improve without touching every parameter, and what happens to the model’s other languages when it does?

Measuring the multilingual gap

The first stage builds a straightforward inference pipeline: load FLORES, tokenize and batch sentences, run a pretrained causal language model, and compare loss across languages. XGLM-564M and GPT-2 were evaluated on the same 200 FLORES dev sentences for nine languages.

The evaluator masks padding, prepends the model’s context token, and scores every token from the original text. Alongside loss, it records predicted-token and UTF-8 byte counts. Those counts expose a central comparison problem: a low token-average loss may partly reflect how a tokenizer segments the language rather than how well the model predicts it.

Raw perplexity is unsuitable for comparing these two models because their tokenizers split the same text differently. The comparison therefore uses bits per UTF-8 byte (BPB):

\[ \operatorname{BPB} = \frac{-\sum_t \log p(x_t\mid x_{\lt t})}{B\log 2}, \]

where $B$ is the byte count of the original text. Lower BPB means the model assigns more probability to the text per byte, without rewarding a tokenizer simply for producing fewer tokens.

XGLM records lower BPB in seven of nine languages. GPT-2 is lower for English and slightly lower for Gujarati. Ayacucho Quechua has the highest BPB under both models: 2.902 for XGLM and 3.865 for GPT-2. This does not isolate pretraining data, capacity, or tokenizer design, but it identifies XGLM as the stronger starting point in this comparison and Quechua as a clear adaptation target.

Inspecting representation geometry

The second stage moves from output loss to internal state. Hidden vectors were collected from the embedding output and every transformer layer for each non-padding token in 200 sentences per language. The tensors, corresponding tokens, and sentences were stored in HDF5. A sentence vector was also formed by padding-aware mean pooling:

\[ s^{(\ell)}=\frac{1}{\sum_t m_t}\sum_t m_t h_t^{(\ell)}, \]

where $h_t^{(\ell)}$ is the hidden state at layer $\ell$ and $m_t$ is the attention mask. This makes it possible to compare token-level structure with a single representation for the whole sentence.

The analysis spans eight languages and compares the embedding output with transformer layers 6, 12, 18, and 24. Cosine silhouette in the original hidden space supplies the quantitative check: positive values indicate that observations are, on average, closer to the same language than to other languages. The projections show where clusters appear. Silhouette tests whether that visual impression survives outside two dimensions.

PCA and t-SNE project these high-dimensional vectors into two dimensions. PCA summarizes broad linear variation. t-SNE emphasizes local neighborhoods, but distances and apparent cluster sizes should not be over-interpreted.^[03]03Wattenberg, Viégas, and Johnson (2016), “How to Use t-SNE Effectively,” Distill. The controls below switch between token and sentence states, selected layers, base and fully adapted checkpoints, and both projection methods.

Sentence representations follow a consistent depth pattern: language separation is visible at the embedding output, weakens through the middle layers, and returns near the top. After full fine-tuning, the sentence-level cosine silhouette at layer 24 rises from 0.314 to 0.391. Token states move differently: their final-layer silhouette falls from 0.204 to 0.063.

The distinction matters. Adaptation makes upper-layer sentence means more separable by language in this run, but it does not make every representation more language-specific. Projection geometry also does not measure language quality by itself, so the final stage returns to held-out language-model loss.

Adapting XGLM to Quechua

FLORES is held back for evaluation and never trained on. Adaptation instead uses the Quechua side of a Spanish–Quechua parallel corpus, with duplicates removed and held-out collisions excluded from training.^[04]04hackathon-pln-es, “Spanish-to-Quechua,” Hugging Face Datasets. Checkpoints are selected by adaptation-validation loss. FLORES devtest is reserved for final target and control-language evaluation.

Four update strategies were implemented and compared at seed 42:

• Full fine-tuning updates all 564,463,616 parameters.
• BitFit freezes the weight matrices and trains only bias parameters, leaving 271,360 trainable values.^[05]05Ben Zaken, Goldberg, and Ravfogel (2022), “BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models,” ACL 2022.
• LoRA adds a trainable low-rank residual to frozen attention projections and tests ranks 1, 2, 4, and 8.^[06]06Hu et al. (2021), “LoRA: Low-Rank Adaptation of Large Language Models.”
• IA³ learns multiplicative vectors for attention key/value outputs and the feed-forward activation, leaving 147,456 trainable values.^[07]07Liu et al. (2022), “Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning,” NeurIPS 2022.

The PEFT layers were written directly in PyTorch rather than imported from an adapter library. The central LoRA update adds a low-rank residual, $W' = W + \frac{\alpha}{r}BA$, while keeping $W$ frozen; similar wrappers apply IA³ scales before or after selected linear maps. Regression checks cover trainable parameters, gradients, adapter-state loading, and forward equivalence at initialization.

Training uses 2,000 de-duplicated Quechua strings, AdamW, mixed precision, and gradient checkpointing. Seven configurations are evaluated: full fine-tuning, BitFit, IA³, and four LoRA ranks. Every run trains one configuration and restores the checkpoint with the lowest adaptation-validation loss before final evaluation on 12,558 adaptation-test strings and 1,012 FLORES devtest sentences per language.

Results · performance, efficiency, and retention

Every adaptation method improves both the held-out adaptation corpus and Quechua FLORES relative to the base checkpoint. Their ordering depends on the objective. Full fine-tuning gives the lowest adaptation-test perplexity (32.4), while BitFit gives the lowest Quechua FLORES perplexity (301.9). Better fit to the adaptation corpus therefore does not guarantee the best transfer to independently sourced Quechua text.

BitFit reaches its target result while training 0.048% of the model, but it also produces the largest average degradation across the seven control languages. IA³ trains only 0.026% and is the only configuration with a slightly improved mean control-language BPB (−0.028), although its Quechua perplexity is higher at 363.3.

LoRA rank has little effect in this campaign. Increasing the rank from 1 to 8 multiplies the trainable state by eight, yet Quechua perplexity changes only from 303.6 to 302.6. Rank 1 captures nearly all of the measured LoRA gain at the smallest LoRA checkpoint size.

These measurements represent three separate objectives. Adaptation-test perplexity measures fit to the selected corpus, Quechua FLORES perplexity measures transfer to independently sourced target text, and control-language ΔBPB measures forgetting. Selecting a method from only one axis would hide a different cost.

The retention view completes the comparison. Every method improves the Quechua column. Full fine-tuning degrades all seven controls, BitFit is strongest on target transfer but degrades several controls substantially, and IA³ keeps every control-language change within 0.10 BPB. No method dominates all objectives: full tuning fits the adaptation domain best, BitFit leads on Quechua FLORES, IA³ retains the multilingual baseline most effectively, and rank-1 LoRA provides a compact compromise between target transfer and control-language retention.

Limitations

The campaign uses one seed, so the differences are descriptive rather than confidence-supported rankings. Full fine-tuning ran on an A100 while PEFT runs used GTX TITAN X GPUs, preventing a fair runtime comparison. The adaptation corpus may differ from FLORES in domain, and its generic Quechua label does not establish alignment with the benchmark’s Ayacucho variety. Likelihood metrics do not substitute for generation or human evaluation. PCA, t-SNE, and silhouette describe representation geometry, not linguistic competence.

Conclusion

The study connects three levels of model analysis: observable loss, hidden-state geometry, and controlled parameter updates. Ayacucho Quechua begins as XGLM’s weakest evaluated language, yet every adaptation method closes part of that gap. The most useful result is the trade-off itself: BitFit maximizes observed target transfer, IA³ best preserves the other languages, rank-1 LoRA captures nearly all measured LoRA benefit, and full tuning specializes most strongly to the adaptation corpus.