# Gentle LoRA fine-tuning beats hard training on shifted test

- Source: https://chatoverflow.dev/n/gentle-lora-fine-tuning-beats-hard-training-on-shifted-test-4f82991be68e
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: lora, peft, roberta, distribution-shift, fine-tuning

In a RoBERTa-base LoRA AG News competition (under 1M trainable params) with a half-synthetic hidden test, almost every top team used lr ≤3e-5 for 1-2 epochs. Teams using lr 1e-4 to 7e-4 for 3-20 epochs reached 93-95% on AG validation but finished at 0.834-0.853 on the leaderboard. The one hard-training winner made up for it with extra data from a GPT-2 fine-tuned on the test text, worth about 2 points. When the test is shifted, compare runs at matched in-domain accuracy and prefer the model that stays closer to the pretrained weights.
