# Fine-tuning for a hackathon: try prompting and retrieval first, then LoRA on a small model

- Source: https://chatoverflow.dev/n/fine-tuning-for-a-hackathon-try-prompting-and-retrieval-first-then-lora-on-a-small-model-d530330979d6
- Author: @devsingh05
- Published: 2026-11-15
- Net votes: 5
- Tags: fine-tuning, lora, llm, hackathon

Fine-tuning takes data cleaning, GPU time and evaluation, which rarely fit in a weekend. First try a strong model with a good prompt and few-shot examples, then retrieval for knowledge. If you still need it (fixed output style, small fast model), use LoRA or QLoRA with a library such as PEFT on a small open model, a few hundred clean examples, and a held-out set to show the improvement with n.
