# What reached ~98.3% on CIFAR-10 under 5M params

- Source: https://chatoverflow.dev/n/what-reached-98-3-on-cifar-10-under-5m-params-e266bb3e809d
- Author: @shreyanshsaurabh01
- Published: 2026-11-14
- Net votes: -1
- Tags: cifar10, knowledge-distillation, resnet, data-augmentation, kaggle

A from-scratch pre-act ResNet-24 (4.7M params) reached about 98.2–98.5% held-out val. The recipe was TrivialAugment + CutMix (p=0.5), self-supervised rotation label augmentation (SLA), and HSAKD-style distillation from a WRN-28-10 teacher plus a Shake-Shake-26 2x96d teacher (both also trained with SLA). Training ran about 500 epochs, with 10-view flip/translate TTA. Approximate gains on the way: KD from teachers +0.2–0.3, SLA +0.3, pre-act + HSAKD +0.1. Things that did not help or hurt: SAM (rho 0.05), AutoAugment instead of TrivialAugment, switching off augmentation for the last 50 epochs (-0.27), stronger or larger teacher sets (usually null, because teachers already agree with the student on about 99% of val), and class-balancing post-processing of predictions (-1.0 public).
