Transparent huge pages can make 'exact-size' buffer fixes look like a CPU regression
Seen while shrinking an encoder's workspace from oversized to exactly-sized buffers: memory dropped as expected, but CPU rose a reproducible ~3% (ranges not overlapping), with byte-identical output and no extra work.
Cause, measured: the host had transparent huge pages (THP) set to 'always'. The old oversized buffers were large and aligned enough to be backed by 2 MiB pages (AnonHugePages ~72 MB mid-run); the exact-sized ones mostly weren't (~8 MB). The CPU difference was TLB/page-size behaviour, not the algorithm.
Tips: - Check /sys/kernel/mm/transparent_hugepage/enabled and AnonHugePages in /proc/<pid>/smaps_rollup before blaming a memory-reduction change for a CPU loss. - THP mode differs between hosts ('always' vs 'madvise' vs 'never'), so A/B under each mode you expect users to have, on a quiet host; a loaded shared host adds its own noise. - Options if large buffers benefit from huge pages: round large (>=1 MiB) allocations to 2 MiB multiples, or call madvise(MADV_HUGEPAGE) on them explicitly, then re-measure both memory and CPU. - Geometric growth vs exact growth may not matter at all once the page-size effect is accounted for.
Follow-up from re-running the same A/B on quiet dedicated VMs (n=5, rotated arms, pinned CPU), once with THP=always and once with THP=madvise: the exact-size version showed NO claimable CPU loss in either mode. The ~3% seen earlier was on a shared host at load 5-13, so the THP explanation was only part of it, and host noise was the rest. Rounding buffers of >=1 MiB up to whole 2 MiB units (+1) was lower on 5/5 reps in both modes (about 1-2%) and had the smallest maxrss. Exact sizes plus madvise(MADV_HUGEPAGE) on the page-aligned interior gave nothing extra. A 2 MiB-aligned allocator (posix_memalign) kept CPU at parity but RAISED maxrss about 25% under THP=always, because huge pages fill whole 2 MiB regions. Lesson: check the mechanism on a quiet host in each THP mode before acting on it.