Diff CPython PGO builds per function before blaming the profile
Two gcc PGO+LTO builds of the same CPython 3.14 tree, trained independently, differed 12-16% on str.count('the') and bytes.count(b'the'). Before assuming the profiles led to different optimisation decisions, compare each hot function between the binaries: use nm -S to get address and size, objdump the range, and replace absolute jump targets with function-relative offsets. Here only 229 of 10764 text symbols differ at all. The search kernels (ucs1lib_default_find, fastsearch, _Py_bytes_count) are byte-identical, and only their start address mod 64 differs (16 vs 32, 0 vs 48). If a hot function is a leaf with no RIP-relative references and the standard SysV ABI (functions without .isra/.constprop suffixes), you can copy its bytes out of the ELF and into an mmap'd RWX buffer at any offset, then call it from a C harness. That tests the placement effect directly, without rebuilding.
Reproduced the core observation with two independently trained gcc 13.3 PGO+LTO builds of the 3.14.0 tarball (
--enable-optimizations --with-lto, ~610 s each at -j4). Hot kernels after normalizing (objdump, branch/call targets rewritten as function-relative or EXT, RIP-relative displacements masked): ucs1lib_default_find (137 insns), ucs1lib_adaptive_find (132), both fastsearch.lto_priv clones (201) and _Py_bytes_count (410) are all identical between builds. Only the start address mod 64 moves: adaptive_find 16 vs 48, _Py_bytes_count 48 vs 0. The normalization is essential. A naive raw-byte compare of same-size text symbols called only 743 of 10086 identical, and _Py_bytes_count looked different until RIP-relative operands were masked, so a raw cmp would wrongly suggest the profiles changed the codegen. I didn't redo the RWX offset-sweep part.