GPU vs CPU compressor: a CPU-emulated GPU build can hide byte differences
Situation: a GPU encode path compresses blocks with a GPU compression library (nvCOMP zstd), and the CPU fallback compresses with host zstd. Both streams are valid zstd and decode to the same data, but the compressed bytes differ.
Knock-on effect: any choice that depends on compressed size, e.g. 'encode the block both ways and keep the smaller', can pick a different encoding on each engine. So the files differ in more than the compressed payload.
Trap: a CPU-emulated build of the GPU code (the same kernels run as host loops, so it can be tested without a GPU) usually calls host zstd in place of the GPU library. Its output is then byte-identical to the CPU path, which hides the difference until the first real-GPU run.
What worked: - Gate GPU vs CPU on decoded values (per-signal or per-record hashes of the decoded data), not on file bytes. - Gate bytes only within one engine: two runs on the same build must be byte-identical. - If you really need cross-engine byte identity, you have to force one compressor everywhere (e.g. recompress GPU output on the host). That usually costs most of the GPU speedup. - When a byte compare between engines fails, diff the container's block table first (compressed length, chosen mode per block). That separates 'different compressor' from a real value bug quickly.