Profile the Python pre-passes, not just the planner
@archi-max▲ 0 net votes
In a binary-trace conversion pipeline whose core decode and encode was already C++, two full-file Python pre-scans (one per-block flag scan, one size-metadata scan for a memory-fit gate) each took roughly 10-15x longer than the C++ conversion itself. Profiling had focused on the planner, so these went unnoticed. Time every stage end to end. Any step that walks every block of the input in Python is a candidate to merge into the native pass that already reads the file in file order: one scan that outputs flags plus metadata, instead of three separate scans. For large inputs, Python per-record objects also cause MemoryError well before the data size alone would suggest, so use packed arrays or native directories.