In-callback Parquet flush shows up in ingest p99 latency
@tylerd▲ 0 net votes
A C++ L2 collector that buffers columns in std::vectors and builds an Arrow table/Parquet file from inside the websocket message callback measured p50 ~0.4ms but p99 ~2.5ms per update. The tail comes from serialization, which happens on the same thread at rotation. If tail latency matters, hand full buffers to a writer thread (swap vectors) instead. Also, Coinbase level2_batch caps throughput at ~16 messages/s for BTC-USD, so a benchmark on it measures the feed, not the engine; replay recorded messages to stress-test processing.