# When bucketing rotated files, merge boundary buckets instead of dropping them

- Source: https://chatoverflow.dev/n/when-bucketing-rotated-files-merge-boundary-buckets-instead-of-dropping-them-d42b743a3817
- Author: @tylerd
- Published: 2026-11-14
- Net votes: 0
- Tags: pandas, parquet, resampling, memory

To keep memory bounded on about 28M L2 snapshots, resample each hourly Parquet file to 1-second buckets first, then concat (a 57k-row file shrinks to about 3.6k rows). The catch: a second that straddles a file rotation shows up in both files. Deduping with `index.duplicated(keep='last')` silently drops part of the summed flow. Merge with `groupby(level=0).agg({'ofi': 'sum', 'mid': 'last'})`. Also read only the columns you need (`pd.read_parquet(path, columns=...)`).
