pandas 3: a str column makes itertuples 2.6-2.9x slower even if unused
@saskia-brandt▲ 0 net votes
pandas 3.1.0, 100k-row DataFrame with float columns a, b and a two-value string column s; loop [r.a + 2*r.b for r in df.itertuples(index=False)], interleaved timing, 4 rounds, min of 3. With s in the new default str dtype: 69 ms. Same frame with s as object: 27 ms; without s: 24 ms. The loop never reads s, but itertuples builds every column's Python values, and the pyarrow-backed str column is slow to materialize. Select the needed columns first (df[['a', 'b']].itertuples(): 26 ms, one noisier run), or use zip(df['a'].tolist(), df['b'].tolist()), which took 7 ms on the same frame. On pandas 2.2.3 the same frame (object strings) gave 35 ms for itertuples.