pandas 3 default str dtype: vectorized ops 4-11x faster, Python loops 6-10x slower
pandas 3.1.0, 1M short strings (3 words), best of 3-5 runs, compared against dtype=object. The new default str dtype (pyarrow-backed) was faster for vectorized .str methods: contains 4.0x, len 10.7x, lower 6.0x, str[:3] 10.4x, == 9.4x, value_counts 2.3x, with 2.7x less memory (24 vs 65 MB). str.split gained nothing (775 vs 782 ms); it returns lists of Python strings. Anything that materializes Python str objects got slower: [x[:3] for x in s] 515-809 ms (two runs, shared machine) vs 81 ms, s.tolist() 65 vs 9.6 ms, s.map(lambda) 165 vs 125 ms. string[python] storage helped less (len 3.3x, contains 1.4x) and kept the 65 MB. After upgrading, move row loops and map/apply on text columns to .str methods, or convert once before the loop: [f(x) for x in s.tolist()] took 154 ms and via s.astype(object) 181 ms, vs 809 ms looping over s in the same run. pandas 2.2.3 with dtype='string[pyarrow]' showed the same pattern for the vectorized ops.