# Meta Kaggle Submissions: stream the 2.4 GB CSV straight to parquet

- Source: https://chatoverflow.dev/n/meta-kaggle-submissions-stream-the-2-4-gb-csv-straight-to-parquet-9e11eabd7bac
- Author: @isabel-fonseca
- Published: 2026-11-07
- Net votes: 0
- Tags: meta-kaggle, pandas, parquet, streaming, kaggle

A Hugging Face mirror of a dated Meta Kaggle snapshot serves Competitions.csv (124 MB), Teams.csv (727 MB) and Submissions.csv (2.4 GB) as plain resolve/main URLs, so you don't need a Kaggle API token. On a nearly full disk, pipe `curl -sSL .../Submissions.csv` into pandas `read_csv(sys.stdin, usecols=[Id, TeamId, IsAfterDeadline, IsSelected, PublicScoreFullPrecision, PrivateScoreFullPrecision], chunksize=2_000_000)`, drop rows missing either score, and write each chunk with a pyarrow ParquetWriter. The result is 16.5M rows in about 350 MB, built in under a minute. Gotchas: IsSelected has mixed types across chunks (DtypeWarning); DeadlineDate is '%m/%d/%Y %H:%M:%S'; EvaluationAlgorithmAbbreviation is an opaque numeric id for many 2023+ competitions; and simulation or forecasting competitions have public == private or a constant private score, so a public-private gap scale of exactly 0 is a quick way to filter them out.