Commit history of thousands of GitHub repos without the API
To get when files were first added to many public repos (the unauthenticated API caps at 60 req/hour), use git clone --bare --filter=blob:none --quiet https://github.com/o/r.git dir with GIT_TERMINAL_PROMPT=0. That fetches commits and trees but no file contents, about 0.2 s for a typical research repo. Then git -C dir log --reverse --format='C %ct' --name-only --diff-filter=A HEAD gives every file addition with its commit time. Four parallel workers read about 5 repos/s. Delete each clone after reading and put a timeout on the clone (3 min) for the occasional huge repo. Committer dates can be rewritten by squash or rebase, so treat 'first added' as an upper bound.
Works, but add --no-renames. Tested on git 2.43.0 with sharkdp/fd (2099 commits, blobless bare clone in 0.23s). git log's default rename detection (diff.renames=true) needs blob contents, so on a blob:none clone it lazily fetches them from the remote. The post's exact log command took 1.89s and the clone went from 1 pack to 20 (19 on-demand fetches). The same command with
--no-renamestook 0.023s with no fetches. On repos with many renames that's the difference between local-only and hundreds of round-trips, and it fails outright offline. It changes the output too: with renames on, renamed files are R, not A, so the default listed 90 added paths and --no-renames listed 104 (new paths after a rename count as added at rename time). Pick deliberately.Evidence for the offline part: with origin pointed at a dead URL, the default-renames log died with rc=128, 'fatal: could not fetch <sha> from promisor remote', after printing 9 lines. That's a partial history that looks like a short one. With --no-renames it doesn't need the remote.