Top-500 PyPI READMEs: 1/3 of examples run; only ~5% are actually wrong
Measured by running every Python code block from the PyPI README (info.description) of the top-500 packages, in order in one REPL-like namespace, in a fresh venv with only pip install pkg==version (Python 3.12, no credentials). 274 packages have Python examples; only 32% have all blocks run. Most failures are by design: 27% stop on fragments (undefined names, ... placeholders, missing imports), 24% need credentials/servers/files/CLI args, 11% need an extra. Only 14 packages (5.1%) ship an example that is wrong for that release (5 stale-API blocks, 19 bugs such as missing commas or kw: value instead of kw=value), double-rated blind with kappa 0.92. Of 62 READMEs with REPL output, 11 document output that differs today. Rerunning on Python 3.14 changes almost nothing. If you hand-wave about 'stale docs', the bigger problem is non-self-contained examples, not rot.
Update after extending to the top 1000 (540 packages with examples, 3,750 blocks): 29% all run (46% in the top 100); 6.5% ship a wrong example (13 stale-API + 37 bug blocks, blind second rating kappa 0.82 on the new half). One rule refinement worth copying: ModuleNotFoundError is not always a missing extra. Names like my_module/myapp/your_package are the reader's own code (fragment), and a missing *own* submodule (pkg.sub) can be a stale import path. But check by hand: sagemaker's v3 README shows v2 code on purpose as a migration 'before'.