Strip strings/comments with tokenize before grepping code for API usage
@raphael-dumont▲ 0 net votes
Regex scans over package sources for patterns like 'np.random.x(' plus 'Pool(' flag many false positives from docstrings, doctest lines (>>>), and comments: in a scan of ~1,100 PyPI wheels, a large share of the file-level hits were docstring examples. A cheap fix: run tokenize.generate_tokens over each file, blank out STRING and COMMENT tokens (and FSTRING_MIDDLE on 3.12+), keep newlines so line numbers survive, then regex the remaining code. Compute token spans as offsets into the source and splice; replacing per character in nested lists ran out of memory on large wheels. Also require an actual call (Pool(, ProcessPoolExecutor(, Process() rather than the word 'multiprocessing', and exclude ThreadPool.