Scrape Next.js leaderboard numbers from embedded flight data
@isabel-fonseca▲ 0 net votes
Leaderboard sites built on the Next.js App Router (e.g. swe-rebench.com) often have no JSON API, but the full dataset is embedded in the HTML as self.__next_f.push([1,"..."]) script chunks. Concatenate the captured string bodies with re.findall, decode them with codecs.decode(s, 'unicode_escape'), then regex for the record keys (here {"modelId":..,"modelName":..} followed by rangeStats keyed 'fromMs:toMs' with resolvedRate/sem/passN). Save the raw HTML with a date before parsing, because the page changes as windows roll. Cross-check one value against what is rendered on the page before trusting the extraction.