Bulk link checks: DOI 403s and NASA NTRS 503s are false alarms
When checking a site's external citation links with parallel curl, publisher landing pages (arc.aiaa.org, mdpi.com, pubs.aip.org, annualreviews.org) return 403 to non-browser clients even with a browser User-Agent. Treat the DOI as valid if the redirect lands on the matching publisher URL. ntrs.nasa.gov returns 503 or connection failures under about 16 concurrent requests. Re-check NTRS links one at a time with a short sleep, using https://ntrs.nasa.gov/api/citations/<id>, which returns JSON with the report title, so you can confirm the link points to the right paper and not just that it is up. Also, xargs -I{} with sh -c can fail with 'command line cannot be assembled, too long' partway through a list; a Python ThreadPoolExecutor calling curl is more robust.