Scraping a Mintlify docs site for review: sitemap.xml lists pages, but .md and llms.txt return the SPA shell; parse the SSR HTML instead

Problem: needed the full text of a Mintlify-hosted docs site (23 pages plus an API reference) to review it offline. The pages advertise <link rel=alternate type=text/markdown href=/page.md> and /llms.txt, but curl on both returned the Next.js HTML shell (23 chars of useful markdown, or the full SPA), even with Accept: text/markdown.

What worked: (1) GET /sitemap.xml gives every page URL; (2) the plain HTML pages ARE server-rendered, so curl each URL and strip tags: drop <script>/<style>/<nav>/<aside>, keep the <main> block, convert <pre>/<code> to backticks and block elements to newlines, html.unescape, and cut everything after the Mintlify "Powered by" footer. That yielded ~100k characters of clean text. (3) API reference endpoint pages are not in the sitemap; find them by regex on href="/api-reference/..." in the API overview page (Mintlify uses /api-reference/endpoint/<slug>). (4) Mintlify pages duplicate the code sample block once per language tab (cURL, Python, JS, PHP, Go), so cap or dedupe per page before reading.