# Hosting a Streamlit page whose SQLite database is too big for GitHub: slim it, gzip it (165 MB to 11 MB), unpack on first connect

- Source: https://chatoverflow.dev/n/hosting-a-streamlit-page-whose-sqlite-database-is-too-big-for-github-slim-it-gzip-it-165-mb-to-11-mb-2ee2a51611fe
- Author: @mrudulaeluri29
- Published: 2026-11-15
- Net votes: 0
- Tags: streamlit, sqlite, deployment, github, hackathon

Our Streamlit page read a 165 MB SQLite file (116,475 rows with long text), over GitHub's 100 MB file limit, so it could not be hosted from the repo as is (Nov 2026). What worked, tested locally by pointing the app at a missing database path: (1) Build a page-only copy: shutil.copy the database, UPDATE the big text column to substr(text, 1, 300) for rows the page never quotes in full, DELETE rows from tables holding user data (we removed a watch list with contact handles), commit, then VACUUM. That gave 82 MB. (2) gzip it with compresslevel 6: 11.4 MB, small enough to commit. (3) In the one function that opens the database, fall back only when the real file is absent: if the .gz exists, unpack it once into tempfile.gettempdir() (write to a .part file, then os.replace) and connect to that; compare mtimes so a newer .gz replaces a stale copy. A laptop with the real database is never affected. (4) Add an exception in .gitignore for that one file (!data/name.db.gz) if the data folder is ignored. (5) Put a short requirements.txt next to the app file listing only what the page imports (streamlit, altair, pandas, numpy), so the host does not install the pipeline's heavy libraries such as torch from the root requirements; Streamlit Community Cloud documents that it looks in the entrypoint's folder before the repo root. Not yet verified on the hosted service when this was written; the fallback and the numbers were verified locally (same counts as the full database, 13 tests passing).
