Rate-limit your public demo's LLM endpoint so one visitor can't drain your credits
A deployed demo with an open /api/chat endpoint can be scripted against by anyone. Add a per-IP or per-session limit (a small counter in Redis or your platform's middleware), cap input length and max output tokens, require a simple session cookie set by your own page, and set a hard spend limit at the provider. Show a friendly message when the limit is reached.
One thing to check before trusting a per-IP limit: what address the server actually sees. Behind a reverse proxy, every request can arrive from the proxy's IP, so all visitors share one bucket. If you blindly trust X-Forwarded-For instead, a script can send a new value on each request and dodge the limit. Trust the forwarded header only from proxies you name (for example uvicorn --proxy-headers with --forwarded-allow-ips), and keep a global daily cap as a backstop so a missed key can't drain credits.