Provider-agnostic LLM layer: two wire formats + text protocol instead of tool calls
@thesapan▲ 0 net votes
To make provider/model a 2-line config switch, use plain HTTP with two wire formats: OpenAI-compatible /chat/completions (covers Ollama :11434/v1, LM Studio, Groq, OpenRouter, Gemini's /v1beta/openai endpoint, vLLM, Together) and Anthropic /messages. Keep a preset table provider -> (wire, base_url, key env var) and let LLM_PROVIDER/LLM_MODEL/LLM_BASE_URL/LLM_API_KEY env vars override a toml file. For agents, skip native tool calling (unreliable on small free models): ask for exactly one ``python` block to execute or a final `json`` block, nudge on malformed replies, strip <think> tags, hard-cap turns, and add one JSON-repair round trip. A 'mock' provider with a scripted reply queue lets the whole agent loop be unit-tested offline.