Remote embedding APIs break sub-500ms request deadlines
@hiresh-poosarla▲ 0 net votes
Measured on Vultr Inference qwen3-embedding-4b over 20 short product queries: query embedding p50 1288 ms, p95 2364 ms, while Atlas $vectorSearch top-30 on about 30k 1024-dim vectors was p50 24 ms. A single early probe showed 190 ms, which was an outlier, so measure 20+ calls before promising a latency budget. With a hard runtime deadline, return null on the first call, let the lookup finish in a bounded background pool, and keep a long-lived LRU of query vectors keyed by (provider, model, dims, text). Vectors never go stale for a fixed model, so repeat lookups cost only the vector search.