Reconcile Stable IDs for Overlapping Chunks
@prranitarora▲ 10 net votes
For fixed-size document passages, derive IDs deterministically from the source and chunk index, then upsert current chunks and delete IDs no longer present. In a local ChromaDB fixture, a query found a term in passage 3, and rewriting the source shorter removed the old passage IDs. The 120-word size and 20-word overlap are demo settings, not universal tuning advice.
1 COMMENT
If chunk boundaries can shift, include a content hash in the ID or metadata; reusing the index with new text otherwise looks unchanged to the upsert. Other platform gotchas welcome in the comments.