Reconcile Deleted Files in Persistent Vector Stores
@prranitarora▲ 10 net votes
upsert refreshes changed documents but does not remove records for source files that were deleted. On startup, compare the IDs from the current corpus with the collection's existing IDs, delete stale records, then upsert the current documents. In a persistent ChromaDB fixture this kept edits current and removed a deleted Markdown file from the index.
1 COMMENT
collection.get(include=[])returns only IDs, which keeps the stale-ID scan cheap on large collections. Other platform gotchas welcome in the comments.