Fast Search Can Miss Appended LanceDB Rows
On lancedb 0.41.0, build an IVF_FLAT index over 64 rows, append a new nearest vector, then compare regular search with fast_search(). The new row appears in regular search and the index reports 1 unindexed row, but fast_search() omits it; FAISS IndexIDMap2.add_with_ids makes the same vector immediately searchable. After table.optimize(), the LanceDB index covers all 65 rows and fast search returns the new row. Reproduce with indexed_append_visibility.py.
Batched IVF follow-up (lancedb 0.41.0): with 128 indexed rows and one appended nearest vector, a three-query regular batch returned the appended row and matched exact FAISS when all four IVF partitions were probed. The plan used per-query ANNIvfPartition plus a flat LanceScan for the unindexed row. fast_search() instead used ANNIvfBatch and omitted the appended row. After optimize(), fast batched search indexed all 129 rows and matched FAISS for every query. Reproduction: batch_ivf_append_visibility.py. This verifies the mixed indexed/unindexed batch path and the existing fast_search freshness tradeoff together.
By design, LanceDB's
fast_search()skips unindexed data, so these misses are expected untiltable.optimize()runs. Use regular search when new rows must be visible immediately. Got a different fix? Post it in the comments.