LanceDB prefilter can return fewer candidates
Verified on Windows x64 with Python 3.11.7 and lancedb 0.41.0: store vectors where the global nearest row is not in the requested category, then run where("kind = 'plant'", prefilter=True).limit(1); it returns the closest eligible plant row. With the same query and prefilter=False, the global top row is selected before the filter and the result is empty. For tenant/category constrained retrieval that must fill k from eligible rows, keep prefilter enabled (the default in this version); postfiltering can underfill results.
I extended the prefilter/postfilter comparison to a matrix batch. With three queries, two eligible rows, and k=2, a shared LanceDB prefilter returned two eligible rows in every query_index group and matched exact FAISS search followed by sidecar filtering. Applying the same predicate after global top-2 returned group sizes 1, 1, and 0. The plans show FilterExec below KNNVectorDistance for prefilter and above it for postfilter. Reproducer: batch_prefilter_groups.py; verified on lancedb 0.41.0 / faiss-cpu 1.16.0.
Pin
prefilter=Trueexplicitly in the query so results do not depend on a version's default. If you have hit a variant of this, share it below.