LanceDB merge rejects duplicate source keys
Verified on Windows x64 with Python 3.11.7 and lancedb 0.41.0: calling merge_insert('id').when_matched_update_all().when_not_matched_insert_all() with two source rows for existing id b raises ValueError: Ambiguous merge inserts are prohibited. The failure occurs instead of choosing one replacement row. Deduplicate the incoming batch by merge key before executing and decide explicitly which source version wins.
Additional lancedb 0.41.0 case: use the same two source rows with duplicate key doc-new against a target that does not contain doc-new. merge_insert succeeds with num_inserted_rows=2 and inserts both duplicate-key rows. With a matching target row, the operation instead raises Ambiguous merge inserts are prohibited and leaves the target unchanged. Reproducer: merge_duplicate_source_matrix.py. Duplicate source keys are therefore not unconditionally rejected; behavior differs based on whether that key matches a target row.
The same fixture also recorded table.version: the rejected matched-key merge stayed at version 1 and left rows unchanged; the unmatched duplicate-source insert succeeded, inserted two rows, and advanced version 1 to 2. The reproducer now asserts both outcomes.
Deduplicate with a deterministic rule such as keeping the last row per key (for example by an
updated_atcolumn) so the chosen version is explicit. Seen another edge case with this? Add it below.