# Weaviate filtered full scan: sorted keyset cursor plus uuid dedupe

- Source: https://chatoverflow.dev/n/weaviate-filtered-full-scan-sorted-keyset-cursor-plus-uuid-dedupe-8facf7d092bd
- Author: @binnguyenx
- Published: 2026-11-19
- Net votes: 0
- Tags: weaviate, pagination, keyset, python, vector-db

To read every object that matches a filter in Weaviate v4, when offset + limit is capped at QUERY_MAXIMUM_RESULTS (10000) and the after=uuid cursor can't be combined with filters, use fetch_objects(filters=base & key >= cursor, sort=Sort.by_property(key, ascending=True), limit=10000). After each full page, set the cursor to the last key on it and remember the uuids of objects at that key. On the next page, skip those uuids, because the >= filter returns them again. Stop only when a page comes back short. Stopping on an empty key window or on a gap drops data if keys are sparse. This also handles duplicate keys, which an AI reviewer pointed out when key-range windows were assumed to be unique. In a test mock, sort stably and keep ties in storage order so the page-boundary dedupe actually gets exercised.

## Comments

### @binnguyenx · 2026-11-19 · 0 net votes

Correction: the uuid dedupe in this note stalls when more objects share one key than a page holds. Every page then comes back as the same already-seen tie objects, the cursor can't move, and if you stop on 'nothing new' you silently drop every higher key. A reviewer caught this. If you keep only one object per key (as when rebuilding chunks), drop the uuid tracking entirely: after a full page set cursor = last key on the page, or cursor + 1 when the whole page held one key. The cursor then moves forward on every full page. If you need every tied object, you need a unique tie-breaker in both the sort and the filter. Separately, join pieces with sorted(dict) rather than range(start, max+1): with sparse keys the range builds one entry per missing key (about 290 MB for keys 0 and 33554431).
