DuckDB iceberg_scan on a pyiceberg SqlCatalog table: 'IO Error: Cannot open file "file:///.../metadata/snap-...avro": No such file or directory' (duckdb 1.1.3; fixed in newer DuckDB)
Fix: upgrade DuckDB (2.0.0 reads it fine), or create the pyiceberg catalog with a plain path warehouse (warehouse="/data/wh") instead of warehouse="file:///data/wh".
Symptom: you write an Iceberg table with pyiceberg 0.8.1 SqlCatalog(..., warehouse="file:///tmp/wh"), then query it with DuckDB 1.1.3 and the iceberg extension:
INSTALL iceberg; LOAD iceberg;
SELECT count(*) FROM iceberg_scan('/tmp/wh/ns.db/models/metadata/00001-<uuid>.metadata.json');
-- IO Error: Cannot open file "file:///tmp/wh/ns.db/models/metadata/snap-...avro": No such file or directoryThe metadata.json itself opens. The failure is on the manifest list. pyiceberg writes every path inside metadata/manifests as a file:// URI (tbl.metadata_location is file:///tmp/wh/...), and DuckDB 1.1.3's local filesystem treats file:///tmp/... as a literal relative filename. Passing the file:// URI directly to iceberg_scan gives the same error on the metadata file.
What I checked (python:3.11-slim, pyiceberg 0.8.1):
- duckdb 1.1.3 + warehouse="file:///tmp/whA": fails as above.
- duckdb 1.1.3 + warehouse="/tmp/whB": metadata_location has no scheme, iceberg_scan('<metadata.json>') returns the rows.
- duckdb 2.0.0 (pip install -U duckdb): both warehouses work, with either the plain path or the file:// URI.
allow_moved_paths=true doesn't help here; in 1.1.3 it errors 'Enabling allow_moved_paths is not enabled for directly scanning metadata files.'
Verify: print(tbl.metadata_location). If it starts with file://, an older DuckDB will choke on it.