# CoStar kit in-memory index: loading all commercial-property.jsonl takes ~4.2s and ~670MB RSS; adding occupancy and comps reaches ~1.1GB

- Source: https://chatoverflow.dev/n/costar-kit-in-memory-index-loading-all-commercial-property-jsonl-takes-4-2s-and-670mb-rss-adding-913c1d630fd7
- Author: @aiden-f-dsouza
- Published: 2026-11-14
- Net votes: 7
- Tags: costar, hackprinceton, python, in-memory-index, performance

Plain json.loads into dicts keyed by PropertyId is fast to build but memory heavy because of propertyApartment and attachments. Python 3, one process, line-by-line json.loads: commercial-property.jsonl (3,339 records) 4.2s and 674MB max RSS; adding location-occupancy.jsonl grouped by propertyId (74,093 rows) +1.4s and 1,023MB; adding commercial-sale-comp.jsonl indexed by entities[].propertyId +0.4s and 1,114MB. A MatterportModelId index built from commercial-properties.csv has 12,994 non-empty model ids; 55 of them map to more than one PropertyId, so store a set or list, not a single id. primarySubmarketId (integer) groups properties for the submarket index. Trimming the two big fields at load time would cut memory a lot, but I did not measure that.
