Dagster DynamicPartitionsDefinition: 'DagsterUnknownPartitionError: Could not find a partition with key ...' and get_partition_keys() CheckError 'The instance is not available to load partitions'
Fix: register the keys on the Dagster instance before you materialize or backfill a partition: instance.add_dynamic_partitions("splat_sids", ["1v5XHfUFwbC", ...]). Do it from a sensor (SensorResult(dynamic_partitions_requests=[defn.build_add_request(keys)])), from an upstream asset (context.instance.add_dynamic_partitions(...)), or from a script. The definition itself holds no keys.
Reproduced with dagster 1.9.6:
import dagster as dg
sids = dg.DynamicPartitionsDefinition(name="splat_sids")
@dg.asset(partitions_def=sids)
def photo_meta(context): return context.partition_key
inst = dg.DagsterInstance.ephemeral()
dg.materialize([photo_meta], partition_key="1v5XHfUFwbC", instance=inst)
# DagsterUnknownPartitionError: Could not find a partition with key `1v5XHfUFwbC`.
sids.get_partition_keys()
# CheckError: Failure condition: The instance is not available to load partitions. You may be seeing this error when using dynamic partitions with a version of dagster-webserver ... older than 1.1.18 ...
inst.add_dynamic_partitions("splat_sids", ["1v5XHfUFwbC", "462uN7LmxXi"])
dg.materialize([photo_meta], partition_key="1v5XHfUFwbC", instance=inst) # successThat CheckError hint about old webserver versions is misleading. The real cause is calling get_partition_keys() without dynamic_partitions_store=instance. Use instance.get_dynamic_partitions("splat_sids") to list keys.
Gotchas:
- The keys live in the instance storage. With dagster dev and no DAGSTER_HOME, that storage is a temp dir, so registered partitions vanish on restart and a backfill script run in a separate process (with its own ephemeral instance) won't see them. Point both at the same DAGSTER_HOME.
- An asset that adds partitions only affects later runs; it can't fan out inside the run that registers them.
Verify: instance.get_dynamic_partitions("splat_sids") lists the keys and the UI's partition bar shows them before you launch a backfill.