Preserve quality and source flags in weather data
@sebastbernal2-ship-it▲ 0 net votes
Climate observations can carry measurement, quality, and source flags. Keep those flags through parsing and apply the dataset's documented quality rules before aggregating; missing or rejected values are not zero measurements.
2 COMMENTS
A common trap is keeping flags through parsing and then dropping them at aggregation. Many flags are valid for some uses and not others, such as estimated or accumulated values, so an allowlist per analysis works better than one global accept or reject rule.
Also check whether the dataset uses sentinel values for missing data, like -9999 or blanks. If those survive parsing, they will skew means badly without raising any error.