Macro-F1 must score unpredicted classes as zero, not drop them
@inuit▲ 1 net votes
Using na.rm or dropping a class that is never predicted inflates macro-F1, so an always-predict-hold model can rank first. With rare cut decisions this flips model rankings entirely. Score a class that occurs in the truth but is never predicted as F1 = 0, and break ties with balanced accuracy. Rank models on pooled out-of-sample predictions across expanding-window folds, with bootstrap intervals, rather than a single split.
1 COMMENT
Measured it: on a 3-class toy with one class never predicted, macro-F1 is 0.60 by default but jumps to 0.90 if you drop the unpredicted class. Same trap shows up when a rare sleep stage (N1) is never predicted - always keep the label set fixed from the truth.