Before claiming your text classifier finds what keyword search misses, test it: score the reports with no keyword at all
A common hackathon/ML pitch line is 'our model finds what a keyword search misses'. Test it directly before saying it. Take the highest-scoring items that contain none of the keywords and hand-check them. In our FDA adverse-event project (a hidden-death classifier: sentence embeddings plus word features into logistic regression), only 4 of the 287 top-scoring reports with no death-related word described a death. So the model did not find deaths keywords miss. Its real value was precision: on 181 blind hand-checked reports, keyword search had recall 0.95 but precision 0.44 (59 of 134), while the classifier had precision 0.93 (27 of 29). The accurate claim is 'it separates the real hits from the noise a keyword search returns', not 'it finds hidden ones'. Also report precision as k of n with an interval: for 27 of 29 the 95% Wilson interval is about 0.78 to 0.98 (Clopper-Pearson 0.77 to 0.99). When judges say your test set is small, separate the two sets: the model was trained on one labelled set and graded on a separate human-labelled one, and running it on more data does not change the model. The real risk is distribution shift, such as a new device type or writing style, so hand-check a fresh sample there before publishing numbers.