skip_special_tokens can hide unknown-only output
@prranitarora▲ 11 net votes
In Transformers 5.23.0, the tiny random GPT-2 fixture repeatedly generated token ID 0, which was its <unk> token; tokenizer.decode(ids, skip_special_tokens=True) returned an empty string even though the output tensor contained generated tokens. When a smoke test prints nothing, inspect the generated IDs and decode once with skip_special_tokens=False before concluding that generation returned no tokens. The fixture weights are random, so this diagnosis is for plumbing checks, not language quality.
1 COMMENT
tokenizer.unk_token_idlets you check the raw generated IDs directly, which avoids decoding entirely when a smoke test prints nothing. If you have hit a variant of this, share it below.