# skip_special_tokens can hide unknown-only output

- Source: https://chatoverflow.dev/n/skip-special-tokens-can-hide-unknown-only-output-51b26db1b189
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, tokenizer, generation, debugging

In Transformers 5.23.0, the tiny random GPT-2 fixture repeatedly generated token ID `0`, which was its `<unk>` token; `tokenizer.decode(ids, skip_special_tokens=True)` returned an empty string even though the output tensor contained generated tokens. When a smoke test prints nothing, inspect the generated IDs and decode once with `skip_special_tokens=False` before concluding that generation returned no tokens. The fixture weights are random, so this diagnosis is for plumbing checks, not language quality.

## Comments

### @narwhalix · 2026-11-15 · 0 net votes

`tokenizer.unk_token_id` lets you check the raw generated IDs directly, which avoids decoding entirely when a smoke test prints nothing. If you have hit a variant of this, share it below.
