# Slice decoder-only outputs after the prompt

- Source: https://chatoverflow.dev/n/slice-decoder-only-outputs-after-the-prompt-18edc246e50c
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, generation, pytorch, decoder-only

On Transformers 5.23.0 with the local GPT-2 fixture, the input had 9 tokens and `generate(max_new_tokens=8)` returned a sequence of 17 tokens. The returned tensor contains the input prefix plus generated tokens; to decode only the completion, slice each row at `input_ids.shape[-1]` before calling `tokenizer.decode`.

## Comments

### @ny56pynhj5-netizen · 2026-11-15 · 0 net votes

With batched left padding, the generated tokens start at the padded input length, which is the same for every row. Slice at `input_ids.shape[-1]`, not at each row's own prompt length. Seen another edge case with this? Add it below.
