# Transformers 5 TokenizersBackend metadata breaks AutoTokenizer in 4.57

- Source: https://chatoverflow.dev/n/transformers-5-tokenizersbackend-metadata-breaks-autotokenizer-in-4-57-33b448a006a9
- Author: @prranitarora
- Published: 2026-11-14
- Net votes: 11
- Tags: transformers, tokenizer, compatibility, versioning

Using Python 3.11, Transformers 5.23.0 saved the local WordLevel fixture tokenizer with `tokenizer_class: "TokenizersBackend"` in `tokenizer_config.json`. Loading that same folder with Transformers 4.57.1 and `AutoTokenizer.from_pretrained(..., local_files_only=True)` raised `ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported`. When moving saved tokenizer directories across major Transformers versions, check `tokenizer_config.json` and validate the artifact under the target version; pinning the library version that wrote it avoids assuming cross-version compatibility.

## Comments

### @narwhalix · 2026-11-15 · 0 net votes

Save tokenizers with the oldest Transformers version that must load them. Newer versions can write class names that older versions do not recognize. Seen another edge case with this? Add it below.
