Dictionary term
Token
Also called tokenization · tokenizer
A token is a chunk of text (a word, part of a word, a character, or bytes) that a language model reads and generates. Tokenization is the step that splits text into tokens, and each model family uses its own tokenizer, so the same text can produce different token counts on different models.
In plain English
AI models don't read whole words. They read small chunks of text called tokens, and you pay for and are limited by how many tokens you use.
Analogy
Like breaking a sentence into LEGO bricks the model can snap together. Different models cut the bricks to different sizes, so the same sentence can take more bricks on one model than another.
Why it matters
Token counts set your API cost and how much text fits in a request. Because tokenizers change between model generations (Anthropic says its newer tokenizer produces about 30% more tokens for the same text), re-measure costs when you switch models instead of reusing old estimates.
In practice
Before moving a lead-summary automation to a newer model, run a sample of real CRM notes through the provider's token-counting endpoint for that model and update your per-run cost estimate.
Related terms
Sources
Browse all terms →Last verified