How LLMs work: tokens, context and variation

Models do not read whole words. They read tokens: chunks that can be a word, part of a word, a character or bytes. Your input is split into tokens, and the model generates its reply token by token.

Tokens are also how you are billed and limited on APIs. A long prompt with a pasted transcript costs more and runs slower than a short one. Different models split text differently, and Anthropic notes its newer tokenizer produces noticeably more tokens for the same text than older models did, so cost estimates need re-checking when you switch models.

Practical habit: before you automate something that runs thousands of times, measure the token count of a typical input with your provider's token-counting tool rather than guessing.

The context window is everything the model can consider for one response: your system prompt, the conversation so far, any pasted documents and the reply itself. Anthropic describes it as working memory, separate from the data the model was trained on.

In a chat, every message and reply accumulates in the window. When a conversation gets very long, chat apps may drop or summarise older turns, which is why a model can seem to forget an instruction you gave at the start.

Bigger is not automatically better. Anthropic's docs note that as token count grows, accuracy and recall degrade, which they call context rot. Send the model what it needs for the task, not everything you have.

Each time a model answers, it is running inference: using its already-trained weights to produce output. Your chat does not retrain the model. If it seems to learn your preferences, that is the app saving notes and adding them back into the context.

Because training data stops at a point in time, every model has a knowledge cutoff. Ask about something after that date and it will either say it does not know, use a search tool if the app provides one, or produce a plausible guess. For anything time-sensitive, such as prices, product versions or regulations, give it the current source or check the answer yourself.

At each step the model scores many possible next tokens and samples one of the likely options rather than always taking the single top choice. Run the same prompt twice and the wording, and sometimes the substance, can differ.

Some APIs expose sampling settings such as temperature to make output more or less varied. On the current Claude API version those sampling parameters have been removed, and you steer behaviour with clearer instructions, examples and a fixed output format instead.

Design for variation rather than fighting it. If a downstream step needs exact values, ask for a fixed format and validate it. If you need consistency across runs, give examples of good output. Never build a workflow that only works when the model phrases things one exact way.