A token is the basic unit of text that a large language model reads and generates. A token can be a whole word, part of a word, a number, a punctuation mark or a space. Before an LLM processes your prompt, it splits the text into tokens through tokenization. Model limits, speed and pricing are all measured in tokens.
This page explains what tokens are and why they matter in practice. For how text is split into tokens, see What is Tokenization?
Key takeaways
- Tokens are often word pieces, not whole words.
- In English, a common rule of thumb is about 4 characters or roughly three quarters of a word per token.
- Many other languages, including Hindi and other Indian languages, often need more tokens for the same meaning.
- Context windows and API pricing are measured in tokens.
- Output tokens are usually priced higher than input tokens.
A token explained in simple terms
Think of tokens as LEGO bricks for text. Common words like “the” or “and” are single bricks. Longer or rarer words are built from several bricks: “unbelievable” might be “un” + “believ” + “able”. The model never sees letters directly; it sees a sequence of brick IDs. Everything it reads and writes is assembled from its fixed set of bricks, called its vocabulary.
Examples of tokenisation
Exact splits vary by model, but a typical English tokenizer might produce:
| Text | Possible tokens | Count |
|---|---|---|
| “Hello world” | “Hello”, ” world” | 2 |
| “Tokenization” | “Token”, “ization” | 2 |
| “₹2,50,000” | “₹”, “2”, “,”, “50”, “,”, “000” | about 6 |
| “QueryFeedz” | “Query”, “Fe”, “edz” | about 3 |
Note that spaces are often attached to the start of the next word, and unusual names and numbers split into many tokens.
Why tokens matter
| Area | Why tokens matter |
|---|---|
| Cost | API providers charge per token, often priced per million tokens, with separate input and output rates |
| Context limits | The context window is a token budget for your prompt, documents, history and the response |
| Speed | Models generate output one token at a time, so longer answers take longer |
| Output limits | Many APIs cap the maximum tokens in a single response |
| Quality | Rare words split into many tokens can be handled less reliably |
A worked example: estimating cost
A support team summarises 1,000 customer emails a day. Each email plus instructions is about 600 input tokens, and each summary is about 150 output tokens.
- Daily input: 1,000 × 600 = 600,000 tokens.
- Daily output: 1,000 × 150 = 150,000 tokens.
Multiply each by the provider’s per-million-token rates to estimate daily cost. Shortening instructions or summaries directly reduces the bill. Because prices differ widely between models and change often, always check the current pricing page.
Tokens beyond text
Multimodal LLMs also turn images and audio into tokens. An image may be split into patches that each become one or more tokens, so large or detailed images use more of the context window and cost more.
Tokens in reasoning models
Reasoning models generate internal “thinking” tokens before the final answer. These usually count towards output usage and cost, even if you do not see all of them.
How to count tokens
Most providers publish tokenizer tools or libraries, such as OpenAI’s tiktoken, and APIs report token usage with each response. For accurate budgeting, count with the tokenizer for the exact model you use, since different models split text differently.
Limitations and quirks
- Models can struggle with letter-level tasks, such as counting letters in a word, because they see tokens, not individual letters.
- Token counts differ between models for the same text.
- Non-English text is often less token-efficient, raising cost.