There is something slightly strange about the way we interact with AI. We type words into a chatbot, read the response as normal sentences, and naturally assume that the model is processing language in roughly the same way we are. That assumption is useful for everyday conversation, but technically it is wrong.
Large language models do not process text as a sequence of words in the way a person does. Before a model can work with your prompt, the text is broken down into smaller pieces called tokens. Those tokens are then converted into numerical representations that the model can process.
This might sound like an unimportant technical detail. It is not. Tokenization helps explain why AI models can sometimes struggle with seemingly simple language tasks, why two prompts with similar word counts can have different costs, and why the number of words in your prompt is not necessarily a good estimate of how much information you are sending to a model.
A Token Is Not Necessarily a Word
The easiest mistake to make when learning about AI tokens is assuming that one token equals one word. Sometimes it does, but often it does not.
A short and common word may be represented by a single token, while a longer or less common word can be divided into several pieces. Punctuation, numbers, spaces, and parts of words can also influence how text is tokenized. As a result, the same number of words can produce different numbers of tokens depending on the text being processed.
This is one reason token counts can feel unintuitive when you first encounter them. Humans naturally measure written language in words, sentences, and characters, while language models operate on a different representation of the same text.
The distinction matters because tokens are the unit that AI systems actually process and, in many cases, the unit that providers use when calculating API costs.
This Explains Some Very Weird AI Mistakes
Once you understand tokenization, some of the stranger limitations of language models start to make more sense.
In May 2026, TechCrunch reported on Google's AI struggling with a remarkably simple task: spelling words and counting individual letters correctly. The article explained that language models do not naturally represent text as individual letters. Instead, their token-based architecture can represent a word as a complete token, a group of characters, or smaller pieces of a word. TechCrunch's report on why Google's AI struggles with spelling
That creates an interesting paradox. A model can generate software, summarize a lengthy report, or solve a complicated mathematical problem, while still struggling with a question such as how many times a particular letter appears in a word.
From a human perspective, counting letters is trivial because we see the word as a sequence of individual characters. For a language model, however, the underlying representation is fundamentally different.
Why Does AI Use Tokens Instead of Words?
The reason is largely practical. Human language is extremely varied, and a model needs a way to represent an enormous vocabulary without requiring every possible word, spelling variation, name, and technical term to exist as a completely separate unit.
Tokenization provides a compromise. Instead of requiring a model to memorize every possible word as an individual item, a tokenizer can break unfamiliar or uncommon words into smaller pieces that the model already knows how to represent.
This is particularly useful for languages with large vocabularies, technical terminology, names, code, and other forms of text that do not fit neatly into a fixed dictionary. A model can encounter a word it has never seen before and still process its constituent pieces.
The downside is that this representation is not always intuitive from a human perspective. A person sees a sentence made up of words and letters; the model sees a sequence of tokens that are converted into numerical representations before being processed.
Tokens Also Determine How Much You Pay
Tokenization becomes even more important when you start using AI through an API.
AI providers generally price their models according to the number of input and output tokens processed. That means the number of words in your prompt is only an approximation of what you are actually being charged for.
For example, two prompts could contain roughly the same number of words but produce different token counts. One might consist mostly of common English words, while the other could contain source code, unusual names, numbers, or technical terminology. Because those inputs can tokenize differently, their actual processing costs can also differ.
This is one reason developers should pay attention to token usage rather than relying exclusively on word count. A prompt that looks short on the screen can still consume more tokens than expected, particularly when it contains code, structured data, or other token-heavy content.
The Same Problem Applies to AI Limits
Tokens are not only relevant to pricing. They also determine how much information a model can process within its context window.
If a model has a context limit of 128,000 tokens, that does not mean it can necessarily process 128,000 words. The actual amount of text that fits inside the context depends on how that text is tokenized.
This distinction becomes especially important when working with long documents. A document containing 50,000 words does not necessarily consume 50,000 tokens, and the ratio can vary considerably depending on the language and content.
For that reason, estimating the size of an AI prompt based only on word count can lead to surprisingly inaccurate expectations.
So How Many Words Are in a Token?
There is no universal conversion rate.
A common rule of thumb is that one token represents roughly three-quarters of an English word, but that is only an approximation. The actual ratio depends on the tokenizer, the language, and the type of content being processed.
English prose tends to tokenize relatively efficiently because many common words and word fragments appear frequently in training data. Other languages, unusual terminology, source code, numbers, and formatting can behave very differently.
That means a conversion such as "1,000 words equals 1,333 tokens" should never be treated as an exact calculation. It is better to think of word-to-token ratios as estimates that can vary depending on what you are actually sending to the model.
If you want to see how a piece of text translates between words and tokens, the Tokens to Words tool can give you a more practical estimate than simply counting words manually.
This Is Why Token Efficiency Matters
Understanding tokens is useful even if you never build an AI application yourself. It changes the way you think about prompts.
If you are paying for an API, reducing unnecessary tokens can reduce costs. If you are working with a context limit, reducing unnecessary tokens gives you more room for the information that actually matters. And if you are comparing different models, tokenization helps explain why seemingly similar prompts can produce different usage numbers.
For developers, the implications are even larger. A system prompt that is repeated thousands of times can become a significant expense if it contains unnecessary instructions. Similarly, an application that continuously sends large amounts of conversation history may be consuming tokens that provide little additional value.
The goal is therefore not to eliminate tokens. Without tokens, modern language models would not work in the way they do. The goal is to understand what is being tokenized and make sure that the information being processed is worth the cost.
AI Understands Language Differently Than We Do
The next time you type a sentence into an AI chatbot, it is worth remembering that the model does not see exactly what you see.
You see words, spaces, punctuation, and letters. The model receives a sequence of tokens that has been converted into numerical representations, and those representations become the basis for everything that happens afterward.
That difference explains some of the most useful properties of modern language models, but it also explains some of their strangest weaknesses. A system can be extraordinarily capable at generating language while still struggling with tasks that depend on the precise character-level structure of that language.
Tokens are the layer between human language and the mathematical machinery underneath today's AI models. Once you understand that layer, AI pricing, context limits, prompt length, and even some of the technology's oddest mistakes start to make considerably more sense.
