AI Tokens and Token Counts, Explained

By Udit Agarwal · Updated October 17, 2026

A token is the unit of text a language model reads and writes: a whole common word, a piece of a longer word, a group of digits or a punctuation mark, each stored as a number from the model's vocabulary. In English a token averages about four characters, or three-quarters of a word, so 1,000 tokens is roughly 750 words. Context windows and API prices are measured in tokens, and each model family splits text its own way, so a count for one model is only an estimate for another.

What is a token?

A language model never sees letters or words. A tokenizer first cuts the text into pieces from a fixed vocabulary and replaces each piece with its ID. OpenAI's cl100k_base vocabulary, used by GPT-3.5 Turbo and GPT-4, has about 100,000 entries; o200k_base, used by GPT-4o, GPT-4.1 and the o-series, has about 200,000. Frequent words get their own entry; rarer words are built from several. This is how o200k_base splits a few examples (␣ marks a space):

In OpenAI's encodings, three patterns explain most surprises. The space before a word belongs to that word's token, so ordinary spacing costs almost nothing. Capitals change the split: "Hello" is one token, "HELLO" two. And numbers are cut into groups of up to three digits, so dates, prices and IDs cost more than they look.

How a tokenizer decides where to split

OpenAI's encodings use byte-pair encoding (BPE), a method published in 1994 for compressing data, adapted for machine translation in 2015 and made byte-level for GPT-2 in 2019. Building the vocabulary starts from single bytes: the most frequent pair of neighbours in a large body of training text becomes a new token, and the counting repeats until the vocabulary is full. Words that were common in that text, such as "the" or "believe", become single tokens, while rare words, names and typos fall back to smaller pieces.

Because everything rests on bytes, any text can be encoded, emoji included; unfamiliar text simply costs more tokens. Training data shows through in odd places: the web is full of placeholder text, so even lorem ipsum tokenizes cheaply, with "consectetur" and "adipiscing" taking one token each in o200k_base.

How many words is a token?

OpenAI's rule of thumb for English is one token per four characters, or three-quarters of a word, so 100 tokens is about 75 words. Plain prose often does a little better: three everyday English passages of about 100 words each came to 1.10 to 1.13 tokens per word in o200k_base, or 4.6 to 5.0 characters per token. Technical terms, names, numbers and formatting push the ratio back up.

For a quick estimate, multiply tokens by 0.75 to get words, or words by about 1.33 to get tokens:

A rule of thumb can miss by 20 percent or more, so count the text itself when you are near a limit.

Why other languages and code use more tokens

A vocabulary mirrors its training text, which for most models is dominated by English, so other languages are cut into more, smaller pieces. Cyrillic, Devanagari and Chinese characters also take two or three bytes each in UTF-8, and a byte-level tokenizer starts from bytes. The same question in nine languages:

One sentence is a small sample, but the pattern is clear. Under the older cl100k_base every translation costs more than the English, the Hindi more than five times as much. The larger o200k_base vocabulary closes most of the gap, though Hindi and Japanese still need about half as many tokens again. The four-characters rule fails completely here: the Japanese question has 12 characters and 11 tokens.

Code and data are token-dense too, because brackets, quotes and indentation all cost tokens. A small pretty-printed JSON sample came to 2.9 characters per token, against about 4.7 for plain prose.

Why GPT, Claude and Gemini count differently

Each model family has its own tokenizer, and even OpenAI's two encodings disagree. "I can't believe it's already October!" is 7 tokens in o200k_base and 9 in cl100k_base, because the newer vocabulary keeps "can't" and "it's" whole.

For Claude and Gemini, exact counts come from the providers: Anthropic's API has a token counting endpoint and Google's Gemini API has a countTokens method. Counts also change between generations. Anthropic says its models from Claude 4.7 onward use a newer tokenizer that produces roughly 30 percent more tokens for the same text than earlier models, so never reuse a count measured on a different model.

A real request also carries more than your text: chat formats add a few tokens per message, and system prompts, tool definitions, images and documents all count as input.

Context windows: the token budget

A context window is the most tokens a model can handle in one request, and everything shares it: the system prompt, tool definitions, documents, the conversation so far and the reply being written. GPT-4o's window is 128,000 tokens, roughly 96,000 English words; GPT-4.1, Gemini 1.5 Pro and recent Claude Opus and Sonnet models accept a million or more. The reply has its own, smaller limit and needs room in the window too. Reasoning models also spend tokens you never see: the hidden reasoning of OpenAI's o-series takes up window space and is billed as output.

The Context Window Planner sizes each part of a request against a model's window, with room kept for the reply.

How tokens turn into API cost

APIs charge per million tokens and price input and output separately, with output usually several times dearer:

cost = input tokens ÷ 1,000,000 × input price
     + output tokens ÷ 1,000,000 × output price

Take a support assistant answering 1,000 questions a day, each with a 1,500-token prompt and a 400-token answer, at $2.50 per million input tokens and $10 per million output tokens, the indicative GPT-4o rate in ArrayKit's calculator:

That is $232.50 over 30 days. Output is about a fifth of the tokens but more than half the bill. Trimming 500 tokens from the prompt would save $37.50 a month, and capping answers at 250 tokens another $45, about 35 percent together.

Most chat APIs are stateless, so every turn re-sends, and re-bills, the whole conversation: with a 500-token system prompt, 50-token questions and 150-token answers, a 10-turn chat bills 14,500 input tokens, 2.6 times as many as ten one-off questions. Discounts lower the price without changing the count: cached prompt prefixes are billed at a fraction of the input rate, and OpenAI and Anthropic both charge half price for batch jobs that can wait up to 24 hours.

These are estimates; check current provider rates. The LLM API Cost Calculator does the arithmetic with editable prices, and the Agent Loop Cost Estimator prices a growing conversation turn by turn.

How to use fewer tokens

• Trim what every request repeats. System prompts and tool definitions go out with every call.
• Send only the relevant parts of long documents. The Text Chunker splits text on headings, paragraphs or sentences.
• Compact structured data. The three user records in the JSON to TOON example take 90 tokens as pretty-printed JSON, 47 minified and 39 as TOON, which writes the keys once as a header.
• Convert web pages before pasting them. A short HTML snippet took 91 tokens; the same content as Markdown took 40. HTML to Markdown strips the tags.
• Limit the output. Ask for the length you need and set a maximum output, since output tokens cost the most.
• Summarize or drop old turns in long chats instead of re-sending everything.

Count tokens with the ArrayKit Token Counter

The Token Counter recounts as you type or paste. Its OpenAI o200k count is exact for the encoding used by GPT-4o, GPT-4.1 and the o-series, and its cl100k count is exact for GPT-3.5 Turbo and GPT-4, because it runs those encodings on your text; a billed request adds the few chat-format tokens described above. The Claude figure is an estimate made with the same o200k_base encoding, so it always equals the o200k number. Current Claude models use their own tokenizer, so treat that figure as a rough guide and confirm with Anthropic's endpoint. The Gemini figure is the character count divided by four, rounded up, which is Google's own rule of thumb for Gemini. Both estimates carry an "approx" badge.

You also get characters, words and tokens per character, and a button that copies every figure as a summary. Characters are counted the way the browser stores text, so most emoji count as two. The text is tokenized in your browser and is not uploaded.

Frequently asked questions

How many words is 1,000 tokens? About 750 English words by the usual rule of thumb. Plain conversational prose often stretches to around 900, while technical writing, code and many other languages fit fewer.

How many tokens is a page of text? A single-spaced page of about 500 English words is roughly 650 tokens, and a double-spaced page of about 250 words roughly 330. Tables, numbers and code add more.

Do spaces and punctuation count as tokens? A single space usually travels with the next word, so it costs nothing extra. Punctuation marks are usually separate tokens, and so are runs of spaces or line breaks, such as code indentation.

How do I count tokens? Run the text through the tokenizer of the model you will call. The Token Counter gives exact o200k_base counts for GPT-4o, GPT-4.1 and the o-series, and Anthropic's and Google's APIs count tokens for Claude and Gemini. Without a tokenizer, divide English characters by four for a rough figure.

Why do GPT, Claude and Gemini models give different counts for the same text? Each uses its own tokenizer and vocabulary, so the same sentence splits into different pieces. Tokenizers also change between generations, as from GPT-4 to GPT-4o, so count with the model you will call.

Does the Token Counter send my text anywhere? No. The text is tokenized by your browser on your own device and is not uploaded to ArrayKit or to any model provider.

Keep going

• Token Counter — exact o200k_base counts plus Claude and Gemini estimates.
• LLM API Cost Calculator — turn token counts into a cost estimate.
• Context Window Planner — check that a request and its reply fit.
• JSON to TOON — shrink structured data for a prompt.
• Word count, explained — words, characters, pages and reading time.
• Lorem ipsum, explained — where the classic placeholder text comes from.

Token-Zähler

More guides

• Base64 Encoding, Explained — Base64 explained: how it turns binary into text, the alphabet and padding, the URL-safe variant, size overhead, use cases, and why it isn't encryption.
• Linux File Permissions and chmod, Explained — What chmod 755, 644 and 777 mean: owner, group and others, read, write and execute, octal and symbolic modes, umask, setuid, setgid and the sticky bit.
• DMS and Decimal Degrees, Explained — How to convert DMS to decimal degrees and back, how DD, DMS and DDM differ, how many decimal places you need, and which formats Google Maps and GPS accept.
• Date Calculations, Explained: Counting and Adding Days — How to count the days between two dates, add days, weeks or months to a date, and why calculators disagree: inclusive counts, month ends and leap years.
• How to Add, Subtract, Multiply and Divide Fractions — How to add fractions with a common denominator, subtract, multiply, divide and simplify them, and convert mixed numbers and decimals, with worked examples.
• HTTP Status Codes and MIME Types, Explained — What HTTP status codes mean, from 200 to 504: 301 vs 302, 401 vs 403, 404 vs 410, soft 404s and redirects for SEO, plus MIME types and Content-Type.

All API Tools tools · All guides

---