AI Glossary
What is Token?
A small piece of text, often part of a word, that language models read and write. AI tools measure input length, output length, limits and pricing in tokens.
Language models do not see text as whole words. They split it into tokens: common words may be one token, while longer or rarer words are split into several pieces. Spaces and punctuation count too.
As a rough guide for English, a token is about three-quarters of a word, so 1,000 tokens is roughly 750 words. Hindi and other languages often use more tokens for the same meaning.
Why tokens matter to you:
- Limits: each tool can only handle a certain number of tokens at once (see Context Window).
- Usage caps: free and paid plans often limit how much you can use, which is related to tokens.
- Developer pricing: AI APIs usually charge per token for input and output.
You rarely need to count tokens yourself. Just know that very long documents or chats may hit limits.