AI concepts
Tokenisation (splitting text into pieces for an LLM)
The process of turning input text into a sequence of tokens, the smallest units a model operates on. A token is typically part of a word, a whole word or a punctuation mark. In English one token corresponds on average to 0.75 of a word, in Polish to 0.5 of a word. The number of tokens determines the cost of a query, the context window limit and the response time.
Primary source: Sennrich Byte-Pair Encoding paper 2016, OpenAI Tokenizer Documentation 2024, HuggingFace Tokenizers Library Reference 2025
Tokenisation is an invisible but cost-driving layer of every AI deployment. Every query, every response and every fragment of context is tokenised. LLM pricing is quoted "per million tokens", not per word. Without understanding tokenisation you cannot sensibly estimate the cost of a production deployment.
The technical mechanism
The dominant algorithm in 2025 is BPE (Byte-Pair Encoding), described in the foundational paper by Sennrich, Haddow and Birch from 2016 (Neural Machine Translation of Rare Words with Subword Units). The algorithm starts from individual bytes and merges the most frequent pairs until the vocabulary reaches a set size (typically 50,000 to 200,000 tokens).
The practical consequence: frequent English words ("the", "and", "of") are one token. Rare English words ("antidisestablishmentarianism") are 5 to 7 tokens. Polish words have far more tokens than English ones, because models are trained mainly on an English corpus.
The impact on cost
OpenAI Tokenizer Documentation of 2024 lets you check the token count for any text. Practical figures for the GPT-4 tokeniser:
- "Najpierw porządek, później AI" (29 characters) = 12 tokens
- "Order first, AI second" (22 characters) = 5 tokens
The same message costs 2.4 times less in English than in Polish. For applications handling large volumes of Polish-language queries, the operating cost is materially higher.
Cost mitigations
HuggingFace Tokenizers Library Reference of 2025 describes three approaches. First, local LLMs trained on a Polish corpus (Bielik, Trurl) have far more efficient Polish tokenisation than GPT or Claude. Second, prompt caching: the same context fragments (for example the system prompt) are counted once on the first query, with discounts of 50 to 90 percent on subsequent ones. Third, summarisation before sending to the model (a trade-off of quality for cost).
A practical estimate
A customer support chatbot application at a Polish bank, 10,000 conversations per month, an average conversation of 8 turns, each turn with a 500-character prompt and an 800-character response: 10,000 × 8 × 1,300 = 104 million characters, around 52 million Polish tokens. The GPT-4o cost: around 750 dollars per month. The cost of a local Bielik on your own GPU: 200 dollars of hosting plus a one-off 30,000 dollars of hardware (amortised over 2 years).
Estimating the operating cost of AI in the Polish context is a standard part of the AI Readiness Audit.