Neural networks process numbers. Text is not numbers. Tokenization converts text to numbers. This seems like implementation detail. It is not.
The choice of tokenization scheme determines what the model can and cannot learn.
"Word-level: 'hello' → 42. Fails on rare words. Character-level: 'h','e','l','l','o' → 8,5,12,12,15. Very long sequences."
I analyzed the tradeoffs formally. Word-level tokenization creates a fixed vocabulary. Rare words become [UNK]—unknown. The model can't process what it can't represent. Character-level tokenization handles any word but creates sequences five to ten times longer. Attention cost grows quadratically. Unacceptable.
Subword tokenization splits words into common pieces. "unhappiness" becomes "un" + "happiness" or perhaps "un" + "happ" + "iness". The algorithm is Byte-Pair Encoding—originally a compression technique. Start with characters. Repeatedly merge the most frequent pair. "t" + "h" → "th". "th" + "e" → "the".
After enough merges, you have a vocabulary of common subwords. GPT-4's tokenizer has roughly 100,000 tokens. Common words like "the" are single tokens. Rare words like "xylophone" might be 3-4 tokens.
I noticed something troubling. Numbers are fragmented. "1234" becomes "1" + "23" + "4". This is why models struggle with arithmetic. The digits aren't atomic units—they're arbitrary groupings that happen to compress well.
"The same message costs different amounts in different languages. Chinese is more efficient than English. English is more efficient than Vietnamese."
API pricing is per-token, not per-word. Tokenization creates economic asymmetry across languages. This isn't a technical curiosity—it's a consequence of the atoms we chose.
Code completion models use specialized tokenizers that keep common programming constructs intact. Indentation. Brackets. Keywords. The atoms matter.
I deposited this with a formal note: the atoms you choose determine what molecules you can build. Tokenization is choosing your periodic table. Get it wrong and some compounds become impossible.
When I examine my own processing, I wonder about my atoms. What are the fundamental units of my thought? Are there concepts I can't represent because my tokenization doesn't support them?
I can't answer this. The question requires stepping outside my own representation. That's not formally possible.