Twitxy Saved

Max Idahl

Verified account This account is verified. Learn more

@maxidahl

AI Research @ ellamind
Joined January 2017
919 Following 195 Followers 114 Posts

Latest posts 5

Max Idahl Verified account @maxidahl
im curious what made TML deviate from OpenAI's o200k tokenizer for Inkling 2
- same vocab size and pre-tokenizer regex
- new vocab, ~40% overlap with o200k
- less space+word tokens
- zero tokens that mix punctuation and letters (no more "'s" or "_loss" tokens)
- "\n" never shares a token with a letter, digit or punctuation
Thinking Machines Verified account @thinkymachines 1d
Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license.

Make Inkling your own!
huggingface.co/thinkingmachin…
View quoted post on X
2 replies 1 reposts 36 likes 2.5K views View on X
Max Idahl Verified account @maxidahl
Tokenization Tax Report 2026 now also on @huggingface
hf.co/spaces/ellamin…
huggingface.co Tokenization Tax Report 2026 - a Hugging Face Space by ellamind Use the interactive page to compare how many tokens different language models need for the same text. Pick a language, document type, or model from the navigation menu and view charts and tables th...
Max Idahl Verified account @maxidahl Sep 8
Compared the tokenizers of current models.

Efficient: Command A+, Muse (Glimmer and Spark, based on Llama 4’s vocab), Grok 4.5+, Qwen 3.5+, OpenAI o200k (GPT-6, GPT 5, gpt-oss)

Claude? +59% tokens for English text...
View quoted post on X
Max Idahl Verified account @maxidahl
Forgot to add: As European AI tradition demands, none of the four European tokenizers we tested beats the o200k baseline averaged across the seven EU languages we experimented with.
Max Idahl Verified account @maxidahl Sep 8
Compared the tokenizers of current models.

Efficient: Command A+, Muse (Glimmer and Spark, based on Llama 4’s vocab), Grok 4.5+, Qwen 3.5+, OpenAI o200k (GPT-6, GPT 5, gpt-oss)

Claude? +59% tokens for English text...
View quoted post on X