Pinned post
First blog post 🎉
- Live demo of @HuggingFace's tokenizers
- Runs in the browser with #rustlang and #wasm
mathieurene.com/posts/text-tok…
#NLP
mathieurene.com Text Tokenization A friend of mine asked me to explain how one went from words to tokens, to eventually represent text in a way that can be understood by machine learning algorithms. - Live demo of @HuggingFace's tokenizers
- Runs in the browser with #rustlang and #wasm
mathieurene.com/posts/text-tok…
#NLP