Twitxy
Saved
@tulkenss's profile picture
View on X

slm tokens

@tulkenss

Chief whatever officer @minishlab | Smallest model manufacturer | Silly clown | Citizen of Antwerp
Antwerp, BE stephantul.github.io Joined February 2015
847 Following 535 Followers 2.6K Posts

Latest posts 5

slm tokens @tulkenss
Here it is! blink-8m!

It is strictly better than potion-base-8m on all tasks and metrics we've seen, and is also better at retrieval than larger models, including our own potion-retrieval-32m model.

The model card contains a ton of evals, see it here: huggingface.co/minishlab/blin…
huggingface.co minishlab/blink-8M · Hugging Face Feature Extraction model with 7.68M parameters. Download, explore files and run it on Hugging Face.
Minish @minishlab 1h
We've just released our newest model: blink-8M 🚀

It's a drop-in replacement for potion-base-8M, and it beats every static model we compared on MTEB at 256 dimensions, including potion-base-32M and potion-retrieval-32M.

🧵
View quoted post on X
slm tokens @tulkenss
New model tomorrow! Full replacement for the entire potion line, in a single model.

In addition to BEIR and MTEB, it beats all other 256-dim models on all other retrieval evals we tried (RTEB, Freshstack, Bright, Browsecomp). It also regularly beats 1024-dim models.

Stay tuned!
1 replies 0 reposts 23 likes 602 views View on X
slm tokens @tulkenss
OTOH, it is incredibly difficult to make models consistently use something other than grep. They are trained to use grep, not trained to prefer your own tools.

(nevertheless, grep is very good)
Han Xiao Verified account @hxiao 2d
grep is not all you need, but it's 85%. I went through 3,710 requests my personal agents over the last few months and plotted how they search. 43.2% of those requests involved local search. 85.5% of first searches are grep/rg. Filename search is 5.2%, hybrid (embedding + BM25) 7.5%, pure embedding 1.7%.

The first search usually works 93.5% return results. But only 36.8% of requests stop there. The average search request runs 3.75 searches. And when a search fails, the agent doesn't change tools, it just changes queries. After an empty search, 75% try again, and 89% of those retries are grep. Even start with semantic, hybrid, or filename search, and 55–64% of the next searches still go back to grep.

So in short: grep fails -> do more grep. other search fails -> do more grep. grep is not just a default, it's the fallback as well.

Note that I didn't count web search here as apparently it's not meaningful to compare against grep.
View quoted post on X