Pinned post
For years since the GPT-2 paper, emergent in-context learning (ICL) from 'next-token' training has been treated as something deeply tied to ๐ก๐ฎ๐ฆ๐๐ง ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐. But โฆ is it?
Thrilled to share our latest result: ๐๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ๐งฌ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ค๐ฃ๐ก๐ฎ ๐ผ๐ป '๐ป๐ฒ๐ ๐-๐ป๐๐ฐ๐น๐ฒ๐ผ๐๐ถ๐ฑ๐ฒ ๐ฝ๐ฟ๐ฒ๐ฑ๐ถ๐ฐ๐๐ถ๐ผ๐ป' ๐ฒ๐ ๐ต๐ถ๐ฏ๐ถ๐ ๐๐๐!
What's remarkable is that their overall pattern closely mirrors LLMs:
โ similar few-shot pattern induction
โ similar log-linear gains with more shots
โ similar improvement with model scale
... all learned purely from DNA (nucleotide) sequences.
๐๐ผ๐ ๐ฑ๐ถ๐ฑ ๐๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ ๐ด๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ ๐๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐? We built a suite of symbolic bitstring-reasoning tasks and encoded them two ways: (1) genomic alphabet (A/T/C/G) and (2) linguistic alphabet (digits). This lets us compare Evo2 (genomic) vs Qwen3 (language) under matched few-shot prompts.
๐ช๐ต๐ ๐ถ๐ ๐บ๐ฎ๐๐๐ฒ๐ฟ๐: To our knowledge, this is the first evidence of emergent ICL in non-[human]language symbolic sequences. It suggests that ICL is modality-agnostic, and a general consequence of large-scale autoregressive training on rich data distributions.
๐๐ผ๐ฒ๐ ๐๐๐ ๐ถ๐ป ๐ด๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ ๐๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ฎ๐ฐ๐ ๐ถ๐ฑ๐ฒ๐ป๐๐ถ๐ฐ๐ฎ๐น๐น๐? No! While share macro-level ICL trends, each shows domain-specific inductive biases traceable to properties of DNA vs human language.
๐๐ผ๐ฒ๐ ๐๐ต๐ถ๐ ๐บ๐ฒ๐ฎ๐ป ๐ต๐๐บ๐ฎ๐ป ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐๐๐ฟ๐๐ฐ๐๐๐ฟ๐ฒ ๐ถ๐ ๐ถ๐ฟ๐ฟ๐ฒ๐น๐ฒ๐๐ฎ๐ป๐? No! But it suggests there may be universal distributional properties across different languages (human, DNA, etc.) that yield ICL. It remains an open question what these properties are.
Draft: huggingface.co/papers/2511.12โฆ
Huge thanks to @N8Programs for leading the work, and to collaborators @anqi_liu33 @aamixsh @mrevsine @mike_schatz.
We're extremely thankful to the Evo2 team ( @BrianHie @pdhsu @garykbrixi @mgdurrant @MichaelPoli6 etc.). Not only these models help advance biomed research, now we see that they can help AI community better understand the fundamentals of pre-training.
Thrilled to share our latest result: ๐๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ๐งฌ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ค๐ฃ๐ก๐ฎ ๐ผ๐ป '๐ป๐ฒ๐ ๐-๐ป๐๐ฐ๐น๐ฒ๐ผ๐๐ถ๐ฑ๐ฒ ๐ฝ๐ฟ๐ฒ๐ฑ๐ถ๐ฐ๐๐ถ๐ผ๐ป' ๐ฒ๐ ๐ต๐ถ๐ฏ๐ถ๐ ๐๐๐!
What's remarkable is that their overall pattern closely mirrors LLMs:
โ similar few-shot pattern induction
โ similar log-linear gains with more shots
โ similar improvement with model scale
... all learned purely from DNA (nucleotide) sequences.
๐๐ผ๐ ๐ฑ๐ถ๐ฑ ๐๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ ๐ด๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ ๐๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐? We built a suite of symbolic bitstring-reasoning tasks and encoded them two ways: (1) genomic alphabet (A/T/C/G) and (2) linguistic alphabet (digits). This lets us compare Evo2 (genomic) vs Qwen3 (language) under matched few-shot prompts.
๐ช๐ต๐ ๐ถ๐ ๐บ๐ฎ๐๐๐ฒ๐ฟ๐: To our knowledge, this is the first evidence of emergent ICL in non-[human]language symbolic sequences. It suggests that ICL is modality-agnostic, and a general consequence of large-scale autoregressive training on rich data distributions.
๐๐ผ๐ฒ๐ ๐๐๐ ๐ถ๐ป ๐ด๐ฒ๐ป๐ผ๐บ๐ถ๐ฐ ๐๐ ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ฎ๐ฐ๐ ๐ถ๐ฑ๐ฒ๐ป๐๐ถ๐ฐ๐ฎ๐น๐น๐? No! While share macro-level ICL trends, each shows domain-specific inductive biases traceable to properties of DNA vs human language.
๐๐ผ๐ฒ๐ ๐๐ต๐ถ๐ ๐บ๐ฒ๐ฎ๐ป ๐ต๐๐บ๐ฎ๐ป ๐น๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐๐๐ฟ๐๐ฐ๐๐๐ฟ๐ฒ ๐ถ๐ ๐ถ๐ฟ๐ฟ๐ฒ๐น๐ฒ๐๐ฎ๐ป๐? No! But it suggests there may be universal distributional properties across different languages (human, DNA, etc.) that yield ICL. It remains an open question what these properties are.
Draft: huggingface.co/papers/2511.12โฆ
Huge thanks to @N8Programs for leading the work, and to collaborators @anqi_liu33 @aamixsh @mrevsine @mike_schatz.
We're extremely thankful to the Evo2 team ( @BrianHie @pdhsu @garykbrixi @mgdurrant @MichaelPoli6 etc.). Not only these models help advance biomed research, now we see that they can help AI community better understand the fundamentals of pre-training.
