Pinned post
๐ Stop using Transformers for weight-space programming
If you are not training with gradients on a GPU, why default to a Transformer?
When an LLM writes network weights directly, letting it also design the architecture achieves:
๐ ~1,800ร fewer parameters at matched accuracy on ARC (AI2 Reasoning Challenge), compared to the best Transformer the same LLM wrote.
โก Up to 17ร faster GPU inference with a memory architecture built for these tables, compared to a Transformer of the same size.
๐งต Paper & code at the end ๐
If you are not training with gradients on a GPU, why default to a Transformer?
When an LLM writes network weights directly, letting it also design the architecture achieves:
๐ ~1,800ร fewer parameters at matched accuracy on ARC (AI2 Reasoning Challenge), compared to the best Transformer the same LLM wrote.
โก Up to 17ร faster GPU inference with a memory architecture built for these tables, compared to a Transformer of the same size.
๐งต Paper & code at the end ๐