Yulu Gan

This account is verified. Learn more

@yule_gan

PhD student @MIT_CSAIL @MIT_CBMM, Research intern @NVIDIAAI/ ex @PKU1898 @MSFTResearch

Cambridge, MA yulugan.com Joined October 2022

208 Following 3.8K Followers 211 Posts

Latest posts 5

Yulu Gan @yule_gan Oct 31

Cool results and worth a read. Looks like there are thickets in logit/output space. I think the same can happen in the input space as well! Some results also align with our finding that voting makes larger gains than random guessing (Figure 11 in our paper).

Nicolas DUFOUR @nico_dufour Oct 30

Neural Thickets (@yule_gan @phillip_isola): add random noise to an LLM's weights, keep the best copies, let them vote. Reasoning jumps, no training.
Why does it work? It's sampling in disguise. Swap noise on billions of weights for one random vector on the logits: same gains 🧵

View quoted post on X

Yulu Gan @yule_gan Aug 19

I see thickets here : ) Reminds me of what we see with LLMs -- diverse task experts are around the pretrained weights!

Generalist Verified account @GeneralistAI Aug 19

For few-shot learning, it can adapt to new physical tasks in as few as 1 - 10 gradient steps on 1 - 5 minutes of data (~10 - 50 demonstrations). In practice, this can be described as test-time training in a low-data regime. We did not tune this procedure or sweep hyperparameters; these results come largely out of the box.

View quoted post on X

Yulu Gan @yule_gan Jul 7

An update to the Neural Thicket paper:

We avoid the K× inference cost of RandOpt ensembling and make RandOpt iterative: a single model with strong performance.

Like policy gradients, but with search in weight space and stepwise supervision. Like ES, but with updates through distillation in function space.

We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% acc, outperforming PPO, GRPO and ES under a matched compute budget.

And @phillip_isola will present Neural Thicket at the ICML 2026 poster session this afternoon! drop by if you want a guided path through the thicket :)
Wed, Jul 8, 2026, 2:30–4:15 PM KST
Hall A #2602

Check details in the blog post: thickets.mit.edu/blogs/iter_ran…
Code: github.com/sunrainyg/Rand…

Yulu Gan Verified account @yule_gan Mar 13

Simply adding Gaussian noise to LLMs (one step—no iterations, no learning rate, no gradients) and ensembling them can achieve performance comparable to or even better than standard GRPO/PPO on math reasoning, coding, writing, and chemistry tasks. We call this algorithm RandOpt.

To verify that this is not limited to specific models, we tested it on Qwen, Llama, OLMo3, and VLMs.

What's behind this? We find that in the Gaussian search neighborhood around pretrained LLMs, diverse task experts are densely distributed — a regime we term Neural Thickets.

Paper: arxiv.org/pdf/2603.12228
Code: github.com/sunrainyg/Rand…
Website: thickets.mit.edu

View quoted post on X

Yulu Gan @yule_gan Jun 7

This is a very cool idea to decomposes sequence modeling into learning what to remember and how to update memory!

Akarsh Kumar Verified account @akarshkumar0101 Jun 7

We never really knew how to train nonlinear RNNs well… BPTT struggled with vanishing grads (no long-range memory) and sequential rollout (hard to parallelizable).

What if instead an oracle told us the optimal memory state m_t at each step? Then the RNN could do one-step supervised learning on (m_t, x_{t+1}) → m_{t+1} labels.

We call this Supervised Memory Training (SMT): a replacement for BPTT that trains RNNs without unrolling them. SMT is time-parallelizable and solves vanishing gradients.

Website: akarshkumar.com/smt/
arXiv: arxiv.org/abs/2606.06479

View quoted post on X

Yulu Gan @yule_gan May 15

A fun experiment comparing a random step with one gradient step:

With a small CNN on CIFAR-10, a random step is basically a disaster. (A gradient step is a ~185σ event.)

That makes sense if you expect a random direction in R^d to be ~sqrt(d) standard deviations worse than the optimal one. So scaling up to a larger model should make things even worse.

But with a 7B model (test on GSM8k), random steps have a good chance of outperforming a gradient step.

(The gradient norm of one PPO update is 1.94, while the L2 norm of the Gaussian perturbation is 85.6. The figure below rescales the Gaussian perturbation to match the PPO update norm, so the random step and gradient step have the same radius.)

We should really rethink the parameter-function map.