Twitxy Saved

Walter Simoncini

Verified account This account is verified. Learn more

@iWSimoncini

Computer Vision & Efficiency
PhD Student @ University of Amsterdam
Amsterdam, The Netherlands github.com/WalterSimoncin… Joined November 2013
499 Following 151 Followers 56 Posts

Latest posts 4

Walter Simoncini Verified account @iWSimoncini
🪗 Your Transformer Is Secretly Elastic 🧵👇

Transformer families come in a few fixed sizes, so you commit to one for your hardware and latency constraints, even as query loads vary. What if a single pre-trained model could serve every budget?

We learn one importance ranking over every MLP neuron and attention head, once, with frozen weights. At inference time, the model can then be pruned to any computational budget by dropping its least important units, without retraining.

Across 27 vision and text encoders spanning 22M to 8B parameters, performance degrades smoothly up to 60% sparsity. For example, pruning DINOv3 ViT-H+/16 to 50% sparsity removes 427M parameters, for only a 0.8% drop in ImageNet-1k linear probing.

Super fun project in collaboration with Dheeraj Varghese (@dhevarghese), supervised by Gertjan J. Burghouts (@gjburghouts) and Cees Snoek (@cgmsnoek)

🌐 Project page: secretly-elastic.ashita.nl
0:47
3 replies 16 reposts 40 likes 2.7K views View on X
Walter Simoncini Verified account @iWSimoncini
We’ll be presenting our paper today at 4:30 PST, come say hi at poster 4709!
Walter Simoncini Verified account @iWSimoncini Dec 2, 2025
This week I'll be at #NeurIPS presenting our latest work "Elastic ViTs from Pretrained Models without Retraining"!

TL;DR: You can make pretrained ViTs elastic in under 5 minutes on a single A100 via structured pruning, without re-training!

Photo: in-context semantic segmentation with DINOv1 ViT-B/16 models pruned from 0 to 50% sparsity.

1/🧵
View quoted post on X
Walter Simoncini Verified account @iWSimoncini
This week I'll be at #NeurIPS presenting our latest work "Elastic ViTs from Pretrained Models without Retraining"!

TL;DR: You can make pretrained ViTs elastic in under 5 minutes on a single A100 via structured pruning, without re-training!

Photo: in-context semantic segmentation with DINOv1 ViT-B/16 models pruned from 0 to 50% sparsity.

1/🧵
2 replies 15 reposts 100 likes 30K views View on X
Walter Simoncini Verified account @iWSimoncini
DM me if you’re around and want to chat!
.@iWSimoncini is at #NeurIPS2024 to present our FUNGI method to enhance pretrained features w/ self-supervised gradients.
If you missed our poster today you can still catch him at the conference for a chat.
He's really fun and just embarking on a PhD journey 🚀
View quoted post on X