Günter Klambauer

@gklambauer

Deep Learning researcher known for self-normalizing neural networks, applications of Machine Learning in Life Sciences; ELLIS Fellow

Joined December 2021

440 Following 4.1K Followers 2.1K Posts

Latest posts 5

Pinned post

Günter Klambauer @gklambauer Oct 20

KronLSTM is out!! | Candidate to replace TRANSFORMER in LLMs

KronLSTM is built on the Kronecker Delta Rule: instead of correcting the vector stored at a key, it corrects a single scalar, how strongly this key is bound to this value.

P: arxiv.org/abs/2610.26834

Günter Klambauer @gklambauer 1d

Will biology/medicine face a similar fate as math??

I don't think so...

Günter Klambauer @gklambauer 2d

It's 2018 again.. you thought you had drawn the attention of the ML community to tabular data with self-normalizing MLPs...

David Tao Verified account @Taodav 3d

It's 2018 again.
Everyone is excited about all these crazy images generated from GANs.
The same guy that made GANs just came out with a deep learning textbook with Bengio.
NIPS just changed their name to NeurIPS and you wonder if that's actually going to stick.
You train your Encoder-Decoder LSTM model with attention.
Some crazy people are starting to use this new model architecture called a Transformer.
Other crazy people are starting to fine-tune with something called REINFORCE to maximize BLEU and ROUGE scores.
OpenAI just solved a Rubik's cube with a robot hand.
Life is good.
View quoted post on X

Günter Klambauer @gklambauer Nov 4

World championship in AI 🏆🏆🏆

Lukas Aichberger @aichberger Nov 4

We just hit 7th out of 5,000+ teams in the ARC-AGI-3 Challenge! 🚀

Biggest takeaway? Under tight compute and memory constraints, a good harness and high inference throughput beat a bigger model!

Huge shoutout to the team: @fses91 @MI99835819 @gklambauer

kaggle.com/competitions/a…

View quoted post on X

Günter Klambauer @gklambauer Nov 2

Self-normalizing neural networks rediscovered: arxiv.org/abs/1706.02515

Here, both the activation function is shifted/adjusted such that both MEAN is zero and VAR is one.

Ziming Liu Verified account @ZimingLiu11 Oct 31

There is a "bug" in commonly used activation functions (ReLU/SiLU/GeLU etc.) ---- The mean of post-activations is not zero.

This bug, however small it seems, leads to pathologies of training dynamics, including loss spikes & representation overshooting.

By removing the bug, we get improved performance in LLMs and ViTs.

The science of AI can never be under-estimated.

View quoted post on X