Tom Yeh

MLP in PyTorch by hand ✍️ ~ 7 steps walkthrough below

Goal: fill in every blank in the PyTorch code to build a multi-layer perceptron.

1. Given

Let us start with a code template on the left and the network it is supposed to build on the right. Every blank in the code can be worked out from the picture.

2. Linear layer

We count: 3 features in, 4 features out. So the weight matrix is 4 by 3. There is an extra column for the biases, which means bias = T.

3. ReLU

Let us apply the activation. ReLU crosses out the negatives, so -1 becomes 0.

4. Linear layer

The input size is 4, because that is what the previous layer put out. The output size is 2. A 2 by 4 weight matrix, and this time no extra column, so bias = F.

5. ReLU

We cross out the negatives again.

6. Linear layer

Two features in, five out. A 5 by 2 weight matrix, with a bias column, so bias = T.

7. Sigmoid

Let us finish. Sigmoid squashes the raw scores (3, 0, -2, 5, -5) into probabilities between 0 and 1.

You have just implemented a three-layer deep neural network by hand. ✍️

== Story ==

Three years ago I gave this exercise to my students, to connect the code to the math.

They found it odd. Every other AI course they were taking lived inside a Jupyter notebook, and here I was handing out paper.

Three years later, my colleagues are the ones rushing to move their materials to paper.

The exercise has not changed. Paper still asks the one thing a notebook lets you skip: do you actually understand what the code is doing? If you can tell me why the weight matrix is 4 by 3, and why bias is F on the second layer, you understand nn.Linear better than someone who has been copy-pasting it for a year.

💾 Save this post!

#AIbyHand #PyTorch #DeepLearning

Variational Autoencoder by hand ✍️ ~ 11 steps walkthrough below

A VAE learns the structure of your data, the mean and variance of its hidden features, and then generates new data from that structure.

A GAN only learns to fool a discriminator. It can make convincing fakes without ever knowing what the data is really made of. That is the difference, and it is the whole reason VAEs matter.

In 2024 ICLR gave its first ever Test of Time Award to the VAE paper, "Auto-Encoding Variational Bayes" by Diederik Kingma and Max Welling, ten years on.

How does it work?

Goal: encode three inputs into a distribution, sample from it, decode it back, and read every loss gradient off the page.

= 1. Given =

Three training examples X1, X2, X3, copied to the bottom as their own targets. Reconstructing your own input is what puts the "auto", meaning self, in autoencoder.

= 2. Encoder, layer 1 =

Let us multiply the inputs by weights and biases, then apply ReLU, crossing out every negative.

= 3. Mean and standard deviation =

We multiply the features by two more weight sets. The first predicts the means μ of the latent distributions, the second their standard deviations σ.

= 4. A random offset =

Let us sample ε from a standard normal, mean 0 and variance 1, and multiply it by σ. This is a random step away from the mean, scaled by how uncertain each feature is.

= 5. Mean plus offset =

We add the offset back onto μ, and these become the decoder's inputs. Keeping the randomness out in ε is the reparameterization trick: it lets gradients flow straight through the sampling.

= 6. Decoder, layer 1 =

Let us multiply by weights and biases and apply ReLU again. Here -4 is crossed out.

= 7. Decoder, layer 2 =

We multiply once more. The output Y is the decoder's attempt to rebuild X from the sampled distribution.

= 8. Gradient for the mean =

Let us push μ toward 0. A lot of math, the SGVB estimator, collapses the KL gradient to simply μ itself.

= 9. Gradient for the standard deviation =

We want σ to approach 1.

= 10. And its formula =

That same math simplifies the gradient to σ minus 1/σ.

= 11. Reconstruction gradient =

We want the reconstruction Y to match the input X. Mean squared error simplifies its gradient to Y minus X.

Takeaway: the two gradients you just calculated each sit at the heart of a modern method, so one VAE teaches you both. The KL divergence is the penalty RLHF like GRPO uses to keep a fine-tuned model from drifting off its base. The reconstruction loss, plain mean squared error, is exactly what trains a diffusion model to denoise. Draw one VAE by hand and you have quietly learned the core of both.

💾 Save this post!

Can you solve these agentic AI math problems by hand ✍️? Getting a bit harder now. Download PDF: byhand.ai/tokens-11-15

Problems 11 to 15:

11. Comparing three calls: same total tokens, three different bills
12. The average call: size a sample, then scale it to the day
13. The system prompt: the tokens you re-send on every single call
14. The system prompt tax: what repeating those instructions costs
15. Calls in a budget: cost one call, then divide

Download, print, solve, and learn. ✍️

~ Prof. Tom Yeh

Kimi 3 seminar recording is uploaded 👉 byhand.ai/v/kimi3

~ Prof. Tom Yeh

Self-Attention vs Cross-Attention interactive diagram. Open byhand.ai/self-vs-cross