Twitxy Saved

Daily Dose of Data Science

Verified account This account is verified. Learn more

@DailyDoseOfDS_

Delivering daily insights in DS, ML, RAGs, Agents & AI Engineering. Trusted by over 100k+ readers!
Join for free → join.dailydoseofds.com Joined May 2022
2 Following 52K Followers 2.9K Posts

Latest posts 5

Daily Dose of Data Science Verified account @DailyDoseOfDS_
Why is the Kernel Trick called a "trick"?

(it's a popular ML interview question)

Many ML algorithms use kernels for robust modeling, like SVM, KernelPCA, etc.

The core objective of a kernel function is to compute dot products in some other feature space (mostly high-dimensional) without projecting the vectors to that space.

But how does that even happen?

Consider the image below.

Let’s assume the following polynomial kernel function:
- k(X, Y) = (1+XᵀY)².

Also, for simplicity, let’s say both X and Y are two-dimensional vectors:
- X = (x1, x2)
- Y = (y1, y2)

As shown in the image below, simplifying the kernel expression produces a dot product between the two 6-dimensional vectors.

This shows that the kernel function we chose earlier computes the dot product in a 6-dimensional space without explicitly visiting that space.

And that is the primary reason why we also call it the “kernel trick.”

More specifically, it’s framed as a “trick” since it allows us to operate in high-dimensional spaces without explicitly computing the coordinates of the data in that space.

RBF kernel is even better in this respect. It lets you compute the dot product in an infinite-dimensional space without explicitly visiting that space.
2 replies 6 reposts 35 likes 1.7K views View on X
Daily Dose of Data Science Verified account @DailyDoseOfDS_
Stanford reinvented text generation.

(it’s neither autoregressive nor diffusion)

Autoregressive LLMs generate one token at a time. Longer answers require longer chains of sequential predictions.

Diffusion LLMs generate many tokens in parallel, but still need repeated refinement passes. Each pass processes the full sequence, and quality drops quickly when you cut the passes.

Stanford researchers proposed another approach, Next-Concept Modeling (NCM). It generates text from coarse structure to fine detail in roughly a dozen passes.

First, a tokenizer learns to represent text as layers of compressed information. Early layers capture broad structure, while later layers add details the earlier ones missed.

A transformer then learns to generate these layers in order, predicting everything within each layer simultaneously. A decoder converts the completed representation into text.

The advantage is that early layers are tiny. Generation needs fewer passes, and those early passes process much less information.

You can see this in the team’s translation example. Early layers produce rough, repetitive text, which later layers refine into a clean translation.

In the researchers’ tests, NCM generated text 10–31× faster than strong diffusion baselines, with comparable or better benchmark scores. On LM1B and OpenWebText, it also used roughly 4–5× fewer training tokens, including tokenizer training.

That speed can also improve quality. Generating several candidates and selecting one based on agreement outperformed diffusion baselines at the same sampling time budget.

The limits matter. Scaling experiments reach 725M parameters, tasks use separate tokenizers, and the release includes evaluation code and checkpoints without a training pipeline.

Some evaluations also use repetition penalties during final decoding, without additional model calls.

NCM shows that parallel generation can work by building increasingly detailed representations. Whether those gains hold at general-purpose LLM scale remains open.

- GitHub repo: github.com/HazyResearch/n…
- Paper: arxiv.org/abs/2610.14162
6 replies 23 reposts 171 likes 6.2K views View on X
Daily Dose of Data Science Verified account @DailyDoseOfDS_
Google open-sourced Harness².

(turn any harness into a self-improving system)

a harness is everything wrapped around the model, like the system prompt, tools, memory, skills, and subagents. today a person designs it once, and the agent runs every new task with that same fixed setup.

Harness² lets the agent refine that setup for each new task on its own, with the model weights left untouched.

on every task, the agent proposes edits to its harness, runs a few candidate versions, and composes a better harness by comparing how those runs behaved.

because it learns from the differences between runs, it needs no correct answer or grader for the task.

that covers a single task. Harness² also carries what it learns from one task into the next.

after each task, it saves the harness it settled on along with the lessons that held up across the runs. the next time it refines a harness, it starts from those notes instead of from scratch.

this means every finished task makes the next round of edits more informed.

Google tested this with Gemini 3.5 Flash and GLM-5 on legal, office, and professional work benchmarks. one round of edits was enough to beat their original harnesses, and giving the agent more rounds pushed scores higher.

those extra rounds can go wide or deep. the agent either tries several candidate harnesses at once or keeps refining a single one step by step, using OpenCode or Codex to carry out the work.

you can find the Harness² repo here: github.com/google-researc…
14 replies 24 reposts 254 likes 8.5K views View on X
Daily Dose of Data Science Verified account @DailyDoseOfDS_
5 LLM fine-tuning techniques, explained with visuals:

Traditional fine‑tuning is impractical for LLMs (billions of params; 100s of GB).

Since this kind of compute isn't accessible to everyone, parameter-efficient fine-tuning techniques (PEFT) were built.

Before we go into details of each technique, here's some background that will help you better understand these techniques:

LLM weights are matrices of numbers adjusted during fine-tuning.

Most PEFT techniques involve finding a lower-rank adaptation of these matrices, a smaller-dimensional matrix that can still represent the information stored in the original.

Now with a basic understanding of the rank of a matrix, we're in a good position to understand the different fine-tuning techniques.

(refer to the image below for a visual explanation of each technique)

1) LoRA

- Add two low-rank trainable matrices, A and B, alongside the weight matrices.
- Instead of fine-tuning W, adjust the updates in these low-rank matrices.

Even for the largest LLMs, LoRA matrices take up a few MBs of memory.

2) LoRA-FA

While LoRA significantly decreases the total trainable parameters, it requires substantial activation memory to update the low-rank weights.

LoRA-FA (FA stands for Frozen-A) freezes matrix A and only updates matrix B.

3) VeRA

- In LoRA, low-rank matrices A and B are unique for each layer.
- In VeRA, A and B are frozen, random, and shared across all layers.
- Instead, it learns layer-specific scaling VECTORS (b and d) instead.

4) Delta-LoRA

- It tunes the matrix W as well, but not in the traditional way.
- Here, the difference (or delta) between the product of matrices A and B in two consecutive training steps is added to W.

5) LoRA+

- In LoRA, both matrices A and B are updated with the same learning rate.
- Authors of LoRA+ found that setting a higher learning rate for matrix B results in better convergence.
0:06