mike
Verified account This account is verified. Learn more@mike64_t
descending the gradient
513 Following 8K Followers 5.2K Posts
Latest posts 5
retweeting for those interested
right, I think you just arrived at the mega kernel discourse from first principles, but register allocation etc is not the only reason why you want to split things into multiple kernels. I think in general composing workloads from kernels when everything has to be one giant binary is a big problem. It's the same dillema as having to recompile a PSO just because you changed some rasterizer state. Your application needs to be able to decide fast and it needs to do so along certain granularities. You already die from data type kernel variant combinatorial explosion just trying to make GEMM-likes work, now think about the possible schedules of orchestrating work for say an NN forward and backward pass. Your hyper parameters are at the end of the day runtime state and you cannot afford to actually compile all those out. I'm not sure I see any other way there than to have a JIT/graph compiler, which I am generally opposed to. I just don't see the mega kernel reacting to config state and then doing the equivalent of control flow on the GPU. That just seems bad. Glueing kernels together with a graph just happens to be a very very good paradigm to get the best of both worlds, not actually needing to have logic to decide what to do next within the kernerls, thus being able to reuse them. Also, a PDL edge is not actually that slow compared to a grid sync. I suspect most mega kernels are wasting more time managing their intra-kernel task-calling convention than they realize.View quoted post on X
And not being able to do this is actively holding alternative hardware back to run actual training workloads. I think hardware that thinks "megakernels are the future" only win under a narrow subset of futures where flexibility of no concern while the performance benefits are close to zero. Now, hardware scheduling might sound less appealing depending on what it is you want to do, but if you view cuda graphs as a time-sharing + schedule declaration / overlap api, I think the design space is just more appealing than having to do any of this in isa space. I believe any piece of hardware that thinks this layer of scheduling must happen in ISA is creating unnecessary trouble and self-sabotaging themselves. Having *both* hardware scheduling and instruction level scheduling available to you does create *more* freedom in how you write and orchestrate kernels. There is no reason to double down on any one principle out of a sense of purism. There are cases where you just want to split things into two kernels and eat the PDL edge at a well understood point. say kernel A has associated work in kernel B, and A dependent kernel C, but C does not depend on B. Just get it out of the critical path and b can take however long it takes and you don't need to worry about the "lifetime" of A being long enough to encompass that.
At the risk of saying something wrong on the internet in hopes of getting corrected, I think one of the best things Nvidia has ever done is the qmd dependent ptr.
as far as i can tell, in 2012 the dependent QMD exists in hardware, with no user-facing API beyond whatever the CDP runtime did with it internally.
in 2018, cuda graphs re-abstracts the mechanism to use dependent chains to drive down inter-kernel latency.
In 2022, PDL on Hopper hides the cold I$ issue of dependent kernels with overlap.
In 2026, Vera Rubin fixes the remaining issue of uneven completion with tile-level triggering.
Now I think the dependent ptr and the decrement dependence counters in QMD are one of the best things nvidia has ever done and I think an under-appreciated aspect of what makes CUDA great despite nobody being consciously aware of it...
now I am cautiously awaiting on which side this places me in this debate 😳
as far as i can tell, in 2012 the dependent QMD exists in hardware, with no user-facing API beyond whatever the CDP runtime did with it internally.
in 2018, cuda graphs re-abstracts the mechanism to use dependent chains to drive down inter-kernel latency.
In 2022, PDL on Hopper hides the cold I$ issue of dependent kernels with overlap.
In 2026, Vera Rubin fixes the remaining issue of uneven completion with tile-level triggering.
Now I think the dependent ptr and the decrement dependence counters in QMD are one of the best things nvidia has ever done and I think an under-appreciated aspect of what makes CUDA great despite nobody being consciously aware of it...
now I am cautiously awaiting on which side this places me in this debate 😳
GPU HW already has a mechanism to track when kernels finish. Traditionally used for things like writing timestamps when a dispatch is finished. So we don't actually need to do atomics at the end of the child kernels. Instead we could do a separate scheduling kernel.View quoted post on X
Holy fucking AAAAAAA.... the larping is truly impeccable.
"become convinced that PDL gives you a strict superset of megakernel capabilities" of course subtweets my tweet yesterday except that's not at all what I said.
I said "a cuda-graph containing co-designed kernels expresses a strict super-set of what a single megakernel can do". But of course the LLM that generated this tweet couldn't freaking read correctly.
"become convinced that PDL gives you a strict superset of megakernel capabilities" of course subtweets my tweet yesterday except that's not at all what I said.
I said "a cuda-graph containing co-designed kernels expresses a strict super-set of what a single megakernel can do". But of course the LLM that generated this tweet couldn't freaking read correctly.
Of course that’s your contention.View quoted post on X
You’re a first-year inference engineer.
You just read SemiAnalysis’s blog and saw megakernels coming to TPUs and AMD, so naturally you believe that’s the way to get to speed of light.
Next month, when you get your hands on an inference engineering cookbook—Baseten’s, probably—you’ll be singing from the treetops about the 500 nanoseconds of critical-path latency you save on every GEMM pipeline drain.
That’ll last until sometime next year, when you watch a pirated geohotz lecture livestream and become convinced that PDL gives you a strict superset of megakernel capabilities.
“Well, as a matter of fact, I won’t, because PDL introduces—”
“PDL introduces unnecessary stalls because its synchronization mechanism is very coarse.”
Yeah, I read that too. HazyResearch’s blog, “Look Ma, No Bubbles!” Just below Figure 2.
Do you have any thoughts of your own on the subject, or should I just point my agent at the SGLang repo and have it do it for me?