Twitxy Saved

Shital Shah

Verified account This account is verified. Learn more

@sytelus

CEO and founder Guni.AI. Ex-Microsoft Research, Microsoft AI, Code infra lead Microsoft Phi models. If universe is an optimizer, what is its loss function?
Redmond, WA shital.com Joined July 2007
15K Following 15K Followers 8.2K Posts

Latest posts 5

Pinned post

Shital Shah Verified account @sytelus
This is exactly what Reverse Polish LISP is. I coded exclusively in RPL in my formative years because the only programmable device available to me was a second hand stolen HP 48SX calculator. My father was able to buy it for $10 because it came with no manuals. The guy selling it was frustrated because even doing just 2+2 resulted in error message.

It took me a full week to figure out how everything actually worked. Eventually I wrote all kind of multi-thousand line programs including even video games! But all of that time I had no idea what was the name of the programming language I was coding in. I'd figured out its syntax only by trial and error.

It was painful using that little keyboard to write thousand line programs. Of course there was no storage device so if internal memory gets full, you delete code and later re-type whole thing all over again to "reload" it.

So I might be the only person alive who coded in language without knowing its name or official syntax 😅.

In any case, I still consider RPL to be one of the most beautiful language I ever coded in. There are no traditional memory allocations. You just put everything on stack. There is no distinction between functions and data so function definitions also go on stack. You use various stack manipulation operators (which themselves can be on stack). It was beautiful.
John Carmack Verified account @ID_AA_Carmack Oct 23
I always thought it was a shame that modest default stack setups made alloca() a generally-frowned-upon technique in C/C++. With 64 bit architectures, give every stack a virtual address space equal to the physical memory by default!

You can do a surprisingly good job of memory management with just the stack, and I contend that it has both correctness and performance advantages.

Allocations are a trivial pointer adjustment, and frees are literally free. Keeping everything linearly packed is friendlier for all layers of the cache hierarchy, and it makes keeping everything in huge / gigantic pages easy.

Leaving temporary intermediate blocks on the stack can seem wasteful, but unless you are dealing with algorithms that make > O(n) temporary allocations, it is usually not a big deal.

Static analysis can check if you store a stack pointer into a structure deeper in the stack, which is a use-after-free, and then memory management correctness is guaranteed.

The discipline of caller-allocates-return is awkward in some dynamic cases. It would take a change in low level calling conventions, but you could plausibly have functions that return with the stack extended from where it was when they were called. I’m not holding my breath for that.

It isn’t obvious if having AI coding agents makes discussions of software architecture more or less important!
View quoted post on X
7 replies 1 reposts 18 likes 2.6K views View on X
Shital Shah Verified account @sytelus
I hope Sutton writes his own clarification of his famous essay some day because it has been interpreted so badly by the industry.

The core invariance in scaling laws is the *shape* of the curve, not the actual scale. The actual values on axis is in fact dictated by the efficiency of the algo.

On log log plot, the scaling laws will appear as line. Better algos “tilt” that line.

So algo research was never dead and in fact it yields higher payoff than blind compute scaling. More importantly, scaling is often bottlenecked on having right algos at different parts of the curve. This had been demonstrated so often (ex MoE, Muon, muP, FlashAttention, GQA/MLA).

Unfortunately people doing algo work often get little support as failures are often, time horizon is longer, risks are high. A lot of advancements have therefore came from underpaid (at the time) lone wolves such as Keller Jordan, Tri Dao, Greg Yang and so on.

Finally, timeline has mysteriously threw folks like Noam Shazeer among us who single handedly carried us for many of the most important such innovations. We shouldn’t confuse our luck in this enterprise with a lack of systematic setup of research culture.
Richard Sutton Verified account @RichardSSutton 2d
Brute force computation is not enough for intelligence. Clever new methods are needed. The science of AI is far from finished. IMHO
View quoted post on X
2 replies 1 reposts 18 likes 1.9K views View on X
Shital Shah Verified account @sytelus
There is an alternate universe where Elon Musk stopped obsessing about shaving off another half second for 0-60.

Instead he went all in to produce cheapest possible normie family SUV. Like Henry Ford had done century ago. He relentlessly reviewed every inch of the car and challenged his teams why it cannot be produced 10% cheaper every 3 months. He said he wouldn’t stop until EVs were way cheaper than gas cars.

By 2026, half of the cars in US were transitioned to electric. People didn’t cared about gas prices at the pump. Inflation was under control and blue tsunami didn’t happen.
Shital Shah Verified account @sytelus Sep 28
I really want EV and was eyeing on Tesla for sometime. I want to reduce my consumption of fossil fuel and really don’t care about politics. But then I actually rented Model 3 for a week in Honolulu. OMG it’s such a massive design disaster.

I actually can’t believe such massive design f’ups happened under Elon:

- rear passenger windows don’t go all the way down like cars in 1990s. This was absolute dealbreaker for kids.

- no shade on roof. Again dealbreaker for kids for taking naps on long drives.

- acceleration and sudden stopping is fun for while but is super jarring for kids in rear. Chill mode didn’t fix it and motion sickness was real bad.

- even in world’s top tourist area there are very few chargers. They were always full. Had to pay $$ just to enter parking lot.

- nothing in car seems to have UX affordances. We constantly had to ask ChatGPT how to do usual things, had to stop car and watch youtube howto videos. Divergences didn’t gave any better user experience.

- even charger design sucked, thing kept falling off, randomly stopped charging, sometimes never turned green. Had to baby sit and lose hour in trip.

- no physical controls is positively absolute utter disaster. wtf, seriously! It made everything unsafe and pain.

- most of the UX is occupied with showing you visualization and it only tells you how bad AI is. Person pulling cart shows up as 2 man walking, fire hydrant as cone, etc. We found one error every hour.

- clearence is too low to take on outdoor places where my gas SUV will have zero problem.

I have always wondered why everyone hadn’t switched to Tesla already and still driving cars designed affectively in last century + dealing with dealerships.

I don’t think it’s politics for many.

It’s just that Tesla is not designed for families, for having people in rear seats, for nice smooth comfortable rides. It is designed for middle aged nerds living by themselves who want their twice a year proof that their car can accelerate 100 miles a second or some shit like that.
View quoted post on X
49 replies 34 reposts 1.3K likes 161K views View on X
Shital Shah Verified account @sytelus
A very rare and honest reflections from John that goes beyond “I am leaving XYZ and everything was so great”.

I and John were part of the leadership team at AI Frontiers in Microsoft Research. I left back in summer to do my own thing.

John was easily one of the most insightful senior researchers with unparalleled engineering chops at commanding agents. Those were the good times!
John Langford @JohnCLangford 3d
The End of AI Frontiers hunch.net/?p=13764006 .

There's a bit of a train wreck here that's painful to express. Nevertheless, it's a significant event for me and quite a few other folks involved so I wanted to memorialize a bit.
View quoted post on X