Pinned post
Dan DiPietro
Verified account Applied Compute@danyulol
building @appliedcompute. founder, previously at Jane Street.
315 Following 305 Followers 78 Posts
Latest posts 5
SFT as a training technique is generally very underrated. People have gotten this idea that you can’t SFT without catastrophically degrading a model. This is completely untrue. Done well, you can often come close to RL performance with an order of magnitude less data (1/n)
Low-precision is the future. Frontier open models like 4.1 Flash and K3 already serve inference with low precision. I think the same thing will happen soon for training. Training actor forwards, backwards, and even optimizer state will move to 8-bit and 4-bit regimes.
0:23
We would need 35 Enrons to cancel out the public wealth creation from Amazon's early IPO.
SOX is a great example of how seemingly well-intentioned regulation can have horrific unintended downstream consequences, e.g. reducing middle class participation in an entire category of wealth creation.
SOX is a great example of how seemingly well-intentioned regulation can have horrific unintended downstream consequences, e.g. reducing middle class participation in an entire category of wealth creation.
Sometimes verifiers are actually very expensive to run, and it’s much cheaper to use an LLMaJ instead. One example: suppose you’re RLing a model to build HFT strategies, and each rollout requires an extensive event-driven backtest that takes hours to complete. (1/n)