Pinned post
Whatever you make of already-clean.goodfire.com, it has brought compositionality and generalisation back into the spotlight. I’d like to see more rigorous work testing these ideas.
The Random Hierarchy Model is one example. By controlling the data's compositional structure, we recover several key features of deep learning: advantage of depth, scaling laws, linear representations and circuits.
Well-motivated toy models are invaluable: they help us explain how these phenomena fit together and make falsifiable predictions about real data.
Fun fact: @nickcammarata and colleagues’ earlier “Zoom In” work inspired the RHM, back when we were still thinking about cats and dogs.
The Random Hierarchy Model is one example. By controlling the data's compositional structure, we recover several key features of deep learning: advantage of depth, scaling laws, linear representations and circuits.
Well-motivated toy models are invaluable: they help us explain how these phenomena fit together and make falsifiable predictions about real data.
Fun fact: @nickcammarata and colleagues’ earlier “Zoom In” work inspired the RHM, back when we were still thinking about cats and dogs.