Pinned post
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
