Pinned post
We just introduced a new physics-inspired extraction attack on diffusion models!
Had a lot of fun working on this with @SteMartiniani.
Excited to explore the implications for privacy and model auditing.
📄 arxiv.org/abs/2606.24000
Had a lot of fun working on this with @SteMartiniani.
Excited to explore the implications for privacy and model auditing.
📄 arxiv.org/abs/2606.24000
arxiv.org Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models We introduce cyclic denoising -- repeated forward and reverse diffusion at controlled noise amplitudes -- as an extraction attack for image diffusion models. Inspired by random organization in... 🚨 We introduce a new attack capable of efficiently extracting training data from diffusion models. 🚨
The attack is completely unsupervised. It does not rely on the text paired with the training images, prompt engineering, training-set lookup, gradients, or access to the model weights. If you have access to the model input you can audit it. All on a single GPU.
🔁 All you need is the ability to run the diffusion process in cycles. Take an image. Add noise partway through the diffusion process. Denoise it. Feed the result back in. Repeat.
At first, the trajectory wanders through generic samples. The images change from cycle to cycle, and they look like ordinary outputs from the model rather than memorized training examples.
Then something striking happens: the trajectory gets trapped. 🪤
It enters a memorized state and stays there for hundreds of cycles. After each noise–denoise cycle, the image comes back essentially unchanged. Those long plateaus in similarity are the signal. 💡
We call these *ultrastable memories.* Many correspond to memorized training content: stock-photo templates, product images, logos, watermarks, web-crawl artifacts, and repeated e-commerce scenes.
Cyclic denoising gives us a new way to audit diffusion models for privacy, copyright, and trademark leakage. If a model can be driven into protected images, brand marks, watermarked assets, or commercial templates without the training caption, the usual story about “just learning the distribution” becomes harder to maintain.
The punchline: memorization in diffusion models is not confined to a few lucky prompts or isolated examples. It is extensive. And with the right cyclic drive, it becomes directly observable. 🔁 👀
👨🔬 Work led by Rishab Sharma @ssh_rishabh at NYU
📖 Paper: R. Sharma, S. Martiniani, "Cyclic Denoising Reveals Ultrastable Memories in Diffusion Models", lnkd.in/gfEYezGuView quoted post on X
