Great work! This such resonates with the philosophy behind SimpleDesign, for protein sequence–structure modeling 🧬arxiv.org/abs/2609.03377
SimpleDesign feeds amino acid seq (text) and 3D coordinates (vision) directly into a unified Transformer, with modality-specific layers and joint attention. no structure tokenizer; no pretrained structure encoder.
from different domains, but same principle 😀
SimpleDesign feeds amino acid seq (text) and 3D coordinates (vision) directly into a unified Transformer, with modality-specific layers and joint attention. no structure tokenizer; no pretrained structure encoder.
from different domains, but same principle 😀
how should multimodal models encode visual input?View quoted post on X
→ pretrained encoder?
→ encoder-free?
→ before the LLM?
published models bundle these choices together. we trained 200+ models from scratch to see which of these choices matters
only one of them does
🧵 [1/n]