Twitxy Saved

Mike Lewis

@ml_perception

Llama3 pre-training lead. Partially to blame for things like the Cicero Diplomacy bot, BART, RoBERTa, attention sinks, kNN-LM, top-k sampling & Deal Or No Deal.
Seattle Joined September 2019
257 Following 8.6K Followers 281 Posts

Latest posts 5

Mike Lewis @ml_perception
tldr; you can go a long way in pre-training by (1) curating amazing data, (2) using a lot of FLOPs, and (3) otherwise not screwing up. All three are harder than they sound, so read the paper... That said, I'm amazed by our progress since Llama 3 - expect big things from Llama 4!
Mike Lewis @ml_perception Jul 23, 2024
So excited for the open release of Llama 3.1 405B - with MMLU > 87, it's a really strong model and I can't wait to see what you all build with it! llama.meta.com Also check out the paper here, with lots of details on how this was made: tinyurl.com/2z2cpj8m
View quoted post on X
4 replies 15 reposts 164 likes 26K views View on X
Mike Lewis @ml_perception
Excited to see the open source release of FAIR's early fusion multimodal LLMs!
AI at Meta Verified account @AIatMeta Jun 18, 2024
Today is a good day for open science.

As part of our continued commitment to the growth and development of an open ecosystem, today at Meta FAIR we’re announcing four new publicly available AI models and additional research artifacts to inspire innovation in the community and help advance AI in a responsible way. More in the video from @jpineau1.

What we’re releasing:
🦎 Meta Chameleon
7B & 34B language models that support mixed-modal input and text-only outputs.

🪙 Meta Multi-Token Prediction
Pretrained Language Models for code completion using Multi-Token Prediction.

🎼 Meta JASCO
Generative text-to-music models capable of accepting various conditioning inputs for greater controllability. Paper available today with a pretrained model coming soon.

🗣️ Meta AudioSeal
An audio watermarking model that we believe is the first designed specifically for the localized detection of AI-generated speech, available under a commercial license.

📝 Additional RAI artifacts
Including research, data and code to measure and improve the representation of geographical and cultural preferences and diversity in AI systems.

We believe that access to state-of-the-art AI creates opportunities for everyone – not just a small handful of Big Tech companies. We’re excited to share this work and to see how the community learns, iterates and builds using this technology.

Details and access to everything released by FAIR today ➡️ go.fb.me/tzzvfg
View quoted post on X