Alec Radford

What I've been working on for the past year! blog.openai.com/p/7fa97c36-611…

Inspired by CoVE, ELMo, and ULMFiT we show that a single transformer language model can be finetuned to a wide variety of NLP tasks and performs very well with little tuning/tweaking.

This is a really fun live experiment with twitch chat predictably oscillating between love and hate based on the sample.

By the way - I think a valid (if extreme) take on GPT-2 is "lol you need 10,000x the data, 1 billion parameters, and a supercomputer to get current DL models to generalize to Penn Treebank."

The DL CV community is having a "oh wait, bags of local features are a really strong baseline for classification" moment with the BagNet paper.

This has always been clear for text classification due to n-gram baselines. It took an embarrassingly long time for nets to beat them.

Nice discussion of the progress in NLU that's happening with BERT, OpenAI GPT, ULMFiT, ELMo, and more covered by @CadeMetz in the @nytimes I'm super excited to see how far this line of research will be able to get in the next few years!
nytimes.com/2018/11/18/tec…