@nljubesic
The ParlaSpeech corpora in Croatian, Serbian, and Polish from the @CLARINERIC #ParlaMint project are now available on CLARIN.SI and @huggingface! They enable exploring parliamentary debates through text AND audio, and development of ASR and TTS for these languages👏View quoted post on X
Well, it's official. YugoGPT 7B significantly beats Mistral and LLaMA 2 and is now officially the best open-source LLM in the world for Serbian & other HBS (Croatian, Bosnian, Montenegrin) languages.
Earlier this summer I was frustrated when I saw how poor the situation is as soon as you leave the English NLP space (w/ few exceptions).
Exactly 57 days ago I came up with this idea while working on the Open-NLLB project and training a machine translation system for HBS.
Why not train an LLM for HBS languages? And so yugoGPT was born.
The problem? I had 1 RTX 3090 GPU on my custom built rig. :) (kindly provided to me by @nvidia back in London!)
Fast forward to today I have 16 80 GB A100s from @togethercompute (whom I seriously recommend - not just for their hardware but also for their support & speed & dedication) and 16 more coming in a week (from a different company -> will share more details then).
I had to learn a ton of new stuff as I was basically a single guy (w/ @nljubesic's tremendous help that saved me a ton of time on the data side) building this whole complex system (shout-out to my Discord community was essential for the Serbian LLM eval project).
And I must say: I learned more over the past 6 months than I did during my whole time back at DeepMind. Unless you're very very senior - you just don't get the level of agency I'm enjoying right now.
If I could do this bootstrapped with 16 A100s, give me a cluster of 1000 GPUs and I'll show you how things are getting done :) (hah!) well more on that later, things are cooking in the background.
I'll just say that I'm already working on the 2nd iteration that'll be significantly better.
As for yugoGPT I’ll be red teaming it and I'll need help from the community over the next period to test it. more updates soon :) keep an eye out
Again big thanks to:
* @togethercompute - for generous help with compute
* @tri_dao for connecting me with together! :)
* @wandb - for generous donation of GPT-4 credits for the Serbian LLM eval project (and also in general for their amazing tool that I'm using for logging in this project! life saver!)
* @smarkovic - for spreading the word!!
* @nljubesic - for help around data he's been working on over the past many years
* and to many individuals & companies from the local region (mostly Serbia) that recently supported me!
---
Results obtained via Serbian LLM eval that I've recently released (yesterday - lol): github.com/gordicaleksa/s…View quoted post on X
There's a new corpora collection on our concordancers: comparable web corpora CLASSLA-web for all South Slavic languages! They sum up to 11B words and 26M texts, all annotated with linguistic and genre information.
➡️clarin.si/ske/#open
➡️Read more: clarin.si/info/k-centre/…View quoted post on X
