Transformers: the closed vs. open-weights battle
Duration 4:22
Nine years after "Attention Is All You Need," the attention mechanism is still at the heart of generative AI. From BERT to GPT-3, from gpt-oss to Mistral 3, from Claude Opus 5 to Kimi K3: this video traces what has actually changed. The new dividing line no longer separates the labs, but the closed models from the open-weights models, which you can host yourself. For a company, that means choosing your provider, your license, and where the model runs.
Chapters
Every chapter below is clickable.
Transcript
In 2017, one paper changes everything: "Attention Is All You Need," by Vaswani and his team. It describes a new architecture, the transformer. Before it, text was processed word by word, in sequence. LSTM and GRU models dominated, but struggled to work in parallel. The transformer weighs the importance of each word according to its context. Wherever it sits in the sentence, it stays connected to the others. This mechanism is called attention. It allows computation in parallel, so faster, over far larger volumes of text. In 2018, Google introduces BERT. Its innovation: pre-train the model on huge corpora, then adapt it to a specific task. Pre-training learns the structure of the language. Fine-tuning then specializes the model: classify a text, answer a question. In 2020 comes GPT-3, from OpenAI. The same principle, pushed to an unprecedented scale, reveals a strikingly fluent text generation. Today, model families keep multiplying. We find GPT at OpenAI, Gemini at Google, LLaMA at Meta, and Mistral at Mistral AI. But the real dividing line has shifted. It no longer separates the labs: it separates closed models from open-weight models. An open-weight model can be downloaded and hosted on your own machines. A closed model stays with its publisher, rented remotely. In August 2025, OpenAI releases gpt-oss-120b and gpt-oss-20b. These are its first open-weight models since GPT-2, under the Apache 2.0 license. On December 2, 2025, Mistral introduces Mistral 3, including the Large 3 model. A European frontier model, installable on your own servers. On the closed side, Anthropic releases Claude Opus 5 on July 24, 2026. Powerful, but accessible only remotely, at its publisher. In China, Moonshot AI opens its Kimi family. K2 ships in July 2025, under a modified MIT license, then evolves into K3 in July 2026. Publishers, licenses, countries: everything sets them apart. Yet all descend from the attention described in 2017. No one has replaced it in nine years. It has only been made larger, faster, and cheaper. Speech, too, has followed the same path as text. On March 23, 2026, Mistral releases Voxtral TTS. An open-weight speech synthesis model, designed for conversational agents. One major obstacle remains: cost. Training a large model like GPT-3 demands enormous hardware, and sometimes reproduces the biases of its data. For a company, the consequence is concrete. Companies can now choose their publisher, their licence, and where the model runs. This is the ground Koaee works on. We design conversational assistants for companies, along this same line of transformers: Mistral, Voxtral and OpenAI. Our assistants speak sixteen languages. A company's customer can therefore write or speak in their own language. The assistant replies out loud, in that same language. This video gives an overview; the full article details everything, and it's in the comments. If AI interests you, subscribe and like the video. See you soon on koaee.ai.