1/ announcing our smallest, most efficient little model is here: muse ember 🔥
built for work that agents repeat all day, it gets you more intelligence per dollar than Haiku, Luna, and all other flash models at any reasoning setting and does it at a third of the cost or less!
New Meta and MIT paper finds reusing a transformer's layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.
🧵 [1/14]: How should we loop a transformer when model size 📦, training compute 🧮, and inference efficiency ⚡ all matter?
Looping is inherently simple: take the hidden representations produced by a transformer (or part of it) and feed them back through the same transformer (or part of it), reusing its weights. 🔁
But every extra loop costs compute. Choosing which layers to loop, how many times, and how to execute those loops determines whether the trade-off is worthwhile.
We explore these choices in our new paper:
“How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference”
Here’s what we found. Thread below 🧵👇
📄 Paper link: https://t.co/kEiYswTrsY
How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference
New paper from Meta.
"Our experiments span models with up to 1B parameters, trained on up to 155B tokens using up to 1.07 × 10^21 training FLOPs, with up to 32 loops. We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained."
"We compare exact weight sharing across 2 loops with small loop-specific low-rank adapters that allow each loop to learn a slightly different transformation. These adapters provide only modest gains when few loops are used, but become consistently more useful as the same weights are reused across more loops"
"Naively applying Parallel Looped Transformers (PLT) in our setting does not preserve the gains from looping: quality can degrade as more loops are added. We find that preserving the current-token representation, separately normalizing it and the recurrent state, and controlling how they are combined substantially improves PLT."
The Muse Glimmer technical report is out: https://t.co/VSLVtwd1Hh
Muse Glimmer is our 30B open-weight model (Apache 2.0), designed for agentic workloads entirely on consumer hardware.
Looking forward to seeing how the community uses our model!
Metas Muse Glimmer tech report is pretty interesting.
Metas Muse Glimmer report describes training agents inside simulated digital lives, complete with 10–14 contacts, weeks of messages, calendars and documents. The agents practice tasks spanning multiple days, while training trajectories containing hallucinated results or unverified claims of success are discarded.
Today we published a technical report on Muse Glimmer, our 30B open-weight model built for agentic tasks on consumer hardware.
It runs at practical speeds on a Mac or PC with a single consumer GPU at 4-bit precision, and leads comparably-sized open models on tool use and multi-step planning.
Read about the model architecture, distillation recipe, quantization and more: https://t.co/H7n9qeDhxn
As promised, Muse Spark is being open sourced today.
This was our first base model we trained under the new MSL org, this open weight version has gone through a lot of careful additional training for open-weight release. It’s an efficient and capable model, try it out!
@finkd Congrats to the team! Amazing to run a model of Muse Spark Open's caliber and scale this efficiently, right from your desk. Mac Studio keeps pushing the boundaries of on-device AI!
The Muse Glimmer Meta Connect talk is out ✨✨✨
Excited to share more technical details about what kept us busy till August when Muse Glimmer was released, a 30B dense open-weight model optimised for local agentic workflows 🤖
Thankful to @yorambac and @maryamfazel for leading this effort and to colleagues from FAIR London, Paris, New York and beyond for sharing the journey 🙏
https://t.co/AVPaWfGbPy
Today, we are launching a preview of our new model, Mistral Large 4 (ML4), aka le Chonk 🐈.
ML4 is a 1T-parameter model with 49B active parameters, trained natively with multimodal capabilities.
It is at the frontier of open models, and by far the strongest open-weight model from the US or Europe.
The RL run behind this preview is still in flight and shows no sign of saturation -- we will release a final version before the end of the month along with the weights of the model.
🧵
1/n