Thanks for all the love on the launch ๐ด
Songe update:
TL;DR: bigger models, MoE + multimodal, more control. Next: RL and frontier distillation.
- Bigger models on one GPU: CPU offload and a paged optimizer are in. Qwen3.8-27B now trains on a single 32 GB RTX 5090.
- Supporting MoE and multimodal models: Inkling-2-Mini (26B) trains with its experts kept packed, 16 GB instead of 49 GB.
- More control: go back to any earlier version, undo a Quick Dream, or remove a single preference from the dashboard.
Next:
- Reinforcement learning. Your model builds a sandbox from your kind of work, tries tasks there, and learns from what worked.
- Frontier distillation. Distill a frontier model into your local model, on your own tasks.
Ideas welcome :)
https://t.co/j5fVhObCgQ
@albustime@0xBakeer we use a different algorithm (check out https://t.co/9P7qAvfZyz). it is a lot more robust. needs no curation from you. image data is a little trickier and we will make sure it works soon.
@XaeaZii@0xBakeer this is a good point. we will add the ability to control the signals used for dreaming in upcoming versions.
i don't agree that this doesn't work for agents in companies. perhaps more robust evaluation is needed, but it is solvable
Slides from my talk yesterday @Columbia_Biz Continual Learning in Minds and Machines Workshop: https://t.co/5SX6bekTYr
Covered:
- Metalearning continual learning algorithms
- Key-value memory in the brain
- (briefly) Retrieval-centric DL
+ hinted at new RNN ideas I'm exploring.
The data from our new continual learning benchmark, Continual-ARC, is now on Hugging Face ๐ค
Thanks @NielsRogge for the nudge!
https://t.co/wxbEosEW6e
Your model should dream while you sleep. ๐ด
Today weโre open-sourcing Songe.
Itโs a local training engine that turns your conversations, instructions, and corrections into training data, then fine-tunes your local model on your own tasks and preferences.
It all runs on your own machine, using compute that would otherwise sit idle.
Your data stays on your device. The model that gets better is yours.
pip install songe
https://t.co/QIHc4caigj
Weโre sharing the first phase of our research @constructor_agi. Our mission is to solve continual learning and develop models that learn natively from experience.
Unlike LLMs, humans donโt reread their entire context; with experience, we learn what matters. Our work is a promising first step in that direction: it shows that a model improves when it learns where to look.
Introducing Link: Teaching Attention to have intention.
Link is a learned, sparse read that lets an LLM choose what to read. At multi-million context lengths, it outperforms dense attention on retrieval accuracy by 53% at 4M token context lengths, alongside 9.9x GPU efficiency.
(1/4) why do image models chop every picture into the same square patches? ๐ค (hint: they shouldn't)
following the bitter lesson, we let diffusion models *choose* how to tokenize an image. it spends fewer tokens on the boring background, and more tokens on our faces! ๐งต