Just saw that the LLMs-from-scratch repository passed 100,000 stars on GitHub!
This is super cool and motivating. I am really happy to see that this open-source repo has helped so many people.
Thanks also to everyone who shared ideas and opened PRs with improvements!
Of course, I plan to keep adding new material, including new attention variants and architectures (while bigger projects like RL and Reasoning From Scratch live in their separate repositories).
I am also currently working on a larger applied custom “small” LLM project. It has been keeping me super busy this month, but I will share more on that soon.
If you are new to it, some of the highlights include
1. Of course, the complete code path from tokenization and attention to pretraining, classification, and instruction fine-tuning, etc. All of it FROM SCRATCH, of course! (RL lives in a companion repo.)
2. From-scratch implementations of Llama, Qwen, Gemma, and Olmo (smaller variants that run locally and can be plugged into the training scripts).
3. From-scratch implementations of attention alternatives and other architecture components, such as GQA, MLA, sliding-window attention, Gated DeltaNet, DeepSeek Sparse Attention, cross-layer KV sharing, and mixture-of-experts
4. Materials on KV caching, training performance, memory-efficient weight loading, DPO, evaluation, and LoRA
So, if you don’t have any weekend plans yet, happy tinkering!
剛剛聽到 @toomore 提到 Cloudflare Wallet,馬上去搶個名字。
I just reserved my Cloudflare Wallet tag: https://t.co/5noabR2ahr.
Reserve yours now at https://t.co/tT4K5Uhsld
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
https://t.co/Ghebq5mG30
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
I never imagined I’d see this so soon but GPT-5.6 now produces health answers that are rated more highly than those from human doctors in blinded tests on medical benchmarks. All while similar models like it are being integrated into apps such as Instagram and WhatsApp, giving them the potential to reach billions of people.