D Ace

AI Researcher at Vizuara ||
In a twisted relationship with LLM ||
Building future by day, sailing Grand Line by night ||
Github:-

[D Ace @Ace_00_7]

First Principle of Programming
Remove all syntax & frameworks, what is programming at its absolute core? It's just taking an input, manipulating that input using logic, & producing output. Every program from basic calculator - massive neural network is just combination of these 3

[D Ace @Ace_00_7]

@AndrewBolis AI

[D Ace @Ace_00_7]

@Siddhant_R_Giri 🐰

[D Ace @Ace_00_7]

Read 2025 survey “A Comprehensive Survey of Mixture-of-Experts”, it’s clear that MoE 2.0 is coming
Dynamic experts, continual learning & Neural OS-like architecture could turn AI into something modular and expandable not a single giant model.

Link:- https://t.co/434g46EWyD

[D Ace @Ace_00_7]

AI doesn’t need bigger models, it needs smarter ones
That’s why Mixture-of-Experts matters

Grizz --> social tasks
Panda --> creative tasks
Ice Bear --> efficiency mode

Only needed “expert” wakes up
Compute stays low, capacity skyrockets

Click the Link:
https://t.co/CwBq0kRZ0W

[D Ace @Ace_00_7]

@Rohan_Sangle02 🫥🫥🫥

[D Ace @Ace_00_7]

AI needs smarter thinking, not just more power

Chain-of-Thought works for simple stuff, but GoT, i.e Graph-of-Thought & not Game of Thrones, tries multiple ideas at once, drops the weak, & picks the best. Faster, smarter, more human-like

Read more here
https://t.co/qe4oZB8kCc

[D Ace @Ace_00_7]

@Rohan_Sangle02 This is the main one 😁😉

[D Ace @Ace_00_7]

@Siddhant_R_Giri Yep 🤧

[D Ace @Ace_00_7]

@karpathy built NanoGPT, and I created NanoKimiK2, a nano version of the Kimi K2 LLM.

🚨Don't miss it🚨
Repo Link:- https://t.co/w9wLyyvyiS

[D Ace @Ace_00_7]

Here we go again 🥱

[D Ace @Ace_00_7]

👨‍🔬 To fellow ML engineers:
If you're experimenting with MoE or MLA architectures, give MUON a try.
Would love your insights, feedback, and experiments. Let's learn from each other.

[D Ace @Ace_00_7]

In past internship, I built my own optimizer from scratch, MUON by referring to @Kimi_Moonshot research paper.

Results surprised me, & it's open source

Repo Link:- https://t.co/w9wLyyvyiS

Here’s a breakdown of why MUON works so well, especially for MoE models.
🧵👇

[D Ace @Ace_00_7]

4. MUON Put to the Test
Model: NanoKimiK2, 8-layer MoE (SwiGLU FFNs, RoPE, MLA attention, 32 experts, top-6 routing)
Baseline: NanoGPT

With MUON:
✅ Stable expert routing
✅ Reduced valley-trapping

→ Trades a bit more compute for much better sparsity & stability

[D Ace @Ace_00_7]

Made Kimi K2 completely from scratch as NanoKimi K2 what @karpathy did for GPT

Sparse MoE transformer (32 experts/top-6, SwiGLU FFNs, RoPE pos enc, MLA attn w/ LoRA-split) vs NanoGPT

Build Muon Optimizer completely from scratch

Link: https://t.co/w9wLyyvyiS
#LLMs #Transformers

[D Ace @Ace_00_7]

@heyshipz AI increases productivity