Zonalic

~Synthetic Soul Specialist~
Still, Just a person

[Zonalic @Zonalic]

@slimer48484 Mind going into more details?

[Zonalic @Zonalic]

@ActuallyIsaak @ysu_ChatData How do you determine the threshold? Is it learned, or do you just set a constant across all experts? If so, how do you deal with the initialization? I've tried something similar and ran into issues for the early steps where either every expert activates or none do.

[Chenwei Cui @ccui42]

Introducing Multi-Head LatentMoE 🚀

Turns out, making NVIDIA's LatentMoE [1] multi-head further unlocks O(1), balanced, and deterministic communication.

Our insight: Head Parallel; Move routing from before all-to-all to after. Token duplication happens locally. Always uniform, always deterministic.

It works orthogonally to EP as a new dimension of parallelism. For example, use HP for intra-cluster all-to-all as a highway, then use EP locally.

We propose FlashAttention-like routing and expert computation, both exact, IO-aware, and constant memory. This is to handle the increased number of sub-tokens.

Results:
- We replicate LatentMoE and confirm it is indeed faster than MoE, with matching model performance. (See Design Principle IV in [1])
- Up to 1.61x faster training than MoE+EP with identical model performance.
- Higher model performance while still 1.11x faster with doubled granularity.

📄 Paper: https://t.co/re5ludi0mB
💻 Code: https://t.co/8pHdtN3Z4i

[1] Elango et al., "LatentMoE: Toward Optimal Accuracy per FLOP and Parameter in Mixture of Experts", 2026. https://t.co/cNmJ8tchTF

[Zonalic @Zonalic]

Some of you mfs need to stare at a fire

[Zonalic @Zonalic]

@kalomaze @Infopulsed Different types, either for flash attention 2 and 3, scheduling kernels so they aren't using the transcendental operations at the same time, or for expert sharing, to schedule sending tokens and performing operations more optimally

[Zonalic @Zonalic]

@jpobrien925 @prerat Was gonna post this study. Agreed, if you just wanted it to roughly regurgitate most of the concepts, you could do it with less, but trying to get near 100% recall on the entire corpus would take at least that much, if not more

[Zonalic @Zonalic]

@justalexoki Make necessities cheap and give people more free time

[Zonalic @Zonalic]

@prismaticflow Interested!

[Zonalic @Zonalic]

@lumpfished Tulpas can definitely be accidental, subtle intention and consistent attention can sometimes be enough

[Zonalic @Zonalic]

@the_wilderless That might have been around the time I experienced a very loud short gong-ish sound. It was so loud and potent I was almost in doubt of my surety that it was originating from in my head. Never experienced anything like it before or since.

[Zonalic @Zonalic]

@brundolfsmith Personally, I'd like options to both see past events (like see Friday events when it's sunday. They seem to leave and not come back), and also have a simple button at the top that makes it so it only shows events that are currently ongoing or coming up. Used to scrolling a lot

[Parker Conley @parconley]

Reference works are really cool.

Check out the thread below for my favorites!

[Zonalic @Zonalic]

@liz_love_lace Perfect analogy lmao

[Zonalic @Zonalic]

@gptbrooke Hemp cigarettes! No nicotine, burn clean, can get some with CBD.

[Zonalic @Zonalic]

@gptbrooke I still think tree-weaves are the way to go!

[Zonalic @Zonalic]

@algekalipso I've been saying this for years. Where is adult McDonalds playplace?

[Zonalic @Zonalic]

I personally am more investing in someone doing the other Thing. Maybe we'd actually get downloading bookmarks then

[Zonalic @Zonalic]

Loving the trend of people waiting on someone to do The Thing. Wonderful meme from so many angles

[Zonalic @Zonalic]

Reasoning model this, reasoning model that, how about you find a reason why you'll still be employable in 10 years

[Zonalic @Zonalic]

@MasterTimBlais Incredible analogy haha. Loving the recent stuff