罗杰斯

AGI @ 池畔 🏖️ Prev. 清华大学

[Mark Saroufim @marksaroufim]

June 9th Researcher Reciprocity License

"if you train on it, you let us generate - reverse terms of use void"

Status quo
1. We teach frontier devs with ICLR/NeurIPS papers, OSS Github contributions
2. They use it to make frontier models
3. Then ban us from exploring our ideas

We need a new license, original thinkers can't be an underclass to a tyrannical researcher fiefdom

[Daniel Auras @rasdani_]

this is the biggest wake-up call to protect and nourish open source AI

if you don't build out sovereign and independent models+infra closed labs will patronize you to an insulting degree

[Lazarz @Laz4rz]

To my future CEO: 👉👈🙄

[JB @JasonBotterill]

Anthropic employees are fucking depressed

[罗杰斯 @dhbrojas]

Please @tenstorrent, deploy DSV4 with 1M context at 500 TSU in the API and I will pay no matter the price 🙏

[罗杰斯 @dhbrojas]

You’re probably not bullish enough on RLM and context management / tool calling as code

[罗杰斯 @dhbrojas]

Chinese providers are on fire

[Hot Aisle @HotAisle]

https://t.co/DISt8UrhX3

"On the flip side, we have the "F-Tier". Providers like https://t.co/w4vKh0dycR, AkashML, SambaNova, and Nebius are clocking in at exactly 0.0% cache hit rates across the models."

[tomie @tomieinlove]

It seems Anthropic revenue has been bolstered by widespread “Who Can Give Anthropic the Most Money” tournaments.

[罗杰斯 @dhbrojas]

@boopdotpng That’s awesome man. I hope that with the ISA documentation progress and simulator a no-TT-dependency Blackhole stack is possible within the next 12-24 months 🤞🏻

[罗杰斯 @dhbrojas]

@Shiwei_Liu66 Great work as always. You guys deserve more compute! I always want to see your stuff tried at larger scales

[Shiwei Liu @Shiwei_Liu66]

🚀 New paper: One LR Doesn’t Fit All for Transformers
Arxiv: https://t.co/vmJC3XKRNU

Transformers look like homogeneous stacks.

They are not.

Modern Transformers are highly heterogeneous: attention layers, FFN layers, embeddings, and
different depths can have very different training dynamics.

But we still give them the same learning rate.

In our new paper, we show that the shape of weight spectrum can diagnose this heterogeneity and turn it into a practical optimizer design: layerwise learning rates.

Weakly trained layers get larger updates.
Well-trained layers get protected.

It works for both AdamW and Muon — and the improvement with Muon is even more considerable.

The result is better module utilization, faster convergence, and stronger generalization — up to 1.5× training speedup across LLaMA/GPT-style models.

[罗杰斯 @dhbrojas]

Are there any labs/researchers working on reducing the hyper-parameter surface of optimisers, large training runs in general?

So much money wasted in ablations!

[罗杰斯 @dhbrojas]

Your university the second you graduate

[Polymarket @Polymarket]

NEW: Nvidia CEO Jensen Huang to join board of prestigious Tsinghua University in Beijing.

[罗杰斯 @dhbrojas]

@JCzarlinski @__tinygrad__ Let me know what you’d like me to try, I may write a couple blog posts

[罗杰斯 @dhbrojas]

I will definitely regret this but the @__tinygrad__ backend is not going to write itself...

[罗杰斯 @dhbrojas]

@igorjmichalak @tenstorrent Interesting, what’s the use case?

[罗杰斯 @dhbrojas]

@__tinygrad__ I don’t plan on using anything from @tenstorrent beside the driver. But yeah, the first part will be documenting everything. I’ll see how much can be extracted from the TT codebases. It will probably (P=0.95) fail, but hey, it’s fun to try!

[罗杰斯 @dhbrojas]

@__tinygrad__ Do you ever bring it up to @jimkxa? What does he say?

[the tiny corp @__tinygrad__]

@dhbrojas Looks like it's still missing a lot of Blackhole. It's crazy to me that $10M+ tapeouts are done without a full spec of each instruction + cycle accurate simulator.