bezdarnost

Opinions are from the voices in my head(they achieve consensus faster than I do).

[bezdarnost @bezdarnost_brr]

@ting_ @huggingface I love you guys so much!!

[bezdarnost @bezdarnost_brr]

@AIQuanting @JiaweiYang118 800 is the most popular option

[bezdarnost @bezdarnost_brr]

@dogacel0 inductive bias is all we need

[bezdarnost @bezdarnost_brr]

Fable 5.2 spend minimum 15 minutes for any question🥴

[bezdarnost @bezdarnost_brr]

@XamituChidirif @gpujack its definitely $6.9 per gpu per hour with 2 years commitment

[bezdarnost @bezdarnost_brr]

where is my dear Qwen4😭😭😭

[bezdarnost @bezdarnost_brr]

@VukRosic99 2, 3, 7?

[bezdarnost @bezdarnost_brr]

@rikiyatakehi @a1zhang @lateinteraction @jayden_teoh_ I am really surprised, I could image it only in my dreams

[Ziming Liu @ZimingLiu11]

There is a "bug" in commonly used activation functions (ReLU/SiLU/GeLU etc.) ---- The mean of post-activations is not zero.

This bug, however small it seems, leads to pathologies of training dynamics, including loss spikes & representation overshooting.

By removing the bug, we get improved performance in LLMs and ViTs.

The science of AI can never be under-estimated.

[bezdarnost @bezdarnost_brr]

@mdzor @jaga_prasanna compute and token budgets? (access to H200s for unlimited iterations and brute force)

[bezdarnost @bezdarnost_brr]

@Leapofcode nvidia?

[bezdarnost @bezdarnost_brr]

@mlgeeks_18 @jino_rohit I think stealth startup

[bezdarnost @bezdarnost_brr]

@ostrisai what's about gFID?

[bezdarnost @bezdarnost_brr]

@KBlueleaf the greatest all the time

[bezdarnost @bezdarnost_brr]

- FlashGDN-2(probably will try also FlashKronLSTM)
- My own open source python library (mostly Rust code) for image processing, I want to leverage a lot of JPEG XL features for better performance
- EMBER, open source ent-to-end T2I diffusion model with small amount of compute(probably I will close this project and publish this model under another name, more details in future)

[bezdarnost @bezdarnost_brr]

@ianhildy similarly as GPT6.1-Sol and GPT6.1-Terra or do you believe that Terra>Sol?

[bezdarnost @bezdarnost_brr]

@ianhildy Because it's indicative? (Argon's training set might have included the exact same or highly similar tasks to those in open benchmarks), whereas Carbon might already contain new private tasks(?) (it's not certain, but there's a chance).

[bezdarnost @bezdarnost_brr]

@toyxyz3 how can I join this slack channel?

[bezdarnost @bezdarnost_brr]

@yingfachen very interesting results!

> stronger linear attention

I didn't see ablations with GDN, Mamba-1/2/3 and other

[Albert Gu @_albertgu]

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.

Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?

Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.