Alazar M

ugrad @ @Northeastern
sys + theory

[Alazar M @Alaxarr]

@ReubenConducts is this cute paper?

[Alazar M @Alaxarr]

in a long shot, with a breakthrough this does mean a new non auto diff based library for ml?

ie bye bye jax/torch?

😝

[Francois Chaubard @FrancoisChauba1]

Zero Order FTW! just imagine if we had a device that was built for it.. how much better it would do

[Alazar M @Alaxarr]

fantastic work from @industriaalist @
& team!

track 3 is where real fun is!

sadly i ran out of gpus (just when things get exciting lol)

[Samip @industriaalist]

I think the last month might have been the biggest month in (zeroth-order) optimization history.

In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.

Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!

Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.

Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.

Expect a lot more from us over the coming weeks.

[Alazar M @Alaxarr]

@sanketpurandare @AIatMeta lolz

[Alazar M @Alaxarr]

@spicey_lemonade nice!

[Alazar M @Alaxarr]

@cs_serdar you’d be surprised how many of these exist lol

[Mayank Mishra @MayankMish98]

Releasing FLAT: Flash Linear Attention on TPUs 🚀

Pallas training kernels for Linear Attention, Mamba-2 and GLA. On one TPU v6e, forward + backward is up to 3x faster than FLA on an H100 for GLA, and 1.5x faster for Mamba-2.

🧵

[Alazar M @Alaxarr]

@henrylhtsang codex app a banger

[Alazar M @Alaxarr]

@ad0rnai dm closed 🙃

[Alazar M @Alaxarr]

@yaboilyrical lol, first guy i heard use coreweave compute

[Zihao Ye @ye_combinator]

@henrylhtsang Stay tuned for open sourcing cake itself :)

[Alazar M @Alaxarr]

@_arohan_ much can be argued, but all can agree the fruits of llms shouldn't be restricted to cloud or big-token(or the researchers inside them with plenty to burn)

this is also regarding safety/cyber

[insert of starving kid with dripping water from pipe that has water sent to fat dude]

[Alazar M @Alaxarr]

@growlygg me and my agents thank u

[Alazar M @Alaxarr]

doesn't this exacerbate the problem more, as far as I can tell

slop idea + slop delivery = slop idea
slop idea + un-slop delivery = still slop idea

this just makes it harder for humans to differentiate between human thinking and echoes of an stochastic parrot

[Jason Weston @jaseweston]

Claim: we've solved the AI slop problem (!) 💩🧹✨

Blog post: https://t.co/mXuwYiy6rq

🧵1/5
Key idea: take *expert* human writing and learn rubrics that find the gap between experts and models. Train with those rubrics.

We train with RL-XAR (RL with eXpert Aligned Rubrics) & see large performance gains on writing scientific paper sections, Pulitzer prize novel continuations and high quality Wikipedia pages.

[Alazar M @Alaxarr]

@a1zhang btw dms are closed :)

[Alazar M @Alaxarr]

@davidbau for the typical consumer, a mac is still the best intelligence/$

at this rate OS models will be needed to do any kind of med/complex coding/design workload

I’d say worth investing in a good M series, and run local models

btw as a neu student, your lab does awesome work!

[Alazar M @Alaxarr]

@luismolinaab @Prince_Canuma 2nding what @Prince_Canuma said, though Fast option should exist for some of the models!

[Alazar M @Alaxarr]

@growlygg coming

[Alazar M @Alaxarr]

does anything ml sys related happen in boston?

[Alazar M @Alaxarr]

in the 11th hour, but nonetheless just wow

[Magic @magicailabs]

Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. Our new recipe matches DeepSeek V4 Pro’s pretrain using 50x less compute – that’s roughly half the FLOPs used for GPT3, or ~$0.5M on GB200.

https://t.co/uBs5o5Arsu