ali

research @baseten

[ali @waterloo_intern]

https://t.co/aivc7B4A7Z

[ali @waterloo_intern]

@zainhas @pavneet1990 “im not going to open my laptop” 😭😭

[ali @waterloo_intern]

your competition is everywhere 👀
@pavneet1990

[ali @waterloo_intern]

to be completely honest, when I read the initial blog post and the figure-to say that I was skeptical would be a … severe understatement.

to
- claim that you could come up with your own architecture
- that said architecture is more efficient than open-source architectures
- that you did not do any distillation from any of the frontier models (need not be from claude, could have been from kimi/deepseek)
- AND that you did NOT purchase any of the high-quality data sets, training exclusively on github data (???)
- AND THAT YOU OUTPERFORMED KIMI (?????????)
- AND ONLY 5m WORTH OF COMPUTE (assuming 50/50 split of tokens and compute spend (to be honest, probably a larger token spend than compute spend, but that's just my guess))

I mean, you have to be a little bit (a lot) insane to think that I would believe this.

then he asked us to evaluate it

[Behnam Neyshabur @bneyshabur]

We asked Baseten to independently evaluate @mirendil’s base models. We shared our model endpoints so they could evaluate them on their own private codebases. Mirendil-Base-Large ended up being the best of all the models they tested!

Thank you, @baseten, for being a great partner!

[ali @waterloo_intern]

@hamostaf04 @askchapter @CobiBGantz @darshan1394 THE ONE
THE ONLY
THE MAN THE MYTH THE LEGEND

[ali @waterloo_intern]

@mj_bilodeau @philipkiely COOKED 😭😭

[ali @waterloo_intern]

the models yearn to forget information…by construction

[Philip Kiely @philipkiely]

Scaling past a trillion parameters requires inference engineers to solve 3 issues:

- Width: Routing between hundreds of sparse experts
- Depth: Mixing global and local linear attention
- Length: Processing 1M-token sequences

I asked Ali and Bryce to draw it up. They delivered.

[ali @waterloo_intern]

@philipkiely what a chat

[ali @waterloo_intern]

@maxkirkby @baselabs

[ali @waterloo_intern]

@thealexker https://t.co/WhU6VR48Rx remains timeless

[ali @waterloo_intern]

@danhergir hey Daniel's twitter bot

[ali @waterloo_intern]

@RaghavMehtaM1 💯

[ali @waterloo_intern]

@AayaanNaqv87775 watch out for the big one @philipkiely

[ali @waterloo_intern]

@thealexker v v well-written

[ali @waterloo_intern]

@shreyasnsharma reading rn, re- confound of memorization in pre-training...idk how we'd account for that tbh

[ali @waterloo_intern]

@awfrd2 interesting, will tal

[ali @waterloo_intern]

@MikeReeseX working on it, will update

[ali @waterloo_intern]

@kwal0203 https://t.co/5WlEHoUxb2

[ali @waterloo_intern]

@ayo_oguntola imo humans are more efficient... i didn't forget how to code after taking calculus 1

[ali @waterloo_intern]

The jaggedness of LLMs, and the excuse that their capabilities emerge from RL on environments curated for revenue potential, is proof enough to me that there is no such thing as RL generalization. It makes me very bearish on RL being the algorithm that gets us to AGI.

And if we do get there, it will be a disgustingly inefficient and unsatisfying success, because we will have done so by mobilizing the world economy to create environments for every fucking task, such that nothing is out of distribution. If every new task needs its own environment, what exactly is generalizing?

If that’s the path, environment/data selling companies will be the most valuable in existence, second only to the labs. This is stupid. The concept of RSI is stupid. Intelligence, by definition, is generalization...but all we have is omniscience.

[ali @waterloo_intern]

@ayo_oguntola imo humans are more efficient... i didn't forget how to code after taking calculus 1

[ali @waterloo_intern]

@oneill_c i signed up to do research but i got an existential crisis...it's all a lie

[ali @waterloo_intern]

@1a1n1d1y who's building said arch?