Kirill Pavlenko

AI R&D @nebiusai

[Kirill Pavlenko @short_cast]

It really fucking worked. Don't trust cudnn attention backward pass, guys.

[Kirill Pavlenko @short_cast]

Literally me

[Kirill Pavlenko @short_cast]

Literally me

[Kirill Pavlenko @short_cast]

when I was a student, I wondered how my advisor could work with so many students at the same time. today, when I navigate 5 projects with codex, I still wonder the same thing. but now I __really__ wonna know.

[Kirill Pavlenko @short_cast]

@norpadon I bet you read less english prose than you read code. so, your pretraining corpus is weaker.

[Kirill Pavlenko @short_cast]

As an ML engineer, reading plots is by far my biggest advantage over current AI agents. I just do it 10-100x better, not even close. I also think this is the biggest reason why autoresearch is not exploding that much yet.

[Ryan Carson @ryancarson]

Every time you think you need a dashboard to look at data, stop yourself.

Do this instead:

1. Ask your agent to make sure that you have all the data to analyze something actually stored in the database.

2. Ask your agent to write a skill to gather that data.

3. Ask your agent to do the analysis and create a temp and throw-away HTML dashboard to answer the question(s) that you have

In my experience, every dashboard that I've created gets less and less use over time and decays.

It's much better to make sure your agent can get the data you need and answer the questions you have, on-demand.

[Grad @Grad62304977]

One thing I don’t quite get on the PPO vs GRPO stuff is that PPO is much better for long horizon but shorter horizons it doesn’t matter much
Argument being the longer the horizon, the more sparse the signal, the bigger the group size u need, while PPO can give u better signal for equal or cheaper compute.
But it’s not obv to me that the value models job of giving the token level advantages is harder and harder the more long horizon u go

[Kirill Pavlenko @short_cast]

@norpadon yeah, it's the first chapter of the book, we assume no sequence parallelism :) in practice, I think you're right that we can reasonably scale sequence chunking without hitting comm overhead. although I'm not sure what happens after 256k; never tried.

[Kirill Pavlenko @short_cast]

In this part of jax scaling book, they discuss how it's reasonable to assume num_tokens < matrix_size. But in modern applications, the opposite is true! So, matmul intensity is actually DF/(D+F). That does not change the conclusion, though. matmul is compute-bound anyway.

[Kirill Pavlenko @short_cast]

@norpadon MoE and any long-context training, both of which are default nowadays.

[Kirill Pavlenko @short_cast]

Finally started reading jax scaling book. What a read

[Kirill Pavlenko @short_cast]

Here is a possible scenario

> US gov will start regulating closed-source models more aggressively
> Every business will start switching towards OSS models hosted via US-based cloud providers.
> US gov will come for cloud providers to regulate them as well
> China Inference services catch up and everyone will switch to them
> Total China win
> US gov and US companies panic
> Regulations relax
> We are back to the current state of things with somewhat equally strong US and China

[Kirill Pavlenko @short_cast]

Lil’logs are the best. I reread some of them each half a year.

[Lilian Weng @lilianweng]

A super long overdue (3+ years?) post on scaling laws.

Compute is expensive. Scaling laws are a way to help us reason about the optimal compute allocation between data and model size before committing to a large run.

The post covers what scaling laws predict, how compute-optimal allocation works, why Kaplan et al. and Chinchilla disagree, and how data limits + fitting details make extrapolation tricky.

https://t.co/HP26eJvjHB

[Kirill Pavlenko @short_cast]

You can train a speculator on your data to make the fastest endpoint for your particular case. absolutely free lunch.

[Nebius Token Factory @nebiustf]

Today we’re launching Custom Speculator Training in Nebius Token Factory.

Train workload-specific draft models from your own data and deploy them alongside the base model.

Generic drafters are a good start, but production traffic is rarely generic 👇

[Kirill Pavlenko @short_cast]

You kinda get agi-pilled when you realise there is no math dataset to train on, that a small-ish Qwen cannot solve with 90% resolve rate already. also, this validation 🙈

[Kirill Pavlenko @short_cast]

I have the same problem people have with Instagram, but it's the wandb charts.

[Kirill Pavlenko @short_cast]

funding justintv is reverend mother's tier foresight.

"Whatever you build couldn't possibly sound more preposterous than a startup we funded in 2006 called https://t.co/9b0KJ4WH8f. It consisted of one guy, Justin Kan, walking around with a camera on the side of his head, live streaming everything that happened to him. But this company ended up doing quite well. In fact you've probably heard of it, but under its new name, Twitch."

[Paul Graham @paulg]

Between 1982 and 2020, the number of the 100 richest Americans who got rich from inheritance decreased from 60 to 27. And yet on the left they think the mid 20th century was the good old days, because economic inequality was lower then.

https://t.co/ggPKu1gN8u

[Kirill Pavlenko @short_cast]

@giffmana If the carry matchup is decent, I would actually build aghs anyway and just kill everyone else. Aghs is that strong imho, no need to grief yourself with useless orchids

[Kirill Pavlenko @short_cast]

@kalomaze oh, I did not know that!

[Kirill Pavlenko @short_cast]

Man, this is THE BEST BEEF EVER AND I WANT MOOORE (i am learning so much!)

[Keller Jordan @kellerjordan0]

I have some mixed feelings about this result:

On the one hand, it's genuinely impressive. I didn't know that Shampoo could be configured to perform this well on the benchmark.

On the other hand, the way this performance boost was achieved seems difficult to call "Vanilla," for the following reason:

According to @_arohan_, the boost depends upon fixing a numerical linear algebra issue that he observed to occur in my initial standard DistributedShampoo run. He fixed the issue by enabling the flag rank_deficient_stability_config=PseudoInverseConfig().

Here's the problem: This is an undocumented flag. It is contained within the 12,000-line DistributedShampoo codebase, but it does not appear in any user-facing documentation.

As a result, if someone tries to train a model using DistributedShampoo without either (a) knowing about this special undocumented flag or (b) being prepared to detect and fix the numerical linear algebra issues that may occur without it, then they won't be able to achieve @_arohan_'s level of Shampoo performance. This level of effort would be considered atypical for mere hyperparameter tuning.

--
[Note on Muon baseline in plot below: Rohan's post compared Shampoo to a slightly undertuned Muon baseline from 2026/05/01, which reached the target loss in 3375 steps. This resulted in a 50-step gap between Shampoo and Muon. In the figure below I'm using the up-to-date 2026/05/03 baseline, which reaches the target in 3325 steps. This results in the step-counts exactly matching between Muon and the tuned/stabilized Shampoo variant.]