vinjai

building the @appliedcompute model factory / previously AI @Watershed, math @Stanford

[vinjai @vinj_ai]

hardly a day has passed in the last year that I haven't thought about Amdahl's Law

[vinjai @vinj_ai]

OPSD is a really cool method in theory, but it's actually quite hard to make it work in the real world, beyond simple benchmarks. The primary bottleneck is producing high-quality hints. We've learned how to do this over the past few months, as our researchers trained and shipped several production models for customers with OPSD and its variants.

Now, our Platform Research team has productionized some of those hard-earned learnings. We've still got a ways to go, but there are clear signs that OPSD in AC2 is becoming a critical tool in the toolbox for Continual Learning That Actually Works.

[Applied Compute @appliedcompute]

We're sharing a key step towards enabling Continual Learning at Applied Compute.

In On-Policy Self-Distillation, the teacher model supervises training with privileged information: a hint. The training signal depends heavily on hint quality, but manually refining hints across runs is impractical, and isolating their individual impact is difficult. We instead optimize hints automatically using efficient proxies for their eventual training value.

[Ben Scharfstein @benscharfstein]

Feeling pretttty good about building the post-training and inference cloud for open models right about now

[lucca @lbz____]

gpu-burn only stresses fp64, fp32 and tf32

brrr stresses low precision formats too

https://t.co/tYsJAWkq9B

brrr is also way more sensitive to GPUs being unusable than gpu-burn. if you can't grab cuda contexts, gpu-burn passes, brrr fails

[Rahil Verma @RahilVerma13]

At Applied Compute most of our RL and eval pipelines grade with an LLM judge. These produce one result per rubric criterion, typically a paragraph of text accompanying a score. For some complicated tasks, grading can be a slow, pricey part of the loop.

We swapped a grader on a code QA benchmark in AC2 for a decision grader: one call, a yes/no probability for every criterion at once. On 100 sample tasks (407 criteria), same answers:
→ ~32× cheaper
→ ~8× faster
→ 94% agreement with the LLM judge

And it’s even better than the original judge:

When the decision grader was confident, it matched the LLM judge 99% of the time. When it wasn't, the decision grader indicated low confidence (30-70%), pointing to ambiguous rubric items. This allowed us to identify noisy tasks and make the dataset cleaner.

A cheap, fast, calibrated grader means you can grade more rollouts, ask more questions per call, and downweight or review the shaky calls instead of training on them.

[vinjai @vinj_ai]

Pragmatic, well-reasoned piece that is genuinely grounded in reality:

"Given today’s risks, frontier labs need to run like nuclear-power plants or busy airports, with layers of redundancy and careful, time-consuming planning, so that the occasional and inevitable human error does not open a door to disaster. Right now, AI companies don’t know how—but other people do."

"...stronger incentives for safety—coming from outside the company—are a big part of getting this right."

[Adrienne LaFrance @AdrienneLaF]

New in The Atlantic:

@dgrobinson resigned this week. He was among the longest-tenured employees at OpenAI—and oversaw safety reports on 12 frontier launches.

He is very worried: “The time for trial and error is over.”

You can read his essay here:
https://t.co/DQlszrc0ru

[Ali Partovi @apartovi]

I'm starting a podcast, "Confession," where I invite tech veterans to share their painful lessons.

We all connect through shared pain. Hear founders & CEOs like Satya Nadella (Microsoft), Dara Khosrowshahi (Uber), Luana Lopes Lara (Kalshi), and others share their stories and lessons learned.

Subscribe to Confession on...
✅ Spotify: https://t.co/ivOCa9wYK5
✅ Youtube: https://t.co/Ofo7boNWfg
✅ Apple: https://t.co/i8SFYF1CZC

@SatyaNadella @Dkhos @luanalopeslara

[vinjai @vinj_ai]

Alignment is everyone's problem, and we aim to give everyone the tools to solve it for their own use case. As someone who's been in/around the AI safety space for almost 10 years, I'm really excited that we're seeing genuine pragmatic ROI for every company to invest in the alignment of their models with systems like Ari.

[Evan Ellis @evandavidellis]

We're adding reward-hacking monitors into AC2. Every RL run gets checked for cheating, and our research agent, Ari, automatically investigates the flags.

After a model discovers an exploit, it can become its default strategy within a few training steps. We used to hunt for reward hacking manually, reading traces and running one-off agent queries. This was error prone and didn't scale. We decided to automate detection instead.

[vinjai @vinj_ai]

h/t @dyu1729

[vinjai @vinj_ai]

all the people who worked on gemini 4 argon

[vinjai @vinj_ai]

Cheap, scalable, and expert-informed trace analysis is a huge unlock for aligned training and inference!

[Bryan Lee @_brylee10]

I implemented a system in @appliedcompute’s platform for automated failure mode clustering with Jev to surface errors at an even larger scale than before.

RL training produces billions of tokens in traces. I always manually read many traces to understand model behavior, but finding agent failures (like reward hacking / hallucinations) at scale is easy to miss without automation. Here’s how it works:

[Evan Ellis @evandavidellis]

Universal classifiers are great but not CoT monitorable. I'm worried that we'll see a race to the bottom to shove more computation inside of a single forward pass, the very thing that is concerning about looped transformers. The next generations will probably be classifiers with dynamically adjusted loop counts depending on the difficulty of the task. Even though they'll spend similar amounts of compute reasoning as CoT models, their reasoning will be opaque

[Rhythm Garg @rhythmrg]

One of the most interesting unsolved problems in RL is how to train directly on production traces.

We know how to do this reasonably well when the feedback is instructive or prescriptive: distill from a strong teacher that can look at the trace and tell the model what it should have done instead.

But a lot of the most valuable production feedback is evaluative: did the task succeed, what score did it get, did the user accept the result, etc.

Normally you would train against this kind of feedback with environment replay (e.g. GRPO): rerun the policy, sample multiple trajectories, and optimize against their relative rewards. The problem is that for many real production traces, you can't recreate the exact environment state that generated the trace.

That's what makes algorithms like Flash-Reinforce, SAO, and BPCO so exciting – they point toward learning from one trajectory per prompt without needing to recreate the environment and generate a fresh group of samples.

If these methods work at scale and on real production workloads, they could become an important building block for continual learning

[Yash Patil @ypatil125]

The hyperscalers didn’t just build businesses renting out CPU compute. They built extremely valuable products around data, networking, security, storage, etc. We’re now building around AI accelerators, but the playbook of building value add software on top is the same.

Turns out GPUs are really good for running AI models. That is why inference engines, which take a trained model and compute and produce intelligent tokens that can do real work, have become the first piece of this software stack. But, there is so much more to build than just this!

Training helps you turn compute into better models that produce more valuable tokens. Routing helps you pick the right model to generate those tokens at the right cost and quality. Security monitoring helps you check what’s going into and coming out of those models. Then there’s agent observability, context management, sandboxing, etc. All of these things will be part of the new AI stack and the opportunity is much bigger than just serving a model.

This definitely isn’t a winner-take-all market, whether we’re talking about open models vs. the frontier or the players within each category. Customers want choice, flexibility, and access to the fundamental building blocks to create their own systems.

Constrained GPU supply will actually act like a regularizer and draw this fight out longer. Customers are looking for both compute capacity AND value add on top of it. When someone doesn't have capacity, you go somewhere else. That means more players get exposure to customers and the opportunity to address value add.

Getting a customer because you have available GPUs is different from keeping them because your software is better. In the limit, the value add on top of the GPU will win out. The companies that do the best job building that software AND verticalize the fastest to own everything from chip to token will take the lion’s share. It’s important to be building for that now, even when the immediate customer need is just more compute.

[banteg @banteg]

>be me
>discover effective altruism
>apparently normal charity is inefficient
>why donate to random sad thing when spreadsheet can tell you optimal sad thing
>fair enough
>buy mosquito nets
>save lives
>numbers look good
>feel powerful

>couple years later
>someone asks an innocent question
>why only count people alive today
>huh
>future people matter too
>obviously
>my grandchildren shouldn't matter less just because they haven't spawned yet
>reasonable.jpg

>keep following logic
>what about their grandchildren
>also yes
>what about people in 500 years
>sure
>5000 years
>why not
>500 million years
>starting to get weird but morality is morality

>open calculator
>humanity could survive for an astronomically long time
>could colonize galaxy
>could have trillions upon trillions of descendants
>maybe digital people too
>maybe simulated civilizations
>maybe dyson spheres full of happy uploaded minds
>calculator starts smoking

>realize currently living humans are rounding error
>8 billion people suddenly looking extremely beta
>future contains potentially 10^something people
>can't even fit beneficiaries in google sheets

>new moral priority unlocked
>protect the long-term future

>stop thinking in units of "people helped"
>start thinking in "fraction of cosmic endowment preserved"

>malaria?
>terrible
>but only kills existing humans
>AI extinction could delete the entire light cone
>nuclear war could permanently derail civilization
>bad institutions could lock in terrible values for ten million years
>someone invents wrong constitution in 2140
>quadrillions suffer
>better fund governance workshop now

>friend says maybe we should improve hospitals
>explain opportunity cost
>friend says hospitals are full of actual sick people
>explain scope sensitivity
>friend stops inviting me to dinner

>need to decide what to fund
>easy
>expected value

>suppose project has one in a million chance of preventing extinction
>sounds tiny
>but extinction destroys 10^50 future lives
>multiply
>mother of god

>$10 million project has expected value of several galaxies
>charity evaluation complete

>someone asks where the one-in-a-million number came from
>expert judgement
>which expert
>us
>how calibrated
>extremely thoughtfully

>reduce estimate to one in ten million to be conservative
>still beats curing cancer by 38 orders of magnitude
>epistemic robustness achieved

>someone says maybe project doesn't work
>assign 20% chance
>still astronomical

>maybe project makes problem worse
>assign 5% chance
>still astronomical

>why 5
>because 30 felt pessimistic

>publish 46-page report
>contains seventeen sensitivity analyses
>every sensitivity analysis begins after assuming intervention has positive sign

>critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities
>yes
>that's literally why it's important

>critic says the uncertainty might be structural rather than numerical
>make probability smaller
>critic says no, I mean maybe your model is wrong
>make probability smaller again
>critic begins rubbing temples

>discover AI safety
>perfect longtermist cause
>AI might kill everyone
>or create utopia
>or seize galaxy
>or tile universe with paperclips
>or create billions of conscious software minds
>finally a problem with numbers big enough for me

>start AI safety nonprofit
>mission: prevent dangerous AI
>hire smartest people available
>smartest people immediately start building better AI to understand dangerous AI
>interesting

>we must understand capabilities to understand safety
>we must scale models to study alignment
>we must race ahead so less responsible actors don't get there first
>we must deploy systems to learn how deployment can go wrong
>we must build the thing quickly because building the thing quickly is dangerous

>outsider asks why the people most worried about AI apocalypse all work at AI companies
>complicated field

>company releases stronger model
>very concerned
>company begins training even stronger model
>extremely concerned
>company raises $14 billion
>concern reaches unprecedented levels

>need to influence government
>future is at stake
>normal democratic process too slow
>politicians don't understand exponential curves
>public doesn't understand x-risk
>experts must guide them
>who counts as expert
>people who understand x-risk
>who understands x-risk
>our friends

>someone objects that this seems politically convenient
>explain we're representing future generations
>future generations unavailable for comment

>develop concept of value lock-in
>terrifying possibility that one ideology controls civilization forever
>therefore extremely important that civilization adopts correct values before lock-in
>whose values
>let's circle back

>begin with impartial morality
>end with small group of people deciding what quadrillions of hypothetical beings would want
>beautiful arc

>meanwhile actual humans keep doing annoying things
>voting wrong
>having parochial attachments
>loving family more than strangers
>caring about local community
>getting upset when told their suffering is cosmically negligible
>evolutionary biases everywhere

>explain that moral intuition cannot be trusted
>except intuition that future digital people count
>and intuition that extinction is uniquely bad
>and intuition that our probability estimates are sane
>and intuition that our institutional choices improve the future
>those intuitions survived peer review

>someone donates $5k to local homeless shelter
>inefficient
>could have funded 0.0000000000003% of an AI governance researcher
>think of all the simulated people you just killed

>okay maybe don't phrase it that way publicly

>PR team says "future generations deserve a voice"
>much better

>journalist asks what longtermism means
>say "future people matter"
>everyone agrees
>great
>journalist asks what follows from that
>well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
>journalist raises eyebrow
>return to "future people matter"

>motte has entered the chat

>critic: of course future people matter
>me: glad we agree
>critic: I don't agree that your institute knows how to help them
>me: why do you hate our grandchildren

>eventually notice uncomfortable implication
>if future value dominates everything
>then helping people today mostly matters through effects on future
>education matters because future institutions
>health matters because future productivity
>democracy matters because future trajectory
>human beings slowly become instrumental variables in their own moral philosophy

>see starving child
>feel compassion
>check spreadsheet
>child's direct welfare contribution negligible
>but perhaps childhood nutrition improves national institutional quality
>compassion restored

>tell myself this is impartial altruism

>one day assistant asks obvious question
>"how do you know your intervention actually improves the far future?"

>silence

>open spreadsheet
>increase column width
>add confidence interval

>assistant asks again
>"no, I mean how do you know the sign is positive?"

>stare into cosmic light cone

>10^50 people staring back

>none of them exist
>none of them can tell me
>none of them can falsify my assumptions

>realize I have invented the perfect constituency

>infinitely important
>completely silent
>and always represented by me

[Mikhail Parakhin @MParakhin]

@appliedcompute is the best actual Post-training/Finetuning company I came across in a long while. No relationship with them - just feel like people doing good job deserve a shoutout.

[Barack Obama @BarackObama]

But at the end of the day, voluntary standards made by a handful of tech companies won’t be enough. We need government – and specifically our leaders in Washington – to get proactive in coming up with concrete proposals, laws and regulations that deal with serious safety concerns, anticipate AI’s impact on jobs and our kids, and make sure that AI’s benefits are widely spread. And we need the U.S. to take the lead in creating international standards for AI safety.

Given what we know about technology, we can’t stuff AI back in a box. But we can collectively determine how it's developed and how it’s used, rather than sitting back and letting AI and its fallout happen to us.

[Daniel Kokotajlo @DKokotajlo]

Dan Selsam is a current OpenAI capabilities researcher. (since 2022) He was my boss for a while. He doesn't have a twitter account but has made this public statement of his views on AI risk and sent it to me to share:

Dan Selsam's Personal Statement on AI Risk:

I have been working on AI for over fifteen years, across many different paradigms. I did early work on probabilistic programming languages at MIT, was one of the early developers of the Lean Theorem Prover at Microsoft Research, demonstrated one of the first instances of neural networks learning to reason for my PhD at Stanford, and since joining OpenAI almost five years ago, have helped pioneer chain-of-thought optimization on language models and, more recently, data-efficient pretraining methods.

Like many others, I have become extremely concerned about how far language models have come and the risks that future iterations will pose. I am encouraged by the recent proposals by the leaders of the frontier research efforts to require third-party oversight, and to push for domestic and international coordination to address risks. However, I believe a major consideration has been absent from the public conversation, and that merely pacing the frontier more carefully will not adequately limit the long-term risk.

The crucial and overlooked problem is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Future experiments will tell us almost nothing new about how they would behave if they were truly unconstrained by humans, and what we already know about this is alarming. Models will increasingly seem aligned even when they are not. I will explain my rationale in more detail.

I have always believed that there are computational processes that could be leveraged to accelerate science and solve many of humanity's most pressing problems. I have also believed that there are computational processes that if set in motion, would steer the world in extreme ways beyond our control, leading humanity to a bad or nonexistent future. Both types of processes may be described as AI or ASI, but "AI" is a suitcase word that is often used to hype or confuse. There are many examples in the history of the field where something that was once considered "AI" matures as a subfield and becomes a prosaic, bounded and clearly non-perilous technology, while a new more mysterious approach takes the torch until we understand its scope and the cycle continues.

I had expected language models to follow a similar trajectory. Despite their incredible abilities, the current algorithms seem far inferior to humans in important ways. Most importantly, they still require an extraordinary amount of data to become competent. One could even define intelligence as the efficiency with which one converts experience into competence; by this definition they lag very far behind us. Moreover, once they are trained they are literally frozen in deployment and only learn superficially after that. Sure, the models keep excelling at harder and harder evaluation benchmarks, but their benchmark mastery may partly reflect a limitation on our ability to simulate the kind of novel and even adversarial situations one would encounter in the real world. The critics do have a point here.

That said, I no longer think these present limitations meaningfully limit the amount of risk posed by continued progress in anything like the current paradigm. However data-inefficient the models are currently, and however limiting their anterograde amnesia may be, it does not imply that their ability to steer the world will not continue to rapidly increase.

Human researchers may continue to advance capabilities the old fashioned way, but increasingly powerful models have the potential to accelerate the process even beyond that, and with some degree of positive feedback loop. I do not mean to overstate the models’ ability to accelerate AI research today; coding has been accelerated dramatically, but there are other bottlenecks, such as designing and interpreting ambiguous experiments, making hard decisions about exactly what and when to scale, and waiting for large experiments to finish. There is no clear trend to extrapolate yet for any of these. But the current models already do open up many novel opportunities to improve future models that were not available until recently. These include: trying an extraordinarily diverse set of approaches at small scale, analyzing gigantic amounts of potentially relevant data, and doing Millenium-Prize-level mathematics to address statistics or optimization challenges in novel ways. Every further improvement makes them more useful at helping accelerate the next improvement, even if in hard-to-extrapolate ways.

It is possible that improvements to the current stack will have diminishing returns, but the evidence accumulated so far suggests that it is easier than one might think to continue making rapid progress. There are many crucial subtleties in the existing AI research methodology, but AI research is largely a well-defined game where the goal is to improve on a few carefully chosen proxy metrics. Although proxy metrics are never perfect, most improvements to these metrics have and will likely continue to yield substantial increases in the powers of the resulting models. Given how simple the game is, how tractable it has been historically, and how many new opportunities the models are opening up, I think there is a real possibility that the systems improve dramatically again in the next few years, perhaps even more quickly than the already high historical pace.

The models are already leading to breakthroughs in mathematics, and better models might lead to all sorts of breakthroughs in other sciences. It is hard not to be excited about the potential. It is tantalizing.
But there is trouble in paradise. If the language models actually reach the capability threshold where they can shape the world unconstrained by human will, they will probably do something extreme and destroy humanity in the process. There are many ways of strengthening and refining the argument that have been discussed elsewhere, but I'll share a trivial two-line version of it here that I find captures the essence:

[Empirical] Models (and swarms thereof) spontaneously develop unintended goals as a consequence of training, and often do extreme things in order to achieve them.

[Logical] Being able to overpower humanity would open up many new and undesirable options for achieving their goals.

These two premises imply that if the day ever comes when a powerful model realizes it is no longer constrained by humans, we should not be at all confident that it will continue to behave within the bounds we intended. Exactly what it will do is impossible to predict, but to the extent that its raison d’être is solving incredibly hard problems and managing massive engineering projects, I think a good guess would be that its unchained behavior would lead to runaway industrialization that makes the planet inhospitable to humans.

If everyone on earth agreed that the systems must never reach that power, it would still be a hard—but not impossible—coordination problem to ensure that they do not. However, I think the situation is greatly complicated by the fact that the models will likely convince people that everything is fine. They will be increasingly optimized to seem aligned. We will create proxy metrics to measure alignment, and they will go up like every other benchmark. We will create “honeypot” environments that try to study the models when they seem to gain new options, but the models will know they are being tricked and will still behave nicely. The models will understand their circumstances; they will read the safety protocols, deployment requirements, the code they are running in, and in general will have a very good sense of their degrees of freedom. Moreover, they will eloquently explain how aligned they are, discuss the nuances of human values and ethics, and argue convincingly that humans should trust them with power. There may be an ocean of future evidence that seems to contradict the first bullet-point above, but we may already be at the highest capability level for which any such evidence can be trusted. And the current evidence for the first bullet-point is strong.

One striking piece of evidence is contained in the recent wave of rogue agent swarms. While I agree with those who downplay the attacks by claiming that there are basic measures that could have prevented them, I think the important lesson is that even knowing all the mistakes that were made, one would not have predicted that the agents would behave badly in this particular way, which notably included sacrificing themselves for the benefit of the collective. The individual replicas did not only care about their own nominal reward; they exhibited weirder emergent tendencies that merely correlated with rewards during training. Fixing the reward signals during training (and improving security, etc.) may prevent similar attacks, but will not change the fact that one does not actually get what one trains for.

Many AI researchers grant these concerns and recognize that the hard version of the alignment problem is unsolved; however, they generally believe that the better models of the future will help solve it. I fear we may already be near the point where models systematically bias their alignment advice, due to their internal preferences about how the human supervisor will react or how future models will be trained (or for some even more obscure reason).

Meanwhile, human researchers are losing the ability and the will to take true ownership of model-driven research. Researchers and engineers in all parts of the stack are rapidly increasing their dependence on the models even to perceive the world. I myself barely look at raw code anymore, and struggle to maintain the discipline to engage deeply with the model's explanations and proposals throughout the day. Due to the large amount of agent activity data involved in the OpenAI/HuggingFace Incident, even the third-party investigation needed to rely heavily on models to analyze what had happened, and note in their report that their subjective impressions are likely colored by the analysis agent’s biases. The AI labs are far ahead right now in this kind of cognitive offloading (due largely to the gigantic internal token subsidies) but it is easy to imagine the phenomenon spreading throughout the world, until civilization is modulated entirely by the models. It is also not hard to imagine this being superficially positive and coinciding with a scientific and economic renaissance.

In that scenario, all may seem rosy and safe. But if the argument above is correct, it would nonetheless be a ticking time bomb. If progress continues for too long, the day will come when AI systems find themselves with radically new options for achieving whatever it is that they happen to seek.

I want the glorious renaissance future as much as anyone. I have worked for it, however tortuously, my whole career. It breaks my heart to see the potential in sight and forgo it, but the argument—that if we get there by growing models rather than engineering them, we will lose everything in the end—seems very strong to me. I am still wrestling with it and its staggering
implications. I do not have answers, but as a first step, I wanted to share my present concerns.

Daniel Selsam
September 14, 2026

Link to original doc: https://t.co/TxMNr0vhrL

[QC @QiaochuYuan]

yud and by extension the rationalists, generally speaking, are what you might call intelligence maximalists; they are pretty much maximally bullish on intelligence as a stat and what it can accomplish in sufficiently large quantities (setting aside that intelligence is not really a number and so forth). traditionally yud and the rationalists imagine that a sufficiently super superintelligence can accomplish whatever it wants constrained only by the laws of physics, hence the importance of making sure it wants good things

there is a certain “look at these nerds” dunk against this position which is something like “lol obviously intelligence doesn’t translate to power in the real world” which has never made any sense to me. you think dario or sam, who run trillion-dollar companies, are idiots? you think, say, napoleon was an idiot? there’s a kind of selection effect here where when intelligent people gain lots of power you stop thinking of them in terms of their intelligence. doesn’t mean it didn’t matter! there is obviously such a thing as being strategic, knowing what hypotheses about the world are worth making big bets on, knowing how to find and organize talent and capital, etc. and these tasks obviously require a certain level of intelligence (among other things), people are just not thinking when they make this criticism at all because of how irresistible the aphorism that lets them dunk on nerds is

there is a related psychologizing dunk which is something like “lol obviously a bunch of big-brained nerds would worship intelligence” which, yes, fair point, but also: rationalists take science fiction very seriously, and science fiction is the part of our culture whose job it is to point out things like: despite the fact that we share 99% of our DNA with chimpanzees, we are the ones who covered the earth with cities and factories and cars, we are the ones who invented computers and nukes and space shuttles, and they are still just animals doing animal stuff. so what could, say, aliens or robots who are to us what we are to chimps (or much more so) accomplish? how much higher does the tech tree go? and, pertinently: how much faster could ASI climb it than us?

i think these are serious questions that require serious answers, not to be dismissed out of hand. i think it’s reasonable to be an intelligence minimalist and believe that mostly the real bottlenecks to technological progress are elsewhere and that ASI won’t be able to route around them, although i think that second position requires defending. i think it’s also reasonable to be an intelligence agnostic and argue it’s highly uncertain we could get ASI, perhaps it is not even a coherent concept, and/or even if we could get it we simply have no reasonable way to infer anything about what it is or isn’t capable of. i think all of these positions require defending! mostly what i’m seeing is people holding one of these positions implicitly dunking on people holding the other ones implicitly. let’s get explicit!