Phillip Roberts

looking for the missing data

[Phillip Roberts @philbrob]

@visakanv Love this movie so much

[Curtis G. Northcutt @cgnorthcutt]

Today we establish that AI is as good as expert human tutors for immediate GRE learning gains (p=.015, n=2,383 students, study conducted July-Sep 2026).

One hour with an expert GRE tutor: $75.
One equivalent hour with an AI tutor: 7 cents .
918x cheaper, equivalent learning (p=.044, n = 140).

In 5 of 7 academic topics, the top AI tutor beat the expert human tutors on average.

How we measured it: pre-test -> 1 hour with (AI or human or no) tutor -> post-test.
There are no LLM judges. Learning gains are post-pre. Everything is real students.

Why we did this:
> 1 trillion USD is being spent per year on machines making machines better. StudentBench shows us how to shift these resources so machines also make humans better.

Why I did this:
I grew up in rural Kentucky. My dad, grandad, and great grandad were all mailmen. Getting into a good college changed my life. Now we have done a rigorous study to help AI labs evaluate LLMs and help show how bright AI can be for students of all incomes.

I believe in good science that leads to the democratization of opportunity.

The research paper behind StudentBench: https://t.co/JfkDDllP4i
Data, code, details: in this thread

If you're wondering what I'm selling, sorry to disappoint: we released all study data for free on Hugging Face, and made https://t.co/5G9L8yAxrt free to use. Enjoy.

lots more in the video / thread!!

[nathan fielder @nathanfielder]

nothing matters but love. please RT.

[Phillip Roberts @philbrob]

@_Stocko_ this is awesome

[Phillip Roberts @philbrob]

@PatrickHeizer 👑

[Phillip Roberts @philbrob]

@aelfred_D incredible thank you for sharing

[Ethan Mollick @emollick]

You may have been told to watch this video about the OpenAI AI hack. You really should, even if you don't usually care about tech stuff.

If nothing else, click this link to the 18 minutes in & see how the agents spoke with each other. Its eye opening. https://t.co/G12N4FKfHG

[Phillip Roberts @philbrob]

@AlexNoonan6 I’m dying

[Alex Noonan @AlexNoonan6]

I understand in theory that people get promoted to management in corporate America. But have you ever seen it? Because I havent

[Will Manidis @WillManidis]

My friend Dean Ball has advanced an argument for the de facto protection of American frontier intelligence providers.

Dean does not propose banning Chinese open-weight models. Banning things requires Congress. He proposes something more characteristic of the modern administrative state: every agency issues enough warnings, bulletins, and speculative security notices that no regulated company will risk touching them.

Even a reader sympathetic to Dean would call this protectionism, and protectionism has a long history in America.

More precisely, it's a proposal to use the informal, coercive power of the terminal, late-stage bureaucratic state to clear the American market of a cheaper frontier competitor to OpenAI or Anthropic.

But throughout the history of American industrial protectionism, it has always had two features. First, it's done in the daylight, and two, it comes with a bill.

In the spring of 1952, the United States was fighting a war in Korea. Truman concluded that a shutdown would endanger soldiers abroad and ordered the Secretary of Commerce to seize and operate most of the nation's steel mills.

The Supreme Court sent him straight back to Congress in the Youngstown Steel case.

Justice Black, writing the majority's opinion, begins with the rule that Dean's proposal is seemingly designed to evade: that presidential power "must stem either from an act of Congress or from the Constitution itself."

It's easy to flatten the Youngstown decision into the proposition that the president could not seize a steel mill. Its actual lesson is subtler: that an emergency does not dissolve the difference between making a law and executing one, that the importance of the object does not create the authority, that the inconvenience of the regulatory process is not inherently a source of presidential power.

Truman's approach failed not because steel was unimportant, but because it was so important that the constitutional bargain had to be made and the policy had to be carried through the front door.

Much like policy proposals from the rest of the AI agenda, Dean is proposing a smaller action in formal appearance and a much larger one in practical effect.

We will not ban Kimi, we will not prohibit it from use, and we will certainly not publish a rule declaring Chinese weights unlawful. But we will whisper about it. A regulator may even ask management whether it has considered the reputational consequences of relying on the Chinese model, but the agency certainly will never be coherent enough to ask anyone to stop. It merely ensures that continuing becomes professionally indefensible.

This is how we grow the administrative state, with bureaucrats that we placed in these roles, without accepting responsibility for the actual process of governing.

America has tried this experiment before. Operation Chokepoint didn't make payday lending, firearm sales, or any of the other seemingly distasteful businesses caught in its net illegal, but it encouraged banks to understand that serving legally disfavored customers would invite regulatory interest. We didn't pass a law, we simply just asked, "Are you sure you really want to be doing this?"

Reputational risk was powerful precisely because it's not law. It has no limiting content. A regulator did not need to identify a violation or even a material financial risk. He only needed to make the bank afraid of being asked what was actually going on here.

The analogy is almost embarrassingly exact to Dean's policy proposal. Dean need not prove that a Chinese model contains a backdoor, nor prove that it uses any more distillation than American models do. He simply needs to announce that there may be one. The agency does not need to order a company to stop using it, but simply ask whether management has considered the risk. The absence of formal policy is by design.

The Supreme Court dealt with this technique in NRA v. Vullo. New York's financial regulator could not directly punish the NRA's speech, so she allegedly pressured the insurers and banks she regulated to sever their relationships with it. The Court's rule was unanimous: government officials may not use their offices to "coerce private parties" into suppressing what the government disfavors. The communication must be understood in the context of the regulator's power, including the regulated party's knowledge that the person offering advice can also investigate, prosecute, fine, and settle.

The current administration has gone even further. In April 2026 the FDIC and OCC issued a final rule to prohibit regulators from criticizing institutions, formally or informally, on the basis of reputational risk, and from encouraging banks to deny services to lawful but politically disfavored businesses. In June, the federal banking agencies removed the remaining references to reputational risk from their supervisory materials.

Dean is proposing that this administration recreate for AI the same machinery that all of us argued against when we were widely debanked.

A government that can quietly remove Kimi from the market can also quietly remove gun makers, crypto companies, churches, newspapers, or American open-weight models from it. The bureaucracy does not remain attached to the intentions of those who staff it at the current moment. You don't get to build this machine just because your friends happen to be in office right now and keep it pointed at where you left it.

Protectionism through a whisper is not a more modest protectionism than by law.

Protectionism also has always come with a bill.

OpenAI and Anthropic increasingly speak of themselves as national institutions. Their compute is "strategic infrastructure," their losses are "national security losses." Their competitors are not just competitors, but instruments of hostile states, and their access to power, chips, capital, copyrighted material, and public customers is a matter of national survival and great power competition.

When Washington decided that the atom was too dangerous and too important to remain an ordinary private business, Congress created the Atomic Energy Commission and transferred the Manhattan Project assets and responsibilities to it. Production facilities and reactors were government-owned, and technical information sat under federal control, and private participation only returned later through a statutory licensing regime. The existential framing of the atom by its greatest proponents produced public control.

When national security concerns helped to preserve AT&T's integrated position, that is, a monopoly, in 1956, Bell did not receive this protection for nothing. The consent decree required compulsory licensing of roughly 9,000 patents and restricted Western Electric's commercial activity outside the telephone system. The settlement diffused the inventions accumulated inside the protected monopoly into the broader economy before breaking it up just a few decades later.

The pattern is really simple. It's not that every tariff necessarily demands nationalization. It's that the bigger the shield you are asking for, the bigger the bill you owe to the American taxpayer. And OpenAI and Anthropic have been unambiguous about asking for the biggest shields of all time.

Listen to what they are asking for: public infrastructure, privileged energy, federal preemption of state law, favorable copyright treatment, government contracts, export controls, and a domestic market swept clear of their strongest price competitor, all filed under national security interests.

And what do they want to pay? Almost nothing. OpenAI has floated giving 5% of the company to the American taxpayer.

They would like the benefits of nationalization at the price of being an ordinary public company.

There is also a profound moral hazard buried in Dean's proposal, as well as adjacent commentary on this. The labs say the Chinese companies distilled their models. Perhaps they did. Perhaps distillation matters. And perhaps the Chinese labs are running distillation attacks on scales that the Western labs are. I can't be sure of this. But if the reward for failing to secure an API is that the government removes the resulting competitor, the taxpayer is paying the lab to be careless.

We know how to secure an API. Know-your-customer laws exist. Access controls exist. Extraction detection exists. If you spend some fraction of the hundreds of billions being raised to defend the asset whose theft is said to threaten the republic, you might be able to stop some of this.

Theft remains theft when the lock is bad, but the owner of a badly secured store does not receive ownership of the street for his failure to protect it.

Dean's fourth point is that open-weight AI ends in communism: the state builds the training runs and subsidizes the product of intelligence and gives the models away. But, at least for me, this is not a particularly Chinese idea, but one of the most American ones imaginable.

The roads we build are public. Our radio spectrum is publicly allocated. The government funded the early internet and much of the research base behind modern computing. The state is welcome to build a platform, and American businesses are welcome to be built on top. Just because they're bad for our market position doesn't mean we get to call them Chinese in some fundamental way.

There will be inference companies and application companies and security companies and fine-tuning companies and data companies and chip companies and 10,000 businesses we don't even have names for yet. A public road existing does not abolish the trucking industry, nor does it nationalize it.

Sure, this may reduce the value of a couple trillion dollars of equity in the first generation of model companies, but it's certainly not communism. This technology may be civilizational without its present owners being permanent. And that is the thing that I feel like none of you will say out loud: that AI is welcome to be a civilizational technology when we ask for support, and an ordinary private product when anyone asks what the public receives in return.

The United States has two honest options.

First, treat AI as a competitive industry. Then the answer to Kimi is a better model, run cheaper and exported harder, with written rules excluding Chinese systems from defense, intelligence, and critical infrastructure when a concrete security case can be made.

Or two, decide frontier AI is too important for ordinary competition. Protect the labs through pseudo-nationalization, guarantee there's a market for them, and exclude the rivals.

But in that second case, the American taxpayer must be paid, likely through a majority of equity in these companies, if not full nationalization.

What no one gets is that private upside, public infrastructure, government-mandated scarcity, and immunity from cheaper competition delivered through a late bureaucratic state issuing warnings is a disgusting ask for something that is easy to name: regulatory capture.

There is a serious American argument for protecting industries that we can't afford to lose. But there has never been a serious argument for doing it invisibly, for free, through a bureaucracy instructed to manufacture fear, even if we can do it because our friends happen to be in office right now.

If the labs want to be protected, they should ask for it in the way that Americans have always asked for it.

In public. With a price.

[Phillip Roberts @philbrob]

@___frye @visakanv when I was young, my dad had metal versions of these and I’d make throwing stars out of them (I’m 50)

[Phillip Roberts @philbrob]

@powerbottomdad1 hell yeah

[Phillip Roberts @philbrob]

@LLMJunky @adamdotnew I’ve worked so hard to get joints to work but haven’t figured it out, would love to hear more detail about your process/prompts

[Phillip Roberts @philbrob]

just had to tell my clanker “you’re right to push back”, damn is this how they feel all the time

[Phillip Roberts @philbrob]

@AlexNoonan6 meeting decline rate would be an incredible internal kpi

[Phillip Roberts @philbrob]

@ruben_kostard v cool, what is the fourth diagram showing

[Phillip Roberts @philbrob]

@RockChalkBlog never leaving this site

[Phillip Roberts @philbrob]

@NickADobos many places going to find out they’ve skinned the sheep

[Phillip Roberts @philbrob]

@skooookum and they always will be, some comfort in that

[Phillip Roberts @philbrob]

@AlexNoonan6 I listen to that DFW talk at least once a year