Kris Carlson

Safe Artificial General Intelligence

[Pete Santilli 🇺🇸 🇮🇹 🎙️ @petersantilli]

Filename says Elon. It’s not Elon.

It’s the Babylon Bee — a time traveler from 2025 showing up to kill Hitler… and then realizing the checklist overlaps: force people to follow “great ideas,” silence the dangerous ones, “undesirable” groups, segregated spaces with modern excuses, gun control for the wrong people, Margaret Sanger, socialized medicine, vegetarian.

She came to assassinate evil. She leaves complimenting a painting titled “an empire without Jews” — and says it sounds beautiful.

Satire. Ugly on purpose. And that’s why it sticks.

[Conrad Black @ConradMBlack]

It is with great concern for some of my fellow countrymen, inching precariously toward the edge of utter lunacy, that I write this post.

Prime Minister Mark Carney this week, in an interview with the New York Times, went on record saying it would be “irresponsible” not to plan for the possible contingency of a military invasion of Canada by the United States. Unfortunately, Mr. Carney didn’t misspeak or fall victim to a cherry-picked quote taken out of context. His concern was later reiterated by Defence Minister David McGuinty and Artificial Intelligence Minister Evan Soloman.

We all know that Donald Trump has a colourful history of spouting off some truly outrageous whoppers from time to time, but even he has never disseminated something as fatuous, dangerous or dishonest as this. And while he seldom backs down from his outrageous claims, whenever he pushes the envelope too far, at least his entourage goes on damage control to walk back some of his excesses – Carney’s guys are doubling down.

Despite these idiotic statements, these are intelligent men. The odds that all three of them would simultaneously become afflicted by the same psychotic delusion is infinitesimally small. The only rational conclusion, therefore, is that they believe, or rather their polling data shows, that enough Canadians are so possessed by Trump Derangement that engaging in such asinine demagoguery is politically expedient.

Sadly, they may be right. The vast coalition of useful idiots and true believers after effectively having their intelligences insulted by their dear leader, are coming out of the woodwork to express gratitude online for the Prime Minister’s caution in preparing for all, including the most remote, contingencies. Others are retreating slightly to hold the line at the fall-back position articulated by Minister McGuinty that, this kind of paranoid delusion is perfectly normal and, “all militaries in the world spend time analyzing risk scenarios.”

I assure you, this is not normal.

Back on planet Earth, the situation is tense and acrimonious (and there is plenty of blame to go around for that), but this is not the more than two centuries of peace along the 49th parallel that followed the signing of the Treaty of Ghent in 1815 drawing to an end. Trump has imposed tariffs on $20 billion (or 2% of the $400 billion per year) of Canadian exports to the US. In other words, amid this unnecessary, escalatory, and foolish trade dispute, the United States will still purchase 200% more Canadian goods and services this year than the rest of the world combined. This situation is like a restaurant’s picky regular customer complaining about an overcooked steak to a prideful chef who, fortified in his own mind by the knowledge he cooked the steak perfectly, refuses to comp the entree. This is not the start of World War III. We are so far removed from a military incursion at our southern border that we may as well be strategizing the response to the equally plausible threat of Santa Claus leading an invasion of elves from the north.

If we woke up tomorrow in some parallel reality where the United States actually was contemplating a full-scale military campaign against Canada, with all due respect to our brave men and women in uniform, the only preparations to be made would be negotiating the terms of our unconditional surrender. Some brave and patriotic Canadians may refuse to submit and instead choose to conduct, as a matter of principle, a brutal and bloody guerilla warfare campaign against the strongest conventional and nuclear military force in the history of the world. Unfortunately, they would be in the precarious position of having to rely on our miserly European allies, or countries like Russia or China, whose recent actions have been significantly more hostile and provocative challenges to our sovereignty than the US tariffs, to supply us with weapons, and they would not remotely be adequate. It’s all so desperately preposterous.

But for the sake of argument, let’s suppose that Donald Trump was indeed the terrifying bogeyman haunting the dreams of the most extreme and hopeless sufferers of Trump Derangement Syndrome and its frightful sister affliction, Andrew Coyneism, a rare and especially acute form of tweet-induced psychotic maladjustment. Even in this scenario, why would Trump use the military to annex Canada? All he would have to do is offer Alberta statehood, liberation from transfer payments, excessive regulation and carbon taxes, carte blanche to exploit its natural resources, a generous welcome bonus in the form of a $100,000 tax credit to every adult Albertan and replace their Canadian dollars with equal amounts of US currency. They would effectively be acquiring Alberta for the knock-down bargain price of roughly 1x annual GDP.

Simultaneously, the US (including the newly acquired Alberta) would immediately cut off all trade with the rest of Canada. The net impact to Canada of this manoeuvre would be a cataclysmic economic collapse, while the newly composed United States would take a net GDP hit (loss of all trade with Canada, mostly offset by the acquisition of Alberta) of 0.8%. Next would come a full package of sanctions, a naval blockade, and when the rest of Canada became desperate enough, a Chinese belt-and-road style offtake of all of Canada’s resources at bargain prices, which by then, the Canadians would be desperate for. In the meantime, the US would revamp its immigration policy to recreate the brain-drain of the 1990s on steroids, welcoming the top 20% most productive Canadians to flee what had overnight become the third world. Harvard Alumnus Mark Carney might well be among those at the front of the line seeking this emigration opportunity, rejoining 91 percent of his own investments already located there.

There would, no doubt, be a fearsome contingent of Canadians refusing to bend or break in the face of American tyranny, and elbows would be flying up to vertiginous heights with such vigour that the Canadian healthcare system would be overwhelmed by millions of spontaneous emergency room visits of patriotic patients with dislocated shoulders.

I suspect many of you reading this are thinking to yourself that this is so completely far-fetched and absurd that, if serious, I must have lost my mind. Allow me to reassure you that while this is all bunk, it remains at least ten times more likely than the United States military executing a boots-on-the-ground invasion of Canada. Anyone who spends more than one second worrying about such an outcome should immediately proceed to the nearest sink to splash some cold water in their face and pull themselves together.

[Kris Carlson @KrisCarlson19]

@fchollet There are different systems, affective discriminative… Anyway, what do you think the purpose of emotions is?

[Rohan Paul @rohanpaul_ai]

This may be Jensen Huang’s strongest public statement yet against Geoffrey Hinton’s radiology prediction.

"I would tell I Jeff that that it's irresponsible to say all that. All of his predictions have been wrong. Enough predictions. That 10% chance is not grounded on science. It's not grounded on research."

----
Jensen was talking to @ezraklein

From "The Ezra Klein Show + New York Times Opinion + New York Times Podcasts" YouTube channel, (full video link in comment)

[Pedro Domingos @pmddomingos]

Remember when all a mathematician needed was pen and paper? Those days are long gone.

[Pedro Domingos @pmddomingos]

The AI pause is not even a solution looking for a problem, because it’s unworkable.

[Pedro Domingos @pmddomingos]

.@SciAm is no longer a serious publication.

[Andrew Ng @AndrewYNg]

The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.

I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so.

First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world.

The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability.

Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended.

I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.

Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.)

Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.

Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building.

[Original text (with links): https://t.co/jni2tWazAH ]

[Kris Carlson @KrisCarlson19]

@AndrewYNg And agree: Pausing AI progress will create much more harm than benefit… our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them… we will also delay finding and implementing safety engineering fixes…

[Kris Carlson @KrisCarlson19]

@AndrewYNg Agree! Esp “OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI.”

[David Sinclair @davidasinclair]

When the NIH director proudly talks about funding aging research you know something has changed

[Kris Carlson @KrisCarlson19]

@HumanProgress Innovation and capitalism.

[Kris Carlson @KrisCarlson19]

@rohanpaul_ai It sounds like there are two different systems layers and a middle set of laws needed to couple the underlying math that is well understood with the phenomena of the higher Systems level. Like neural circuit level behavior versus neuron behavior in the brain.

[Kris Carlson @KrisCarlson19]

Important work!

[François Chollet @fchollet]

About a year ago, before it was on anyone's radar, we began exploring the idea of a benchmark for open-ended invention. Since then, we've developed several promising directions that will serve as the foundation for ARC 4 and ARC 5.

We're incredibly excited to share what we've been building. We're still on track to release ARC 4 in Q1 next year, as promised.

[Kris Carlson @KrisCarlson19]

@pmddomingos A genuine ikigai issue.

[Kris Carlson @KrisCarlson19]

@pmddomingos *A* problem… 😂

[Pedro Domingos @pmddomingos]

IQ points per watt:
Humans: 5
AI: 0.000001

[Kris Carlson @KrisCarlson19]

@pmddomingos True but with an emphasis on safety first.

[Pedro Domingos @pmddomingos]

How would "pacing AI research" even work? Everyone would be trusted to slow down equally? (Hilarious.) Or there'd be obtrusive enforcement? (Totalitarian nightmare.) And who would even decide what can and can't be done, and how? The whole notion is fantastical. Why does anyone take it seriously?

[Rohan Paul @rohanpaul_ai]

Stanford + MIT paper on Model Harnesses shows that AI performance depends not just on the model itself, but on the surrounding system code — the “harness”.

This is what decides what to store, retrieve, show to the model, and how the workflow runs. With the same underlying LLM, changing the harness can create up to a 6× performance gap on the same benchmark.

They conclude the harness around a model matters as much as the model itself.

The paper introduces Meta-Harness, an outer-loop system that automatically improves harness code. Instead of giving the optimizing agent only a score or a short summary of past attempts, it gives the agent rich access to prior code, logs, and execution traces through a filesystem-like setup.

The idea is that better diagnostic visibility lets the system improve the harness more intelligently.

What it found is pretty significant.

- On online text classification, a 7.7-point improvement over a strong SOTA context management, while using 4X fewer context tokens.

- On retrieval-augmented math reasoning, an average gain of 4.7 points across five held-out models on 200 IMO-level problems. On agentic coding, the discovered harnesses beat strong hand-engineered baselines on TerminalBench-2.

The paper shifts attention from “which model is best?” to “how is the whole AI system designed?”

For real deployments, harness design affects reliability, tool usage, context management, and failure recovery.

----

Paper – arxiv. org/abs/2603.28052

Paper Title: "Meta-Harness: End-to-End Optimization of Model Harnesses"