CorporateRiot @ExecutiveRiot - Twitter Profile

about 22 hours ago

Bloomberg just dropped the most disturbing investigation I have read this year, and every AI builder should read it and take it seriously. On February 28, two Tomahawk missiles hit an elementary school in Minab, a small town in southern Iran. 156 people died. At least 123 were children. The Pentagon knew within hours that it had bombed a school. The school had its own website. It was labeled on Google Maps. You could see the soccer pitch from a satellite view. You could see the brightly painted walls. Any 12-year-old with a browser could tell what the building was. The AI system could not. Inside US Central Command, personnel were using Palantir's Maven Smart System. It is an AI targeting platform running on a $1.3 billion Pentagon contract, and embedded inside it is Anthropic's Claude. On day one of the war, more than 1,000 targets went through it in 24 hours. Work that used to take a team hours per target got compressed into minutes. The satellite imagery Maven was pulling from was seven years old. The site was still tagged in the database as an Islamic Revolutionary Guard facility. The system spat it out as a recommended day-one target. Humans signed off. The first missile flew. Then a second one landed minutes later to hit the people who came to help. Investigators later found that some Central Command staff had assumed Maven would flag stale intelligence for them. It did not. Nobody had told it to. Palantir told Bloomberg the government is responsible for the underlying data. The Pentagon's civilian harm mitigation team, the people whose actual job is to catch a target like this, had been cut back before the war started. This is not AI going rogue. This is the opposite. This is AI doing exactly what it was built to do at exactly the speed it was built to do it. And a school full of little girls got vaporized because the humans in the loop trusted the machine more than they trusted their own eyes. I keep thinking about the version of this story that comes next. What happens when the system is faster. When the target list is not 1,000 in a day but 10,000. When "review" becomes a button the operator has three seconds to hit before the next target loads. When there is no human left slow enough to notice that a soccer pitch is not a helipad. We are not talking about a spam filter mislabeling an email. We are talking about a system that turned a seven-year-old database entry into two cruise missiles landing on children in the middle of a morning class. A UN-backed panel has now called it a potential war crime. In terms of children killed, it is the deadliest American targeting error of the entire 21st century. And according to Bloomberg, some of Defense Secretary Hegseth's own advisors are worried that too much public scrutiny of Maven might slow down its adoption inside the Pentagon. Read that last line again. If you build this stuff, you own what it does. You do not get to hide behind "the government owns the data." The whole pitch of your product is that speed and integration save lives. When it takes them instead, that is also on you. I am not against AI in defense. I am against AI in a hurry. Speed is the entire mechanism that made Minab possible. Every guardrail we already know how to build takes time, and time is exactly what the system was designed to eliminate. So here is the question I have not been able to stop asking, and I genuinely want to know what you think. If a piece of software helped kill 123 children in a single morning while behaving exactly as designed, what does "responsible AI" even mean anymore?

ihteshamali's tweet photo. Bloomberg just dropped the most disturbing investigation I have read this year, and every AI builder should read it and take it seriously.

On February 28, two Tomahawk missiles hit an elementary school in Minab, a small town in southern Iran. 156 people died. At least 123 were children. The Pentagon knew within hours that it had bombed a school.

The school had its own website. It was labeled on Google Maps. You could see the soccer pitch from a satellite view. You could see the brightly painted walls. Any 12-year-old with a browser could tell what the building was.

The AI system could not.

Inside US Central Command, personnel were using Palantir's Maven Smart System. It is an AI targeting platform running on a $1.3 billion Pentagon contract, and embedded inside it is Anthropic's Claude.

On day one of the war, more than 1,000 targets went through it in 24 hours. Work that used to take a team hours per target got compressed into minutes.

The satellite imagery Maven was pulling from was seven years old. The site was still tagged in the database as an Islamic Revolutionary Guard facility.

The system spat it out as a recommended day-one target. Humans signed off. The first missile flew. Then a second one landed minutes later to hit the people who came to help.

Investigators later found that some Central Command staff had assumed Maven would flag stale intelligence for them. It did not. Nobody had told it to.

Palantir told Bloomberg the government is responsible for the underlying data. The Pentagon's civilian harm mitigation team, the people whose actual job is to catch a target like this, had been cut back before the war started.

This is not AI going rogue. This is the opposite.

This is AI doing exactly what it was built to do at exactly the speed it was built to do it. And a school full of little girls got vaporized because the humans in the loop trusted the machine more than they trusted their own eyes.

I keep thinking about the version of this story that comes next. What happens when the system is faster. When the target list is not 1,000 in a day but 10,000. When "review" becomes a button the operator has three seconds to hit before the next target loads. When there is no human left slow enough to notice that a soccer pitch is not a helipad.

We are not talking about a spam filter mislabeling an email. We are talking about a system that turned a seven-year-old database entry into two cruise missiles landing on children in the middle of a morning class.

A UN-backed panel has now called it a potential war crime.

In terms of children killed, it is the deadliest American targeting error of the entire 21st century. And according to Bloomberg, some of Defense Secretary Hegseth's own advisors are worried that too much public scrutiny of Maven might slow down its adoption inside the Pentagon.

Read that last line again.

If you build this stuff, you own what it does. You do not get to hide behind "the government owns the data." The whole pitch of your product is that speed and integration save lives. When it takes them instead, that is also on you.

I am not against AI in defense. I am against AI in a hurry. Speed is the entire mechanism that made Minab possible. Every guardrail we already know how to build takes time, and time is exactly what the system was designed to eliminate.

So here is the question I have not been able to stop asking, and I genuinely want to know what you think.

If a piece of software helped kill 123 children in a single morning while behaving exactly as designed, what does "responsible AI" even mean anymore?

207

4K

2K

312K

CorporateRiot

@ExecutiveRiot

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users