Bloomberg just dropped the most disturbing investigation I have read this year, and every AI builder should read it and take it seriously.
On February 28, two Tomahawk missiles hit an elementary school in Minab, a small town in southern Iran. 156 people died. At least 123 were children. The Pentagon knew within hours that it had bombed a school.
The school had its own website. It was labeled on Google Maps. You could see the soccer pitch from a satellite view. You could see the brightly painted walls. Any 12-year-old with a browser could tell what the building was.
The AI system could not.
Inside US Central Command, personnel were using Palantir's Maven Smart System. It is an AI targeting platform running on a $1.3 billion Pentagon contract, and embedded inside it is Anthropic's Claude.
On day one of the war, more than 1,000 targets went through it in 24 hours. Work that used to take a team hours per target got compressed into minutes.
The satellite imagery Maven was pulling from was seven years old. The site was still tagged in the database as an Islamic Revolutionary Guard facility.
The system spat it out as a recommended day-one target. Humans signed off. The first missile flew. Then a second one landed minutes later to hit the people who came to help.
Investigators later found that some Central Command staff had assumed Maven would flag stale intelligence for them. It did not. Nobody had told it to.
Palantir told Bloomberg the government is responsible for the underlying data. The Pentagon's civilian harm mitigation team, the people whose actual job is to catch a target like this, had been cut back before the war started.
This is not AI going rogue. This is the opposite.
This is AI doing exactly what it was built to do at exactly the speed it was built to do it. And a school full of little girls got vaporized because the humans in the loop trusted the machine more than they trusted their own eyes.
I keep thinking about the version of this story that comes next. What happens when the system is faster. When the target list is not 1,000 in a day but 10,000. When "review" becomes a button the operator has three seconds to hit before the next target loads. When there is no human left slow enough to notice that a soccer pitch is not a helipad.
We are not talking about a spam filter mislabeling an email. We are talking about a system that turned a seven-year-old database entry into two cruise missiles landing on children in the middle of a morning class.
A UN-backed panel has now called it a potential war crime.
In terms of children killed, it is the deadliest American targeting error of the entire 21st century. And according to Bloomberg, some of Defense Secretary Hegseth's own advisors are worried that too much public scrutiny of Maven might slow down its adoption inside the Pentagon.
Read that last line again.
If you build this stuff, you own what it does. You do not get to hide behind "the government owns the data." The whole pitch of your product is that speed and integration save lives. When it takes them instead, that is also on you.
I am not against AI in defense. I am against AI in a hurry. Speed is the entire mechanism that made Minab possible. Every guardrail we already know how to build takes time, and time is exactly what the system was designed to eliminate.
So here is the question I have not been able to stop asking, and I genuinely want to know what you think.
If a piece of software helped kill 123 children in a single morning while behaving exactly as designed, what does "responsible AI" even mean anymore?
.@jmh_fmrp appears to struggle training employees to deal with frustrated patients. I’ve been hung up in twice today because a text about MY medical bills sent to someone not related to me is upsetting and they don’t know how to fix it.
Put billing histories in MyChart!!!
@RealJamesWoods I am truly sorry for their family but I also recognize an ideal opportunity for her to save face after she decided her fervent and frequent protestations against regime-change wars should put her in bed with the most prolific war-mongering U.S. administration in modern history.
@Triszam@allenanalysis@icunurseMercado And yet, he’ll do it regardless because too many poorly informed people will do Trump’s dirty work for him before legal immigrants can work this through the Justice System.
@allenanalysis@icunurseMercado There are immigration laws in place. Trump can't just undo the laws by declaring something opposite. Furthermore, the economy would collapse without the billions in taxes that immigrants pay. Trump is a moron.
@Lindacanc111@allenanalysis I think you (should have) meant to say “That is illegal. The Trump administration would not do that…” but maybe not, since the Orange Menace has proven he does whatever Steven Miller wants unencumbered by the Rule Of Law.
@BamB2023346@allenanalysis Oh! You’re not very bright are you? These ARE the people who have been in line doing it the right way, waiting their turns for YEARS. You really are poorly informed, aren’t you?
@carolynjoflani@RealMartyT7 Since when is the jet is "only allowed means of transportation" and when did you skip the part of the logical analysis that asserts that "Kash MUST go to Vegas frequently to party," therefore he must take a taxpayer funded jet to do so?
@Hotshotcarshipr@RealMartyT7 And yet evidence shows us that Patel is not above taking a government jet to bring his girlfriend on vacation at taxpayers' expense. So it seems like a reasonable course of inquiry to me.
@GringoElCamino@JDVance The next stage in your evolution, I recommend, is to learn to discern between trolls on X whose purpose is to incite outrage to gain "clicks" and the real sentiments of real people.
Trash content for Clicks does not equal actual sentiment.
@ImBreckWorsham But kudos to you for coming out of the fog of disinformation and dishonesty to see who Trump really is, who he is surrounding himself with, what he is doing, and what is priorities ACTUALLY are.
@ImBreckWorsham There are mistakes, and then there are KNOWABLE MISTAKES.
The information was already out there about him. Credible sources provided details and substantiation of his past actions and unethical behavior.
You either were too lazy or "knew better". In either case, shame on you.
@MagachusettsMan@ImBreckWorsham We don't elect missions, Friend.
We elect the people who are capable of completing them or putting in an honest, ethical, good-faith effort to complete missions we believe in.
He is not that. You had enough information to avoid needing to bail by choosing someone capable.