Daniel Maenle

Senior Consultant at SQL Solutions Group.
The views expressed here are my own and not of my employer. He/him

[Daniel Maenle @DanTheSQLMan]

I'm late to the party. But I built my first power bi dashboard to be able to elegantly present some data for better software development priorities in database pipelines.

[Daniel Maenle @DanTheSQLMan]

Well this is terrifying

[Aakash Gupta @aakashgupta]

The scariest number here: 3.61% of CPUs in one large-scale study were found to cause silent data corruptions. Not “a few bad chips.” Nearly 4 out of every 100 processors doing math wrong, silently, with no error log.

Google coined the term “mercurial cores” in 2021 after their production teams kept blaming software for data corruption. They’d debug for weeks, find nothing wrong with the code, swap the machine, problem gone. The actual cause: manufacturing defects at sub-7nm that pass every factory test, then degrade unpredictably months or years after deployment.

Facebook confirmed the same thing independently. Hundreds of affected CPUs across hundreds of thousands of machines. The defect doesn’t crash your system. It just gives you 5 instead of 6 when you multiply 2x3, under specific microarchitectural conditions, with zero indication anything went wrong.

Now think about what this means for AI training. A single corrupted GPU or CPU in a distributed training cluster doesn’t just produce one bad output. It feeds corrupted gradients into a synchronization step that gets averaged across every accelerator in the cluster. One bad chip can silently poison an entire training run. NVIDIA published a whitepaper on exactly this problem. Loss spikes during LLM training that nobody could explain traced back to silent hardware corruption.

The part that keeps infrastructure engineers up at night: traditional defenses don’t work. ECC memory can’t catch this because the corruption happens during computation, not storage. Checksums like CRC heavily use vector operations, which are themselves one of the most vulnerable instruction types. The tools designed to detect corruption are running on the same flawed silicon.

Google’s current detection method? Roughly half human-driven, half automated. And of the machines humans flag as suspicious, only about 50% are actually confirmed mercurial on deeper investigation. We’re debugging trillion-parameter models on hardware where we can’t reliably tell which chips are lying to us.

Moore’s Law gave us more transistors. It also gave us transistors we can’t fully verify.

[Machine Pun Kelly 🇺🇦 @KellyScaletta]

https://t.co/YTMVMPqHtQ

[Daniel Maenle @DanTheSQLMan]

@NothingCMFFans Awesome

[Shooter McGavin @ShooterMcGavin]

This man volunteered to tell a joke and absolutely killed it

[Daniel Maenle @DanTheSQLMan]

@AnirudhPai5 Nothing

[Daniel Maenle @DanTheSQLMan]

@SQLGroup Brewster's millions would have been over with a poor entity framework query.

[Daniel Maenle @DanTheSQLMan]

@ShishirShelke1 You canceled my order 2 times thinking it was fraud.

[Daniel Maenle @DanTheSQLMan]

@lavabit_support I have been unable to login and have not had response yet from email support request.
Some attempts saying server maintenance,others saying account is locked.

[Daniel Maenle @DanTheSQLMan]

@matt_vanswol I recently had to get a chest x-ray. Paid in cash. 7$. Oh wait I live in the Philippines.

[Daniel Maenle @DanTheSQLMan]

@erikdarlingdata @GFritchey And make sure you talk about sp_quickiestore

[Daniel Maenle @DanTheSQLMan]

@SQLGroup How do I register?

[Daniel Maenle @DanTheSQLMan]

@kekline I have several on my GitHub and can go over them

[Daniel Maenle @DanTheSQLMan]

@AMAZlNGNATURE @samrei

[Daniel Maenle @DanTheSQLMan]

@GrandPOOBear I give to help prevent more Poo stomps to other great players.

[Daniel Maenle @DanTheSQLMan]

@sqlagentman I moved out of the USA. Welcome to come to the Philippines. Happy yo go diving and play DND if you do. 😃

[Protect Kamala Harris ✊ @DisavowTrump20]

RETWEET if you stand with Jack Smith against Trump!

[Daniel Maenle @DanTheSQLMan]

Going to be taking more active effort in life. What books help a reflective and meaningful life to you? Taking recommendations.

Starting with Born a Crime by Trevor Noah.
And Guns Germs And Steel. By Jared Diamond

[Daniel Maenle @DanTheSQLMan]

Kansas city doesn't have enough heart medicine to handle these games!

[Daniel Maenle @DanTheSQLMan]

@erikdarlingdata But what if everything I know I learned from you?