Vangelis Koukis

I like to explore technical things.

[Scott Hanselman 🌮 @shanselman]

Here’s the thing folks. I’ve been coding 32 years. When something like this happens it’s an organizational failure. Yes, some human wrote a bad line. Someone can “git blame” and point to a human and it’s awful. But it’s the testing, the Cl/CD, the A/B testing, the metered rollouts, an oh shit button to roll it back, the code coverage, the static analysis tools, the code reviews, the organizational health, and on and on. It’s always one line of code but it’s NEVER one person. Implying inclusion policies caused a bug is simplistic, reductive, and racist. Engineering is a team sport. Inclusion makes for good teams. Good engineering practices makes for good software. Engineering practices failed to find a bug multiple times, regardless of the seniority of the human who checked that code in. Solving the larger system thinking SDLC matters more than the null pointer check. This isn’t a “git gud C++ is hard” issue and it damn well isn’t an DEI one.

[Lulu Cheng Meservey @lulumeservey]

CrowdStrike CEO is getting pummeled for his response to the global outage.

Why everyone hates it:

1) WEAPONS-GRADE CORPO SPEAK

Let’s be clear. Legalese doublespeak is designed to dodge and obfuscate rather than inform or communicate. This statement was obviously written by a committee of lawyers and middle managers whose only goal was to avoid legal risk and threats to their own job security.

If you can’t understand what the statement is even saying, it’s working as intended.

2) NO APOLOGY

The first words should be “I’m sorry” — but you won’t find that anywhere in this statement. Nor the watered down “I take responsibility.” Not even the weasely “We regret…” Nothing!

It comes off as cowardly and callous. CrowdStrike caused an outage that took down airlines, a stock exchange, hospitals, ICUs. People might have died.
And the CEO is not sorry.

3) PASSIVE VOICE THROUGHOUT

This is such classic move to avoid accountability, it’s even become a joke: “Mistakes were made!”

This statement is almost comical in its efforts to dodge assigning responsibility.

“This issue has been identified…a fix has been deployed.”

Which issue? (Global outage)
Who caused the issue? (You, CrowdStrike)
What fix? (🤷🏻‍♀️)
Did it work? (🤷🏻‍♀️🤷🏻‍♀️)

4) DISMISSING CUSTOMERS

Don’t bother us with your petty complaints of power going out in your local hospital!

“We refer customers to the support portal” and “we further recommend customers ensure they’re communicating with representatives through official channels.”

You, the customer, are bothering us and making our lives harder.

5) USELESS INFORMATION

So many words, so little meaning. This statement says nothing useful — not what the problem was, who caused it, what they learned, what the fix it, how long it might take, what they’re working on, or anything at all.

It assigns extra work to the customer by telling them to go through official channels but does NOT then link to the official channels. The onus is on you, customers!

If you want to learn more about how CrowdStrike has ruined your day, you go do the work.

[Vangelis Koukis @vkoukis]

@George_Kurtz @CrowdStrike The root cause is not a content update, it is a bug in csagent.sys, a fact which the technical overview omits deliberately:
https://t.co/06t5u3RcxO

[Vangelis Koukis @vkoukis]

4. There is no escape plan when this critical component misbhehaves, apart from hijacking the machine as an attacker with local access [Safe mode, Command Prompt, assuming access to the BitLocker recovery key], and this is exactly what has happened now.

Thoughts?

[Vangelis Koukis @vkoukis]

The root cause was a bug [a "logic error"] in *csagent.sys*, which I understand is the main in-kernel agent for Falcon Sensor. The "single content update" [in a nutshell: antivirus definitions] only *triggered* the bug. The bulletin still doesn't answer two critical questions:

[Vangelis Koukis @vkoukis]

3. I bet an update to csagent.sys itself is coming soon, to fix the actual root cause. For all we know, their internal testing of content updates could be using an updated version of csagent.sys before pushing, this is why they never saw machines bluescreening almost immediately.

[Vangelis Koukis @vkoukis]

It's a shame that the technical bulletin on the global @CrowdStrike incident avoids being explicit about what the root cause was. So, let's embark on a bit of guessing. The bulletin, for context:
https://t.co/NiOOGyttcA
[Thread ⬇️]

[Vangelis Koukis @vkoukis]

Λοιπόν, the plot thickens, και το γράφω μήπως φανεί χρήσιμο και σε κάποιον άλλο: Η @Alpha_Bank λέει ότι δεν μπορώ να πληρώσω οφειλές στην @AADE_IAPR με πιστωτική, μόνο με χρεωστική. Βέβαια, με πιστωτική άλλης τράπεζας έπαιξε. Ποιοι βάζουν τον περιορισμό, και πού τον περιγράφουν;

[Vangelis Koukis @vkoukis]

Το σύστημα πληρωμών με πιστωτικές @Alpha_Bank είναι κάτω, ή είμαι εγώ ο τυχερός; Πληρωμή ΑΑΔΕ --> ΔΙΑΣ --> βάζω τα στοιχεία της κάρτας που ξέρω απ'έξω και χρησιμοποιώ καμιά 10αριά χρόνια --> άμεση απόρριψη. Τηλεφωνική εξυπηρέτηση, "έχετε δοκιμάσει με άλλον περιηγητή;" 🙄

[Vangelis Koukis @vkoukis]

Ζητάω να το escalate, χρόνος αναμονής άνω των 20', και μετά από λίγο μου το κλείνει. Δοκιμάζω με άλλη πιστωτική, και παίζει κανονικά. Πόσο δύσκολο είναι να ενημερώσουν τους ανθρώπους τους που απαντούν τα τηλέφωνα "ναι, έχουμε τεχνικό πρόβλημα, θα το έχουμε φτιάξει μέχρι <τότε>;"

[Sotiropoulos @Sotiropoulos]

Είδατε πουθενά δήλωση του Λοβέρδου ή του Χρυσοχοϊδη που ειναι και τώρα υπουργός για την καταδίκη της Ελλάδας από το ΕΔΔΑ για τις οροθετικές γυναίκες;

Δεν καταλαβαίνω γιατί κανένας δημοσιογράφος δεν τους έχει ρωτήσει κι αν το εχει κάνει γιατί δεν έγραψε ότι τους ρώτησε μάταια.