🧵Understanding technical challenges behind @CrowdStrike Falcon EDR caused Windows outage.
I see a lot of opinions on how CrowdStrike or Microsoft could have prevented what is said to be the largest IT outage in history. Let's breakdown some of the technical details to understand why it is not that simple.
🧵1/8: Why Windows let one program take down the entire system?
Like any mainstream operating system, Windows isolates userspace programs and will not allow them to directly affect each other or the operating system itself. Bugs happen in code and applications do crash, it does not normally affect anything outside of that application. However, the program in spotlight, CrowdStrike Falcon, is an EDR (Endpoint Detection and Response) software. It allows enterprises to monitor and defend their systems from cyber threats and also to respond and recover from security incidents. Think of it as a sophisticated anti-virus. An EDR needs to monitor and control other programs which as I said is not possible for an isolated userspace program. Hence, part of the Falcon EDR is implemented as a kernel driver that runs in higher privilege and has higher authority over other userspace programs. Now, kernel drivers are sensitive pieces of software that links and runs together with the operating system kernel. A crash in kernel driver is treated a crash in the OS itself and it will bring down the entire system. This is no different in Linux or Mac OS. Enterprises knew what they were doing when they installed Falcon and its kernel driver.
🧵2/8: Why Windows is known for BSOD unlike Mac OS?
Mac OS or Linux are not immune to kernel driver crashes either. When such error occurs, Windows displays a BSOD to prevent further damage to the system. MacOS and Linux results in a “kernel panic”, which is less visually dramatic but serves a similar purpose. Having said that, the reason we see it more often in Windows is that Windows is designed to support a wide range of hardware, which brings a lot of third-party device drivers that cause issues. Mac OS, on the other hand, is designed to only work with a limited set of hardware, which Apple controls. It allows Apple to have tighter control over the drivers used.
🧵3/8: Why Windows needs anti-virus while MacOS and Linux don't?
It is absurd to think that Mac OS and Linux are not affected by malware. Popularity of Windows makes it a prime target for malware creators. There are other factors also in play such as software distribution methods. In a nutshell, enterprises deploy EDR on Linux and Mac OS too. The CrowdStrike Falcon sensor is available for Linux and Mac OS.
🧵4/8: Why CrowdStrike didn't do phased updates instead of one global update? Why Microsoft does not test updates to critical drivers?
Phased rollouts allow for early detection of issues in a controlled environment, minimizing the impact of potential faults. While this could be done, there is an urgency factor to be considered. The Falcon EDR needs to be continuously updated to keep up with fast evolving threats. The EDR contains a knowledge base of TTPs (Tactics, Techniques and Procedures) used by threat actors. When CrowdStrike's threat intelligence discovers a new TTP, it is in everyone's interest to update the sensors as soon as possible so that all their customers are protected against newly discovered threat. For this reason, updates to Channel files (such as C-00000291*.sys) occur several times a day as mentioned by CrowdStrike in their blog [1]. This means they cannot afford a comfortable QA cycle. Having said that, it is their job to release well-tested updates. I am just pointing out the challenges involved.
🧵5/8: Why can't we use safer languages instead of C/C++ for kernel drivers?
Initial analysis posted on X shows that the crash was due to an illegal memory access causing a segmentation fault. While there are languages that don't allow programmers to cause memory safety issues, most of them come with fat runtimes. Let's just say we cannot write kernel drivers in language of our choice for reasons that cannot be explained in one post. Operating systems are written in C/C++ and the drivers that link with them are too. While operating system code is extensively tested to not cause such crashes, third-party drivers become the weak link. To solve this exact problem, Linux recently introduced support for writing kernel drivers in Rust, a safer alternative [2]. There aren't many rust drivers out there so far and only time will tell [3].
🧵6/8: Are they any alternative approach for EDRs than kernel drivers?
Yes and no, Linux kernel supports a feature called eBPF (extended Berkeley Packet Filter). If you recall my answer to first question, kernel drivers are not subjected to same isolation as userspace programs. Well, eBPF provides sort of an isolated runtime inside the kernel for observability usecases. However, it has limitations compared to a kernel driver. Microsoft is working on eBPF support for Windows and Microsoft's EDR (Defender for Endpoint) leverages eBPF in Linux [4,5].
[1] https://t.co/xPXegoZCEF
[2] https://t.co/MiQbohWgLz
[3] https://t.co/nto7PZZgpO
[4] https://t.co/vfxO78P63Z
[5] https://t.co/4hR54PfhRR
#Crowdstrike #Windows #CrowdStroke