Re: Crowdstrike fiasco

On 2024-07-21 02:11, Rob van der Putten wrote:

The problem isn't CrowdStrike. Or even Windows. The problem is
unattended automatic updates.
With a manual update, if things don't work quite the way they should,
you just revert back to the previous situation. And with enough
redundant systems, the whole just keeps working while you fix this
particular computer.
With unattended automatic updates on a vast scale, half the world just
crashes.
It's a bean counter thing: Why spend money on IT staff when the software
can do the IT thing for you.

I slightly disagree: At a certain scale, automatic updates are the only feasible option.  But there are good and bad ways to do automatic updates, and Crowdstrike (and the IT departments that deployed it) seemingly went with bad.
The good way to do this would be like how big tech companies roll out new versions of software to their fleets of servers: In gradual stages, with comprehensive monitoring and an automatic halt if important metrics start to tank.
Assuming Crowdstrike collects any sort of telemetry from its deployments, they should have been able to gradually roll their updates out across customers, and then notice when those customers weren't coming back online.  Likewise, assuming Crowdstrike provides any knobs for IT departments to control its updates, individual IT departments should have been able to control this update's deployment and react accordingly.  This can all be done automatically, especially with such a black-and-white metric as "is the client online".
So I agree that manual updates would have prevented this.  But I don't think manual updates are feasible in many circumstances, and this could have been prevented even with automatic updates.