Crowdstrike causes world-wide IT outages

[overscan (PaulMM)]

So, our new security manager was talking about bringing in Crowdstrike where I work. Hopefully this idea will now be kiboshed after Crowdstrike deployed an update which

1) made Windows servers and desktops crash
2) rendered Windows servers and desktops unable to boot

The only fix is to boot the affected device to safe mode, log on with an admin account and delete a driver file. Most corporate computers affected will probably use Bitlocker, so you'll also need to obtain and enter a 48 digit key. Fun.

Some companies have thousands of servers and tens of thousands of laptops affected.

 

[Graham1973]

I've had to break open my 'piggie-bank' and 'slit open the matress' to get some cash out as I've heard that point-of-sale equipment has been affected by this. And people present a 'cashless' society as being the inevitable wave of the future...

 

[Orionblamblam]

A practice run on what's coming...

https://twitter.com/x/status/1814301431626694940

View: https://x.com/ianbremmer/status/1814301431626694940

 

[Orionblamblam]

https://twitter.com/x/status/1812336670009700476

View: https://x.com/MostlyPeacefull/status/1812336670009700476

 

[Orionblamblam]

 

[overscan (PaulMM)]

A cloud based financial application which is vital to our business was unavailable for 10 hours, but luckily starting at 4.33pm on Friday, so direct user impact was limited.

 

[Orionblamblam]

A cloud based financial application which is vital to our business was unavailable for 10 hours,

Just wait till this happens *on* *purpose.*

"Remember when you'd walk into a room and flick a switch and lights would come on?"

 

[overscan (PaulMM)]

The fact that half of the Fortune 500 all use the same system is pretty terrifying when you think about it.

This "channel file" was apparently just a configuration update, not code, but the load and parsing of this file is done by the driver running at kernel level, and loading this bad configuration file caused the driver to completely crash Windows.

This is garbage programming. Presumably there is no validation that the configuration file was valid, it's just applied, which allowed this to occur. No error handling.

The "fix" only replaces the bad channel update with a good one, it does not fix the bad code in the driver. If Crowdstrike make the same mistake with another bad channel file update today, or tomorrow, everything will crash again.

 

[aim9xray]

"Yeah, it turns out that Fred, the new guy pushed the channel file."
"Oh, the new guy. Didn't we just hire him away from F-35?"

 

[Orionblamblam]

https://twitter.com/x/status/1814352521860714771

View: https://x.com/thdxr/status/1814352521860714771

 

[BlackBat242]

Crowdstrike

Well, it lived up to its name... and made the crowd of Windoze computers go on strike!

 

[Forest Green]

Skynet is Genesis is Crowdstrike.

 

[Orionblamblam]

Skynet is Genesis is Crowdstrike.

Skynet was *competent.*

That seems to be the main difference between Sci-Fi AI threats and reality: real world code is jam packed with flaws and vulnerabilities. The real world is looking more and more like Star Trek: TOS: by the time we can actually literally communicate with AI by having a conversation with it, some dude with smarts and charisma will be able to talk the AI into killing itself.

 

[publiusr]

Well, it lived up to its name... and made the crowd of Windoze computers go on strike!

Crowdstrike sounds like a hacker name to begin with.

I bet you'll find they have 2600 magazine hardcopies scattered here and there.

 

[Grey Havoc]

Microsoft IT outage: Global services slowly recovering after bug causes chaos

The incident has sparked concern over the vulnerability of the world's interconnected technologies.

www.bbc.com

Holidays in chaos after global IT meltdown

Hundreds of thousands of people have had their foreign summer holidays thrown into jeopardy by a global IT outage that caused chaos at British airports on Friday.

www.telegraph.co.uk

Airports, trains and checkouts: Everything affected by global IT outage

Microsoft disruption causes companies around to world to grind to a halt

www.telegraph.co.uk

 

[Grey Havoc]

CrowdStrike file update bricks PCs around the world

Falcon Sensor putting hosts into deathloop - but there's a workaround

www.theregister.com

CrowdStrike shares sink amid global IT crisis

Emergency services, medical practices, airlines, banks, and more all crippled

www.theregister.com

Number of NHS IT systems hit by CrowdStrike outage grows

Cancer treatments are in jeopardy across multiple healthcare facilities

www.theregister.com

Reboots best fix for Azure VMs CrowdStrike borked, says Msft

Have you tried turning it off and on again, like, a bunch?

www.theregister.com

An angry admin shares the CrowdStrike outage experience

CrowdStrike? More like ClownStrike! Amirite?

www.theregister.com

The non-IT impacts of CrowdStrike's bad patch on daily life

Oh, was it supposed to be Y2K24?

www.theregister.com

CrowdStrike Windows patchpocalypse could take weeks to fix

Our vultures gather to review this very freaky Friday

www.theregister.com

 

[Forest Green]

Do they not have the ability to restore the systems to a previous date?

 

[Grey Havoc]

Due to the nature of cloud computing based systems, that's not really an option. The situation has been made even worse by the remote working/WFH fad that was all the rage in recent times.

Here are a couple of more articles in passing from the Register yesterday:

Technical Details on July 19, 2024 Outage | CrowdStrike

Learn more about the July 19, 2024 CrowdStrike outage and the technical details related to it.

www.crowdstrike.com

Cybercriminals quick to exploit CrowdStrike chaos

Who loves a global outage? Phishers, fraudsters and all manner of creeps

www.theregister.com

 

[sferrin]

A cloud based financial application which is vital to our business was unavailable for 10 hours, but luckily starting at 4.33pm on Friday, so direct user impact was limited.

Now imagine we were all dependent on crypto currency.

 

[Grey Havoc]

[shudders]

 

[Orionblamblam]

Now imagine we were all dependent on crypto currency.

Or using some form of "social credit score" in order to access not just purchasing, but transport, medical care, etc. Bad as that would be in normal operations, when the system goes down nobody can do anything.

Some people want to get rid of private vehicles in favor of public transport and rideshares. A system glitch or hack could easily cause all the vehicles to just stop, if not suddenly decide to merge with walls at 100 MPH.

 

[Grey Havoc]

https://edition.cnn.com/2024/07/20/tech/timeline-crowdstrike-system-outage/index.html

From Wall Street darling to firm behind the world's worst IT outage - who are CrowdStrike?

Crowdstike has been one of the best-performing stocks this year, thrilling investors with AI-enabled cyber protection.

news.sky.com

 

[Forest Green]

Now imagine we were all dependent on crypto currency.

 

[martinbayer]

The real world is looking more and more like Star Trek: TOS: by the time we can actually literally communicate with AI by having a conversation with it, some dude with smarts and charisma will be able to talk the AI into killing itself.

I'd pay to watch that.

 

[publiusr]

Or using some form of "social credit score" in order to access not just purchasing, but transport, medical care, etc. Bad as that would be in normal operations, when the system goes down nobody can do anything.

Some people want to get rid of private vehicles in favor of public transport and rideshares. A system glitch or hack could easily cause all the vehicles to just stop, if not suddenly decide to merge with walls at 100 MPH.

True, but I would mandate all businesses have the old mechanical credit card clunk-chink deals and at least one mechanical cash register.

 

[Orionblamblam]

True, but I would mandate all businesses have the old mechanical credit card clunk-chink deals and at least one mechanical cash register.

If The System goes down, no form of credit card will really work. But cash should always be valid.

 

[Orionblamblam]

I'd pay to watch that.

Here ya go. I accept PayPal.

View: https://www.youtube.com/watch?v=dIpsvF50yps

View: https://www.youtube.com/watch?v=EtWmLIoN6Sg

 

[martinbayer]

Here ya go. I accept PayPal.

View: https://www.youtube.com/watch?v=dIpsvF50yps

View: https://www.youtube.com/watch?v=EtWmLIoN6Sg

I meant IRL, not cheesy make believe.

 

[PMN1]

 

[PMN1]

 

[Orionblamblam]

I meant IRL,

You did not specify. And here I thought Germans were supposed to be precise.

not cheesy make believe.

Which, of course, this isn't.

 

[Arjen]

The Real Life part: two actors following a script. Shatner and the actor who had the thankless job of voicing the prop that represented an AI-driven machine.
What the clip doesn't show, is an AI driven to suicide.

The real world is looking more and more like Star Trek: TOS: by the time we can actually literally communicate with AI by having a conversation with it, some dude with smarts and charisma will be able to talk the AI into killing itself.

You didn't deliver, there was no AI involved.

 

[Foo Fighter]

Cash is fine when the ATM or the computer system in one of the few bank branches remaining, works. Other than that we are back to barter.

How much cash do we actually carry on an average day and how long would that last in an average week?

How much is the average week grocery bill at the average stupor market?

Can you get to the average stupor market on the average week?

I have to have mine delivered, it is not fun.

 

[Cannonfodder43]

https://twitter.com/x/status/1814392957203784061

View: https://x.com/MCCCANM/status/1814392957203784061?s=19
Obvious parody but I got a good laugh out of it. SFO was quiet on Friday.

 

[Grey Havoc]

Joint Chiefs Chairman says DoD operations not affected by widespread CrowdStrike 'glitch' - Breaking Defense

"I'm sure our adversaries are looking at this as a way to, I would say put sand in our gears when we're trying to generate combat power," Gen. CQ Brown said.

breakingdefense.com

He would seem to be trying to run a rather threadbare bluff there.

 

[aim9xray]

Here is an expert analysis and succinct explanation on what happened:

View: https://www.youtube.com/watch?v=wAzEJxOo1ts
It's so easy, even a rocket surgeon could understand it.

A most cogent comment was "If we don't sugar coat it, it's a kernel-level remote backdoor running unsigned code. You can't make it any worse. CrowdStrike is not the only one. Ubuntu live patch at least betatests patches on free users first. Then we have Intel ME. WiFi cards running unaudited binary blobs (which is why you often can't boot from it). You are not in control and this will repeat."

 

[sferrin]

Test run.

 

[overscan (PaulMM)]

Nope, just inadequate testing and bad practises.

 

[overscan (PaulMM)]

Here is an expert analysis and succinct explanation on what happened:

View: https://www.youtube.com/watch?v=wAzEJxOo1ts
It's so easy, even a rocket surgeon could understand it.

A most cogent comment was "If we don't sugar coat it, it's a kernel-level remote backdoor running unsigned code. You can't make it any worse. CrowdStrike is not the only one. Ubuntu live patch at least betatests patches on free users first. Then we have Intel ME. WiFi cards running unaudited binary blobs (which is why you often can't boot from it). You are not in control and this will repeat."

It's a good analysis.

 

[Grey Havoc]

Microsoft blames EU rules for allowing world’s biggest IT outage to happen

European Commission deal prevented software giant from making security changes that would have blocked CrowdStrike update, claims tech giant

www.telegraph.co.uk

How the West became dangerously dependent on Microsoft

Tech giant’s ubiquity among businesses means any failure has wide-reaching implications

www.telegraph.co.uk