Today in health ID, we're going to take a look at what we are to do about these automated updates. Gnarly problem. My name is bill Russell. I'm a former CIO for a 16 hospital system. And creative this week health set of channels and events dedicated to transform healthcare. One connection at a time. Today's show is brought to you by Panda health.

Digital health is hard and Panda makes it easier. Quickly and comprehensively vet digital health solutions and be fully prepared. And informed for your next meeting. Panda helps health system leaders make confident decisions about digital health without the complexity and burden. Figuring things out on their own. They help you to get smarter, faster through peer input, market intelligence and advisory services.

Check them out at this week. health.com/panda.

All right. Hey, this story and all the new stories recovery you can find on this week. health.com/news. We are like inches away from our $150,000 goal for raising money for Alex's lemonade, stand and fighting childhood cancer. If you want to be a part of that, put us over the top. Go ahead and click on the link on our website on our front page and give today. Last thing, share this podcast with a friend or colleague. Here's what you do.

You share it with them. They listened to it. You listened to it and you just talk about it. That's mentoring. It's really that simple. It's. It's a form of metric. It's not complete, but it's a start of a process where they understand your thinking, you understand their thinking and you're helping to mold the minds of the next generation. Of leaders, they can subscribe wherever you listen to podcasts. All right.

Let's take a look at this story. This is on C I O. Dive. And here's summary the recent software update failure in CrowdStrike's. Let's see. The name of the story is after Kraft Stripe outage, what will become an automatic it updates. Good question. Here's a summary, the recent software update failed. Failure in CrowdStrike's platform costs, significant disruptions, worldwide height highlighting. The risks associated with automatic updates in it systems, the incident which led to blue screen of death on millions of windows, computers impacted various sectors, including airlines, banking, and resulted in healthcare. And resulted in financial losses, exceeding $5.4 billion for fortune 500 companies. This event underscores the need for it leaders to adopt more stringent quality assurance and risk mitigation practices such as Canary deployments. And staggered updates to prevent widespread issues from faulty software updates in the future.

Let me let me touch on those last couple of sentences here. Go down in the article a little bit. Preventing the next big one. Here we go. Analyst's has flagged the importance of risk mitigation techniques, such as Canary deployment preliminary roll-outs under controlled conditions prior to product deployments. Following the outage CrowdStrike announced it is taking steps to restore customer confidence. Such as adding additional validation testing and releasing new updates through a staggered deployment strategy.

There you go. Quality assurance and regression testing are critical, said Jen clang, Microsoft global partnership director at tech systems. You cannot just blindly trust, whatever updates are being pushed out. Executives must consider whether critical systems. And applications suited here immediately to release cycles. Or if it's pertinent to delay updates until their operational safety can be confirmed. When you do your business continuity and your disaster recovery planning, you have to think about how quickly you accept those updates said, cling. There were a lot of companies that immediately accepted what was pushed out. In a post CrowdStrike world, a shift in perspective is already underway.

There's going to be, and already is a greater level of stringency around testing upgrades. Freshed. Said you can do digital twin models. More and better use of synthetic data, better testing of things. Before they occur. Okay, so that gives you some idea of what they are. Suggestion suggests suggesting it's early in the morning.

What time is it? Five o'clock in the morning, recording this for. For the day did not get it done yesterday watching the Olympics over here. Sorry about that. Let me tell you how I'm thinking about this. This problem is not going to go away. In fact, the. The problem is only going to get more acute and the reason it is going to get more acute. Is because this is the world we live in. The attackers are taking advantage of holes quicker. They're using AI to look at the same. Information that you are looking at.

So when there's a release of a vulnerability, they get that information, they feed it into their AI model. They develop a, an attack that quickly and they go to town. And the reason we went to Crosstrek in the first place or some similar type of technology. Is because those updates get detected very, or those attacks get detected very quickly. And could get detected, say in Australia. And then the fix is identified and then rolled out via crowd strike around the world. And that process is very fast because it's very fast.

We feel more protected. We have someone who's looking at after us. It's sorta like ADT on your house, right? There's someone who's monitoring the world that you can not possibly hire enough people to monitor. And creating the the protection against those things. So that's the reason we did it.

Let's not forget that. That's it for starters. And the the reason I say it's only going to continue and it's only going to get worse is AI is going to take over more and more of these administrative tasks for the very reason that I just rattled off, which is. The pace at which things are moving. Are. Beyond human response times.

We cannot fall back into what we were doing before. It's just not possible. Now we are going to talk about how to put some parameters around CrowdStrike in other. Other releases, but I think it's incumbent upon us to understand that these releases are happening. It's CrowdStrike, it's Microsoft it's probably any number of other systems you have.

It's definitely every cloud system. We have one of the things I've heard. From CEO's as we're having these conversations is how do we get in front of these? These cloud. Updates that are going on some of these systems, which happened to be okay for 95% of the users, if not 98% of the users, but in an healthcare it world where there's a significant amount of complexity. Those updates might break a workflow or break some other thing that's going on within the healthcare system. And so it's it.

You see how broad the problem is. It starts with, some core systems that you're taking these releases on for security and. And continuity reasons. And then it. It quickly moves out to the cloud systems that you're utilizing. And so it's going to continue. How do you get in front of it?

A conversation with CrowdStrike is definitely a way to go. To understand what they're doing and how they restoring confidence. When you sign a contract with a company like CrowdStrike with Microsoft. It's not just generic, big tech. It is these are companies that you're placing an awful lot of confidence in you're putting a sensor. Essentially at the below the trusted ring of the computer, which is what we've done with CrowdStrike.

And when you're doing those kinds of things, it requires a different level of relationship with that organization. And so I'm surprised because this essentially. And it, and to be honest with you, I would not have been ahead of this either. But every CIO in the world did not push CrowdStrike on this prior to the incident. Now every CIO post this incident. Are going to push CrowdStrike on it, but the question is, are we going to push Microsoft?

Are we going to push. Whoever the other. Companies are that we allow this level of access, this level of immediate updates to our systems, especially automated and without a human in the middle. I would identify every single one of those that you have today. I would then sit down with each one of those partners. And I would understand their testing and their deployment methodology.

I would also understand if it is possible for you to limit their push within your four walls of your, the. The domain that you're responsible for. Limit that push to a certain number of machines to start. That staggered deployment kind of thing they say they're going to do it. I'd want the ability to do that within my health system. Hey, I want you to deploy it over here first. And we will see if anything happens.

If nothing happens, we will then deploy elsewhere.

Definitely want to do that. I don't know. I, I'm trying to think of what else I would do. I would definitely have everything documented. I would definitely see if we can get a staggered release. Again, you're not going to have too much time between the staggered release, but keep in mind that this update was only out there for a little over an hour, if that, and it impacted most. Most of the world in that hour. I think we're going to be pushing them.

I'd like to see us push Microsoft as well. I they're going through this fairly unscathed and I'm not sure I understand why. They're going through this unscathed. I will say again, Microsoft is sponsor of the show, but I will say again the fact that release could take down that system. Also speaks to the fragileness of that operating system. Like it should be able to catch that code and keep from doing a blue screen. Now again, I'm not at the level that I can understand. What was happening and that kind of stuff.

But I do know. That that people who are looking at this can ask the question, would this happen to a Unix machine? Would this happen to a Mac? And I think the answer is no, it would not have happened. But we will have to see time, will tell. This is a gnarly problem. We have to get in front of it.

It represents now a whole bunch of work that we need to do on something that we had taken for granted for a long time. And that's just what we need more work on our plates, but this is what they pay us for. It's to make sure that the systems are reliable and do not go down. And part of that is managing these vendors. All right.

That's all for today. Don't forget. Share this podcast with a friend or colleague. Use it as a foundation for mentoring. And we want to thank Panda health for investing in our mission to develop the next generation of health leaders. 📍 You can check them out at this week. health.com/panda. Thanks for listening.

That's all for now.