This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

 Welcome to this Weekend Health It. It's Newsday. And today we're gonna go through the Sky Lakes ransomware event in detail with former CISO for Intermountain Carl West. My name is Bill Russell. I'm a former CIO for a 16 hospital system and creator of this weekend Health it, uh, channel dedicated to keeping health IT staff current.

And engage. Special thanks to Sirius Healthcare, our news Day sponsor for today's show and health lyrics and WWT intel for investing in our mission to develop the next generation of health leaders. If you wanna be a part of our mission as well. You can become a show sponsor. Send an email to partner at this weekend, health it.com.

Uh, quick note. Check out our latest article on the changing role of the CIO. This has gotten a ton of hits. It's a great article. It highlights BJ Moore, ed Marks, William Welders, Ressa Springman, Craig Richville amongst others. The role is changing. You want to hear how it's changing from those people who are living it.

All right, onto today's show. . Tuesday. Okay. Today it is a Newsday episode and we're gonna do something a little different. We're gonna camp on one topic and one story, and that is, uh, ransomware and the attack, specifically on Sky Lakes Medical Center. And today we have Carl West, with us, with Sirius Computer.

How, how you doing, Carl? I am good and you've picked an awesome topic, bill. Oh, really? It's relevant these days. This is a conversation everywhere I go and so you've picked a and, and certainly the president is having this conversation right now at In his world. I was shocked to find. This video, to be honest with you, I was, I was shocked for a couple reasons.

One is as ACIO, Deloitte was our internal auditor, and they forbid me from talking about our security posture with the media. They just flat out said articles and whatnot, do not talk to the media. Now, if you wanted to share it in a smaller setting, a more controlled setting. With just healthcare leaders, by all means, go ahead and do that for the good of the industry, but do not allow it to get published in black and white.

And I, I remember way back when after I was told that, uh, I saw that Darren doin, I. CIO at Cedars wrote an article for the Wall Street Journal on Preparedness for Healthcare for security. If you haven't seen that article, it's exceptional. It just shows the thinking that Darren has. And when he got done, I, I, I saw him at a conference.

I said, man, that was an exceptional article. He goes, I got in so much trouble for writing that article. You know, and, and in fact, I think most healthcares have that posture bill. We don't want to talk about, we don't want to broadcast what we're doing. It puts a target on everyone's organization, and so the article.

You're referencing actually the video from Sky Lakes is, it is so telling and, and behind the scenes Bill, many of us as CSOs we're calling, talking about, we are calling and talking about what did just happened and we make notes of the details. But to see it so open in every aspect from the attack, this threat vector, what happened minute by minute.

If people haven't seen that Sky Lakes Medical Center presentation, go out and take a look at that. It is very telling. It's the kind of thing every CISO needs to be looking at, understanding what happened. What are their top five or six recommendations, what are they telling us we should be doing, and what do we need to be doing?

Very informative. All right. Here's how we're gonna structure this. You and I are gonna go through it. And we both watched the video. I've outlined it here, so we're gonna start by the events that happened. So identifying that you've been ransom, because there was a time period where things were happening and they're like, maybe this is just normal slowness of the network or systems and that kinda stuff, which happens, which is part of the process.

We've gotta determine what it's, so we're gonna identifying it. We're gonna do a section where you and I talked about working through it, right? So now ransomware, what are the steps they. Uh, we'll go back and forth on that a little bit, and then they do share lessons learned. I wanna talk about their lessons learned, but I also wanna tap your brain on what are your lessons that, that you learned and, and I'll share some of the ones that I learned.

Does that make sense? And Bill, right as we go into this for our listeners, I would tell you what you're going to hear is if you look at a maturity scale, it was ad hoc what occurred as ad hoc, and that's what happens in most cases. So the very first thing I would tell you as you listen to this and you think, why didn't they?

Why didn't they? You need to have documented processes and playbooks so that you know exactly what you want your caregiver to do, exactly what your essays, your DBAs, your network engineers should be doing. And as you listen to this, I'm just so pleased that Sky Lakes was so candid and we'll hear ad hoc.

Yeah, it's, it's very easy to sit there and go, oh, why didn't they, why didn't they, why didn't they? Okay, so put this in context.

Okay. They're, they're, they serve a radius of about 75 miles. In other words, 75 in some cases is a hundred miles or 140 miles to the next hospital. Alright, so this is not in downtown la This is, is pretty remote. It's, it's a community connect site, which means they probably don't have a significant IT budget.

I, I would guess there's staff. 25. I don't.

A little generous. I think that's, I think that's way too high. Yeah. I think if they had two or three, I'd be surprised. Dedicated to cyber. It's a small shop. Yeah, I was talking 25 total in it, because their EHR is being handled by Asante and others. So it's a small organization. Right. Small organization, small budget.

Small allocation for cybersecurity. So I, I would say to people, give, cut them some slack. Now we're gonna point something and Bill, this is healthcare in America, so you don't have to cut 'em any slack. This is 80% of healthcare in America today. All right, well, that's, that's a little scary. But, but we'll come back to that because we're gonna look at both things.

I wanna take some time and look at the position that as was put in as the host system. Because we have large health systems that have community connect sites and other types of sites. They're sharing systems with smaller health systems in their community that they could be opened up to attack through those sites.

So I wanna look at that, the host system, but I also really wanna look at the events that went on at this health system and, uh, see if we can't get some wisdom out of it. So October, let's see, October 26th. 26th. Yep. Yep. Phone calls stating systems were slow, the computers were slow. Earlier that day, about noon on October 20 employees at the medical center.

It was an email, email that had information about bonuses. She clicked on it, and we're gonna talk about the email in a minute, but she had met with HR previously and assumed the email was about that meeting. Within about 30 minutes, the system went out to a malicious site that delivered the payload to the computer that contained the ransomware code.

Her computer froze and she couldn't control her mouse. When she regained access to her mouse, she closed the window and didn't think much about it, nor would any of us. Your system slows down, a window, slows down. You close that window, you go back and, Hey, everything's working. Well, about an hour later, PowerShell command, cobalt Strike was enacted.

Alright, so Carl, let's walk through this a little bit. That's a pretty, I would assume that's a normal or typical ransomware attack. But that presupposes that there is a normal attack. Is there a normal attack? This is very common. It's, uh, the kind of thing I hear every day. And those symptoms, those things that are happening are things around which.

Processes structure needs to be, uh, de developed. And the CIO is very funny, candid in his video when he says, how would you feel when this happens? And you get this call, we know how we'd all feel just like we've just been punched in the gut and we're all terrified. And I just felt for him as he's representing, here's what's just happened and, and it's been happening so recently.

A window that software or even a monitoring would've.

I think a number of things that could help our listeners. First, email education, phishing education. She had something happen later, other things occur. Uh, we, we need to have our users educated so they're alert to, if we can have a phish hook, uh, some type of an alert in the mail system so that if you're worried, if you're concerned you've been trained enough through phishing education, click.

This goes right to cybersecurity and they're taking a look, analyzing the email prior to email coming in. Many organizations have put into place sandbox technology and what it does, it examines every piece of mail coming into an organization. It's a best practice. It's a thing I strongly recommend. There are many providers of this type of technology.

It would have detonated and attempted to open. And would have removed this, should have removed it from the email system. But even if you miss, if you are doing regular phish campaigns, if you're educating, training your employees when you see, when you click, be careful, don't click links, uh, don't give out credentials.

Those are all things that should have been a part of. The processes and, and I'm not saying anything that Sky Lakes and their CCIO didn't point out. He knows that he recognized it after the fact. I think we just have got to get these kinds of products and these kinds of solutions funded. Yes. So it's multilayer.

I started at the point that the email got clicked, but there, that email probably shouldn't have.

His or her role should be, I've got to get all of this stuff out of the system because no matter how much I educate, no matter how much I train, these people are good. And this looked like that's what I, I caught the lady said, I just sat in an HR training and this looked just like. What we were told, so someone knew someone was doing reconnaissance inside their organization, knew about a bonus plan, made something look like that we, we've got to get education about what is real, what isn't.

We've got to have people aware of and trained. And we've got to have tools to eliminate everything that we can. And in fact, it goes beyond just phishing because we miss many of the phish attacks because we get numb. We get hit every day by so much spam. The stuff that used to come to our doors. We would see it and throw it right in the ma in the junk.

We need as CSOs, we need to be eliminating spam, not only doing the sandbox that I mentioned, but also help your users. Don't let them be confused. There's so much coming in, they're weary and then they accidentally click, or in this granting best of intentions to this poor lady. It looked like the meeting she had just been in.

And that's how Phish attacks are getting through. You're, you're, you're kinder than me. I'm gonna go, I'm, you are a lot kinder than me. I could just tell by your disposition. Uh, I would be a little more critical because he has a picture of the email and it's an obvious, it's a, it's a, it's obvious. I, I saw I this

software number. Yeah, would say, look, we should be well beyond clicking on PDFs from people we don't know at this point or any of those other things. Now, there's a little sophistication in the email, but not much. But then the sandbox technology and other things, there's a lot of things we can do around this.

So let, let's talk about the email a little bit. The title is Annual Bonus Report, PDF. Now, the person who was the person who met with HR just said, I met with hr, but it wasn't really meeting around her. Or anything to, to do with payroll. But this, you know, this is the kind of email that just blanketed one out.

In fact, we're gonna hear later, the same email is the one that took down St. Lawrence in New York. The same exact email. So what it's pre predating on is we, hackers will sit there and go, you know, we're, we're gonna send this out to a thousand people. I bet you at least five of them met with HR this week.

And we'll say, Hey, here's a note from hr. Here's information on your bonus. I'm looking at this thing, the links to the files are just Google Docs. That would be a red flag for starters, especially if you're not a Google Docs kind of place. If you're a Microsoft Place, you're probably not using a lot of Google Docs for internal emails, is my guess.

The, it has things like bonus report. Click here has PDF, click here, expand and preview. Click here. If the document, it has a document name and that's a link as well. And then it has, uh, if it doesn't open automatically, click here. Another link. And then it says, published by Google Drive report Abuse. And that's a little sophisticated, but I just had a conversation.

I just did a show with the people from Proofpoint and they were talking to me about how sophisticated they're getting. Now they're watching your social media and other things. And they are now coming to you saying, you know, an email, Hey, welcome back from vacation. We missed you while you were gone. Love to catch up with you on the project that's going on.

Before we do that, can you review the document and let me know if the budget is on track for the new tower? Right. So they did enough, they did enough research to know your organization's doing a new tower. They did enough research on you to know you, you were just on vacation. And that kind of email, which just looks like a run of the mill, I'm sending it to you saying, Hey, what's the status?

The initial, the software that would be identifying that kind of of email. Yeah. We need to be running phish campaigns in our organizations and the many people don't do favors to their organizations 'cause they make the phish, um, attack look very bad, very easy to spot and the things that are happening.

Are much more aligned with what's going on now inside our organizations because these bad actors know what happens. So they're making it look just like a regular review of your, your information as an employee or. Your annual review process. And so I, I think we have to just train, teach, look at the Earls.

You said that you spotted it right off. I looked at it and thought, why would you have clicked this? And oftentimes someone will ask me, will you look at this? And I just say, first hover on that link. Does that link look like it's going to hr? Does that look like it came from hr? And, and if we can do those kinds of things and train that and then say question, if you have any questions, forward that over to your cyber team.

Have them take a look at it for you. I think the other thing that needs to be done here is in, in this case, you point out and the CIO points out from Sky Lakes. This was the ransomware coming out of Russia. Lots of intelligence, lots of threat information around that. So in cyber, in our network teams, we've got to be harvesting, mining that information the bad actors do.

That's how they know what we're doing. We need to know what they're doing, which means we need to have a program to block. Known bad sites. Yeah, so going further down the chronology of what occurs now, we've got to go to a site, click on it, and we've got to receive information from a bad site. Those are things we need to be doing in our hygiene block, known bad sites.

Keep up on those kinds of things. The information comes from the hi sac. There's information available from hfs, from the FFBI. You can go out and subscribe. FBI will send you threat intelligence. Homeland Security will send, and if we keep up on that, we can block many of these things before they can get started.

Phishing is still the number one way they're getting in, isn't it? It is. And if you take a look, bill at 4 0 5 D out of Washington HHS in a partnership with the, the provider organizations put out a list, what are the top risks? Email is, uh, number one on their list, and it's these phish attacks. And having appropriate email protection has got to be on everyone's radar.

Carl, what's gonna replace email so that we, we don't have this problem? Or is it not gonna matter? Is there like a slack equivalent of this? I, I'm starting to see now on my phone bill almost as much spam coming to my phone, independent of the mail system coming to the channels that I use, my actual text messaging system.

And they're all the same kinds of things that have a link for me to collect, to go out and take a look at. And some of 'em are selling, some of 'em are advertising, some of 'em are just . Malicious it is going to spread to those other channels. It is spreading already. Yeah. And I, I don't wanna hear people, it might sound like I'm being critical of the person who clicked on the email, but at, at the end of the day, we're all at the level of our education, right?

So if, if that person hadn't gone through cyber education and, and those kinds of things, she would not have been aware that this exists, which is what you're talking about. You know, making people aware, doing those campaigns, internal campaigns. So that people can practice, they can learn those kind of things.

And, and I'm not saying that I'm not a, a target of phishing campaigns or could succumb to a phishing campaign. I'm sure I could, and I've talked to many CIOs who have, as part of those campaigns they do internally. Eventually they'll click on one of 'em and go, ah, I can't believe I did that. But it, it's hard moving fast throughout the day and.

Click on it as a matter of habit. Yeah, and you hit some important kinds of points that, and, and you're talking about detection. Can I, as a user detect this is anomalous, this is wrong. And detection. If you look at my responsibility as a CISO and organization's responsibility, we've got to educate those users, but we've also got to have

Detection systems in addition to the sandbox that I mentioned. I, I listened to the CIO who said they had an endpoint detection system. It wasn't fully deployed, it was in an initial phase of rollout, was what I understood. And I think that becomes so important if you look at this risk that you identified, and it's number one on HHS and OCR r's, uh, threat.

Risks for healthcare, we've got to have a detection system, one half of it. One part of it is me as a user. What can I do as a nurse, as a physician, as a caregiver, that I can spot this, but what can the system do to help me? And really having that endpoint that's server-based protection. All around the environment and you've got to measure, monitor, make sure you're not in the initial phases, you're not half baked with the cake that needs to be complete.

That is, that has got to be done and that's gonna help us. So the detection, even if. My, my caregiver missed that like she did. I needed to have that endpoint fully rolled out so that it could have protected everything that happened after it got through. So let me keep going through the, the identifying it, whatnot.

It was three days later, actually, October 30th, the FBI put out their warning that 400 health systems were gonna be targeted and three of 'em had already succumb to it. That was Sky Lakes. Of and health system that's not completely succumb. It's not, others didn't have infiltrations and and issues, but they may have had the right software in place to limit its impact or, or to detect it early and those kind things.

But those three health systems had significant events and as I mentioned earlier, St. Lawrence was the same exact email. Do we have a way that we're sharing that information like with other health systems so that they can be prepared? I think a, a couple of ways people can know that first join, be a part of hi sac.

That's where this information comes from. They run a threat intelligence center. That threat intelligence center is connected to a another center that's run by the federal government. All disinformation flows through those environments. We can also encourage all of our listeners. Go to FBI or to Homeland Security webpages, sign up, ask for their threat intelligence.

They will send it to you. In fact, it, it, it, uh, flooded my mailbox, but it was important. Every day, I, I get 50, 60 pieces of email about what is happening coming from FBI from Homeland Security, this Isec feed that I talk about. It will tell you all this information that the press is sending out. But if you're relying on Google, you're a day late, maybe already hit as in the case you mentioned October 26th.

They're hit four days later, information comes out. It's for them. A little bit too late. A little too late. The I, I knew some, I talked to some CIOs who were on that call with the FBII think it was post actually these, these systems being hacked. And one of the complaints they had was the FBI was kept, uh, the party line, which is we can't comment on a investigation, we can't comment on investigation.

Now it close.

But there, there, is there a way to get that kind of information because you, you really want it on the same day. This is the email that's floating around is, are these groups that you're talking about that current? I think that the recommendations they give are very consistent. They're very, I've sat through, I, I have secret clearance with the FBI have sat through regular briefings over the past many years.

They are consistent. They're predictable. They're going to tell you, watch, get the signature. Rauch is not new. What happened? October 28 is not new rout. We have been watching monitoring in organizations for three or four years. If you already know this signature. Update your files. Update your perimeters.

Update your endpoints to make sure of the most current. I don't have to wait for Sky Lakes to send me the information. Say, here's the specific signature. There are other recommendations that always come out. Bill, they always, and you'll see at the end of this video, they give you their seven or eight.

This was critical. This was high. These were things we needed to do. Always in that list of things, what we've already begun to talk about. In addition, certainly multifactor and net network segmentation, which we'll, I know we'll come to in our conversation, but those are critical kinds of things. And the fish, uh, education that we've talked about, the sandbox technology, they're always consistent recommendations.

And if you look at Sky Lakes and say, what did they do? Those are the things. So what should I be learning from this? Don't wait for the threat intelligence to come from, from Homeland Security or iec. Do those things now. Get your MFA in place. Get your segmentation in place. Email phish education going on.

Get your detection built up. Block known bad sites. Do that right now. Don't wait for another strain of RA from the Russians. All right, so I'm through this identifying it. Is our first section. And the last thing I would say is something you already mentioned, COVID was trending up in the community. So people are preoccupied and they already had a network ticket that the network was running slow.

So there's a bias towards that being the problem initially. And uh, as you said, they had endpoint security that was not fully deployed and not fully configured. So that's where we were at. I'm gonna guess that ask question very common. How common is that situation across healthcare where. Security software is not fully deployed or fully configured.

Very common. When I said 80%, really 80% of the healthcare in America is what we consider small, and that's what we're looking at here and, and then there's the medium organizations, and then there's those very large. I think if you looked at the large. They are not going to fall into this, but the large represents three to 5%, 10% of our, all of our healthcare infrastructure.

So this is very common in 80% of what's going on in healthcare. Alright, so let's start working through it. They identify that it's a, that it's a ransomware attack. I keep wanting to say malware. Is it a type of malware or is it it it is a type of. It, it is a type of malware and it is financially motivated with different types of instruments that are delivered.

There may be a financial, there may be an ip, there may be, they may be looking to steal data and collect the data and sell it. In this case, it was all about money and these, it, it was quite clear if you read the, if you read the stories would've happened recently. These folks just want money. That's, and they don't care that you're a healthcare.

They're just about getting money. Yeah, agree. So initially they're working with their cyber insurance company who says, look, you need to bring in some third parties. They bring in, uh, Cisco Talos, they bring in Vu, KVU. The Talos was, and, and they split 'em up. So Talos determine the root because you need to determine the root other, recover the offline.

Next, or he has these five things that they did that in, and I'm not sure they're exactly in order, but We'll, we'll just go with it. So the first steps were segment the network, shut off all systems. I would think at this point, when you say segment the network, I unplug all the routers, hub switches and everything.

I'm not sure I'm configuring anything. Let's just shut it down, make sure it can't propagate anymore. And I, I think that's what communicating here happening. It, it's too late. You have to do everything you can to cut off access. Is that the right first step? That that is correct. And if you have a good plan, you don't have to do that.

If you're segmented, then you can say, ah, this is only in segment A or B or C. Shut off that segment. But again, for most organizations, and this is a significant concern to HHS, they recognize that healthcare runs flat networks. Everyone can go anywhere. And so that why they described this is that's what they had to do.

They did not have segmentation, no way to detect to, to separate what's going on. But a good strategy and a plan would include on the backend, I know where my data is, I know where the virus is, the malware. Can I just segment that one piece? Some organizations that I have talked to have been able to successfully do that, which is why if we see that 400 were hit, it's why probably we didn't see all of those stories like Skylight somewhere.

They were able to catch it segment. Do some cyber hygiene to prevent the rapid spread of this virus through the organization. If you don't have that is what's gonna occur. And I think one other piece you, you talk about how he talked, he as the CIO was talking about this, he talked about different processes.

I would stress to our listeners, three processes you must understand. You must know the difference when you get hit detection, response and recovery. They are not the same. They're not even close to the same. What causes this rapid reinfection? Think of Covid. We think we've got a handle on it, and then it comes back at a resurgence.

The problem that occurs is people go from detection before they've completely detected it, and they start their response and then they recover systems. Then those systems. Get infected and we have a resurgence. Understand the difference between detection, response and recovery. Don't move until the difference until you've completely identified in, in a good shot.

You should be measuring how long to detect and know that you have. The threat vector completely understood how long to respond, how long to recover. If you measured those three things, you'll know how you're improving in your capability and maturity, and that's what we want to be doing. If we're building healthy, good, resilient infrastructure, like President Biden is stressing, start measuring.

How long to detect and know that we've got that right, how long to respond, which means I'm down, I'm hurt. How long does it take me to fix that and then recover, get the EMR back up, get the O 365, the imaging back up. It's interesting because you, as you would imagine, once you shut off the network and all the systems, their communication was really shot.

They had cell phones and they had Cisco WebEx teams. So that was the two forms of communication that they had. So they said, we gotta get our nurses and, and, and clinicians communicating again. So they had, they brought that back up as quickly as they could and it was immediately infected again. So that's that point you're making right there.

Yeah, exactly. Bill. When I read, when I listened to that, I thought, oh gosh, he didn't get, they didn't get detection complete. And it, it's a little bit like what we have just witnessed. It's very analogous to what's just been going on for us in Covid. We think we understand it. We start down a path of opening up, and then we have a resurgence and we've got to understand, do we have this contained?

We know exactly where it came from and we have containment. Now you can move to this respond and recovery and each are different phases. All I wanna camp on this one for a minute. Next step was be he contacts the Asante health system, CIO. Alright, so keep this in mind that at noon the previous day, they were infected.

He contacts the Asante c io at 7:27 AM The question that's not really answered, he said we cut off connectivity. I'd wanna know when they cut off connectivity, because my guess is. Given the sophist of their response, whatnot. They probably didn't cut off connectivity until the CI for As said, you, you've been infected, we're cutting off connectivity.

That's my guess is what happened. And I guess my question to you is, if you're the host EHR, system CIO, is it appropriate to cut off access? You're cutting off access to their EHR. Essentially with every organization that, I don't wanna mention names, of organizations because they become targets, but the playbook should call for both sides.

If I have a partnership with Epic, with Cerner, with afa. I should have a process that says We cut, you. Cut. We both cut. We both have, and we know exactly how to do that and we know the consequences of it. In many instances, I have seen organizations proactively cut because they see something happening to a third party partner and they have a playbook that tells them, cut the network and, and what happens is good organizations will be scanning, they'll see something happening.

They will sever, both organizations have to have that cyber hygiene in place so that they isolate and segment their systems and networks and, and yes, to your question. That that who's their EMR provider. Their third party partner should have processed, they should have been detecting and probably they were scared stiff when they called back the next day, they were probably asking this question.

I. Do you know the source of the threat? Can you tell us that? And if you can't, we're all segmented. We're all separating. That's what you must do until they can identify, and literally it would be best practice in your playbook. Never reestablish that link until the CSO comes back and tells you, I know if they tell you, oh, we're bringing back up systems, but they can't answer question one.

You don't want to reconnect, you will be part of their poor cyber hygiene. Yeah, that's, you went exactly where I'm going next, which is la. Later on in the presentation, Asante comes back to Sky Lakes and says, look, and it was disconnected At that point, it's seven 30 or when, or earlier than that. It was disconnected.

I came back and said, look, in order to reconnect, here's what we want. We want the, uh, steps that you're gonna take. We want the build documentation. 'cause essentially you're relaunching, it's almost 30 days of downtime. You're relaunching the EHR and all the connectivity and all the systems that you're gonna connect in.

They made them sign, didn't make them sign. They asked them to sign a memorandum of understanding, andum of understanding four.

Clean bill of health, so third party to come in and audit them, which makes sense because they probably didn't have that sophisticated audit capability internally. Annual risk assessments with pen testing, so penetration testing, that's interesting. Annual may not be enough, but it might be what they can afford kind of thing.

Incident notification, a more timely incident notification. My guess is, as was. The better part of 12 to potentially 19 hours. So we want incident notification and more timely response. And then they have D is interesting, NIST V two framework, security, posture, and culture. So they're asking them to implement one of the security frameworks, which happens to be nist Version two is is are there other things you would add to the memorandum of understanding or is this a good start?

That is a great start and, and I think as I looked through that, I thought they've done a good job. They brought in a good, they did some great things. They brought in good partners, they separated processes. I think, I think in, in a list of things, I would want to know what is the signature, what's the strain of this virus?

Is endpoint protection complete? Did you do some scans? Is there no ra uq anywhere else in the environment? In an offline, in a segmented, once you've completely eliminated, then that piece of the process is good. Now you can move on all. Well, that's, so we on that for a while, a for system. Some community connect partners or other EHR partners that you're serving them.

If, if you don't have this kind of language in your contracts, would, would you go back today and make sure that language is cleaned up? Definitely, and, and I think I'd start first on my processes internally, and I, I would, the processes internally have to involve. Your ability and knowledge of how to segment if something occurs.

Make sure you have language around our ability to sever your ability, your responsibility to protect me by severing. And probably in your process, you need to make sure you have phone numbers. I've seen many people at seven 30 in the morning, like we heard the CIO saying. Where is the phone number for the CIO of Asante?

I can't remember where that was. It's not on my speed dial my computer's down. You need to have a playbook. Where, where is Asante? Where is Epic? Where is Cerner? Where is my key providers? And, and we've got to be able to quickly call them, notify we're infected, or expect the notification process from them.

All right. Let take you through the rest of what they. That's, uh, pretty obvious in standard procedure in, in something like this, they had to go to huddles. Communication, obviously was very, uh, stilted. So not only was there a command center around this, but command center around the caring for the patients and whatnot, making sure that all those processes were in place.

They had to start prioritizing servers and bringing 'em back up. He talks about the fact that they had a lot of different lists of what was the most important thing to bring back up and whatnot. They finally determined that the, uh, place to start was cancer. But before I get there, uh, I wanna talk about some other things.

The, what they did is they created a, an environment where they had what they called the dirty vlan. So that's the infected VA, they had the staging VA where they could bring systems back up, see how they acted for a period of time, I would imagine. And if they remain clean and whatnot, they.

So that they could start to bring systems up and online. I assume that's pretty much best practice there. Yeah. In spite of all the things they did bad, there are many things they did well, and that's a pattern I would recommend our listeners, listen to this, review it, put it into your processes and playbooks.

If you get to this point and cross our fingers, we, we don't get there. That is best practice right there and, and I think some things you can do in advance. They talk about they're doing this and at the same time they're prioritizing . Service A, service B, application Y and Z. That should be a process you go through now in advance.

It's called the criticality matrix. You build a matrix and say, I need, what do we all need? The EMR, that's number one. We all know that. And so put that as number one. But before you finish, notice what services are required to have the EMR up. You have layer one, physical, layer two, network. Layer three is application.

What's at that lowest layer that the EMR. Must have. You can't bring the EMR up until your hardware is up. Your network is up, your database is up. So a criticality matrix tells me what are the critical applications and what. Do they depend on, in order to be up, interfaces need to be up. I need to have hardware, I need to have network, build the criticality matrix now so that in the middle of the crisis, you know exactly.

Number one is, and here's the services required, and number two is . Here's services required and that will help you in this process? Absolutely. We brought a guy in from financial services who helped us with that. He called it something else. But essentially what we did is we identified our, it was criticality.

We identified tier one, two, and three apps and tier, tier one, two, and three systems, and he was adamant that we keep the documentation. Goes, because when you're flying around, you don't want to pull out the binder that's this thick and people go, Ugh, you need to pull out the piece of paper that says, here are the, here's the list of applications, or two pieces of paper that says, here's the list of applications in the order.

And like you said, he came up with multiple lists right here on the fly. And that's, that's something we could do ahead of time. Very, uh, not easily it.

You don't realize how interconnect it's healthcare. Everything's interconnected. It's crazy. A real quick recommendation for our listeners. As you look at this exercise, you can make it simple. What is life critical? It must be up in three hours, two hours. I. Put a bucket around that. What could I do without for 12 hours, 24 hours?

What could I do without for a week and what could I go 30 days? That's if you deal with a, a big organization that does . Business continuity and disaster recovery. They'll ask you to put things into buckets like that. So as you build your criticality matrix, just start to think, what's life critical? What do I have to have?

I've got patients in hospitals. We're gonna go to divert mode. Patients are gonna be at in life critical situations. What is that category? What's next day? What's a couple of days a week, and what's 30 days out? The, this is one of the more interesting slides. So they had to rebuild 2,500 PCs, essentially any PC that was connected to the network.

At the time of the infection, they decided they were going to rebuild and they had to replace about 680 Legacy PCs as well in the process. I don't know about you, but I went through a significant process while at St. Joe's to get us down to six images. It was Herculean effort to get us to six images.

You're smiling because it, it's not, it was hard to get to that 2,500 PCs in this kind of organization. How many images do you think they had at Intermountain? We had 13 and it was. It was like pulling teeth from a bear to get to 13. It was very hard. And if you don't have that, what it means, if they don't have and don't know how many they have, it means they probably have many images that are customized for everyone.

And so if you can get, that's another critical kind of a thing, bill, that as we come down to, what should you do in a future model, you need to have workstation classification, which means . Tell me what this is, this a blue, a red, a green? What's the classification of this workstation? And, and that rebuild that they did because they didn't have scanning and ability to do, it's probably best practice if you had internal scanners.

Ability to do, could have prevented some of that, but. Those were things probably just at the last minute flying, like they were, they just said best thing to do, start clean. Yep. And that probably makes sense. Uh, third party systems. Pyxis Diagnostic Imaging, uh, cts, MRIs. X-rays and others smart pumps. So you have all these devices online, which, and even monitors, they talk about the fact that they disconnected it all from the network.

So the monitors still work, they collect all the information locally, but at some point you're gonna bring them back onto the network and you have to, uh, verify and certify those devices. What's best practices around those devices? Or does it depend on the device? Yeah, you, you hit a kind of a bit of a sore spot for me, in the middle of fighting the fire, which is, and I just was so sensitive as I watched this.

I thought, oh man, they were in the middle of the biggest battle to fight. But there's an obligation or responsibility to be contacting your third party partners. Probably you need a partnership with someone in communications because your team's not gonna have time while they're trying to scan and determine, and detect and respond and recover.

So . You need a process that says, contact the partners so that they can protect themselves. Give 'em the signature, let them know what it looked like, what it is, and make sure that they're protected. And yeah, that, as I heard that, I, that would've been a huge, it, it is a huge help, but in the middle of the fight of the fire that they're dealing with, it isn't something they take a look at.

Yeah, they, I'm just gonna keep moving through this. If people want deeper answers, they can contact you. You bet. They go on and say they redid their, their password policy, and I think this was dictated by the vendors to be honest with you. They came in and said, what's your password policy? No, that's not good enough.

And they, uh, made harden that across the board. They bring up the cancer system. This is a probably, I don't know, maybe 20 days in 25 to about 20 days in to bring up the cancer center. But he also talks about these side roads, like they didn't expect there to be a snowstorm and, and the snowstorms coming and they said, Hey, wait, our heating systems and some of these things are connected up to the computers.

We don't know how to turn this stuff on without this. And, and that's a good point here. He, he also talks about they had clinicians that had only worked in the EHR world. They had not worked in the paper world, so they weren't, they weren't used to paper processes. They weren't, they don't know how to turn on these systems that are all digitally controlled.

And those kind of things are gonna come up as your, as you're laser focused on the clinical systems. Those kind of, of sidetracked things are gonna come up and they get bubbled up through the command center. I, I think it's a great lesson learned for our listeners. Bill, as you go through the processes of thinking ahead, always in IT and in cybersecurity, we will think of the technology kinds of things, but the business processes have to be thought through, which means if you're putting together a good business continuity plan, you need, you need home care, you need hospital.

Was you need pharmacy sitting in the room because for them to survive the CMO, the CNO for them to survive an outage, . They have to know what are their processes that are not it processes, and if we learned anything from what has just occurred in our gas pipeline shutdown, in our meat packaging shutdown that just occurred again, some Russian, uh, attackers.

What we need to learn and understand is the business process, not . The IT process. It is part of enabling the business. Look at the whole business process and say, what would we do if and what's going to occur in the absence of that kind of a plan. The CISO's gonna be under immense pressure, just restore everything and he or she's gonna be fighting to detect, and the business is saying, I have to have this up.

I can't do ophthalmology without, therefore you must just bring me up first. If you have a thought out plan where you say, what is your business continuity? What did it look like before we had technology? What can you do in a non ? Technology based process, have those plans ready, have them built out. It's not an IT thing, it, it usually is led in good, sophisticated organizations, either by the risk officer or by the chief medical officer.

But IT and cyber have to sit there and be a part of that conversation. Okay, Carl, so here, so we're gonna transition to the last section, which is, what did they learn? And they have on the first slide, big takeaways. They have, you have to have the tools of the old ways to make it work. In other words, paper, toner, paper processes, those kind of things.

You have to, because they, in his words, they had good processes to be down for a day. They had bad processes to be down for 30 days.

We had a good process to be down for 30 days at St. Joe's. That would've been to have the data center down to have the primary clinical systems down for 30 days. I'm not sure many health systems do have that kind of thing in place. Lemme go through the other. So there are providers who, who only knew the EMR, not paper, which is what we talked about.

And they said there's a massive backlog of PA paper post outage. So there's restore the data. They had to plan on the data that was generated as. Re and all that stuff. So those were three of his, uh, key findings. He talked about key findings around prevent prevention. He said you need to have good backups.

Security operations center, seven by 24, 3 6 5. Education is first line of defense. Playbook for extended outages and plan for rapid deployment of systems. And those are pretty good and strategic recommendations and priority has implement multifactor authentication is high. Continuous monitoring is. Was centralized Log repository was incident response team on retainer and low was incident response and security awareness program.

So those were the major findings I'm to go through real quick, but as usually the case you. We have so much to talk about. What, what other learnings or, or what do you take from, from takeaway from this incident? Some things I'd prioritize a little differently. I would tell people, spend and invest time on your incident detection, response management monitoring that is going to.

You cannot prevent this from happening, and the CIO concludes that what you can do is reduce the impact. If you can detect and respond and recover quickly, spend time, and invest in network segmentation. They didn't talk about patch and patch management. What makes a system susceptible is they forgot to get their vaccine.

And, and this is not, uh, political and it's not moral. Just get the vaccine and vaccine your systems. It's called patch management. You need to be doing that. I think that's so critical. Building the BCDR plan. Identity wasn't talked about heavily here, but I would tell you I didn't invest time, spend time understanding the identity.

Where should this identity be used? Is it okay for, for bills in the case, this lady Susie, or whatever her name, . Can Suzy's identity be used in Russia? If we had identity management, we would've said, that's anomalous. Why is Suzy's identity being used in Russia? Catch that even if you missed everything else, get an identity management strategy, the multifactor strategy.

Those are things I would encourage our users to take a look at. I would only add to that buying the, buying the right security software is not enough. You have to get it installed and configured correctly. And by the way, I would check those configurations on a pretty regular basis because you just have to, things are changing and whatnot.

I, I do wanna touch on one comment I hear over and over again, and it just sits the wrong way with me. And that is, it's not if, but when everybody says it's not if we get hacked, but when. And when I hear that there's something that's defeatist about it and there's something that says, look, I'll agree with this.

It's not if you're gonna be attacked, but when you're gonna be attacked. And it's also not if you're gonna be infiltrated, but when you're gonna be infiltrated. But it shouldn't be. It's not if they're gonna shut down the EHR, but when they're gonna shut down the EHR. There's different levels of where that statement is true.

Yeah. You're gonna be attacked. Yes. They're gonna get in. Yes. Get to or systems. It should be so hard for them to get to the EHR and shut it down the PAC system and completely annihilate that. They, they completely annihilated their PAC system. Bring up a different PAC system. I'm curious when you hear that, if it, it's not if, but when it doesn't sit well with me, but I'm, I'm curious what your thoughts are on it.

I like what you're saying and I tell you that's why Bill, when they said incident response would be a low, I put it at a high or a critical. Because if you can detect, respond, and recover quickly, the damage is minimal. Organizations, uh. An organization I used to work at would see seven to 10 riot like attacks per week.

And, and the difference was if you can detect that in minutes, respond and recover in minutes, they're not going to affect, they're not going to take down, which is what you're saying. Don't say, uh, it'll never happen. That's a fatalist kind of a view. It's going to happen, but we can minimize the effects if we have capabilities in our systems.

To protect, detect, respond, recover, get that right and you can sleep well at night and you can say it will happen seven times a week, eight times, 10 times. We can root that out before it has any impact. We took the cyber hygiene and we did that, that necessary step. Carl, I I want to thank you for your time.

I know this. And I really appreciate you going through this, taking the time to watch the video and then going through it with me. I really appreciate it, bill. Thanks you for educating. This needs to be out there. We need to invest more money. That's one of our issues as well for us to improve. President Biden is pushing.

We've got to invest as healthcare and you educating is so helpful. Thank you Bill. Appreciate it. Take care. Thank you. Wow. What a great conversation. I really appreciate Carl coming on the show. If you want more on this topic, specifically the Sky Lakes ransomware event, go ahead and hit Today. In Health, it's the daily show that I do.

I did three episodes on this last week and we go into more detail, actually, not we, I go into more detail. It's a show where I just take one news story and break it apart. So three, three days, Tuesday, Wednesday, Thursday of last week. And, uh, I, I, I think it's one of the most important topics that's facing healthcare right now.

Alright, that's all for today. If you know someone that might benefit from our channel, please forward them a note. Let them know that they can subscribe on our website this week, health.com, or wherever you listen to podcasts. Apple, Google Overcast, Spotify, Stitcher, you get the picture. We are everywhere.

We want to thank our channel sponsors who are investing in our mission to develop the next generation of health leaders. They are VMware, Hillrom, Starbridge Advisors, McAfee and Aruba Networks. Thanks for listening. That's all for now.