This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

Newsday: Preparing for Cloud Outages and Phishing Attacks with Charles Knight

If your team is discussing public cloud options for your EHR, EHC consulting should be on your radar. With decades of epic expertise and proven cloud migration experience, they help health systems confidently plan, execute, and support their move to public cloud. Ask about one of their free guides, like where to host Epic or their Hyperdrive delivery decision matrix at this week.

health.com/ehc.

I'm Bill Russell, creator of this week Health, where our mission is to transform healthcare one connection at a time. Welcome to Newsday, breaking Down the Health it headlines that matter most. Let's jump into the news.

Bill: all right, it's Newsday, and today we are joined by Charles Knight, founder and managing partner at EHC Consulting and the incomparable direction to Ford. Sorry I couldn't come up with an adjective for you. Charles, welcome to the show.

Charles: Thanks. Thank you. It's good to be here.

There's a lot going on in the world. We're we're gonna talk infrastructure a little bit today. We'll talk about what's going on. We will probably hit on cybersecurity 'cause that's Drexel's, center of gravity ends up being cybersecurity as well. I'll tell you the it, it's, it's been interesting to look at the news lately and I forget who I was talking to.

I was interviewing somebody and we were talking about. Resiliency in the world because of the fact that it's not just one cloud that's going down. You had I think CloudFlare went down. You had you had outages in AWS, you had outages, you, Azure, I mean, there. They're they're not perfect.

I don't know if our expectation of the cloud was, it was going to be perfect, but it's not perfect. And I'm curious what you guys are hearing out there and how people are responding to the lack of perfection in terms of the, the cloud infrastructure that, that we're residing on these days in healthcare.

But they do happen. I think what we're also seeing here a little bit is folks are starting to take a look at if I have an issue with one cloud, we need to start having more of a hybrid cloud. Environment. Right. So they've got some headway on, I have got a DR environment in a different cloud, so if one goes down, I can pivot to the other.

Are,

Bill: are they generally the same like cloud backing up in AWS or are you saying AWS backed up by Azure backed up by something else.

Charles: Yeah, it's a good qualifying question. So what we're seeing is, you know, obviously there's resiliency built into the clouds themselves and you can deploy into different regions and that gives you kind of, you know, base resiliency if one region goes down.

skilling for your staff make, you know they're comfortable with Microsoft. Are they comfortable with an Amazon cloud? And kind of do they understand even just the terminology between the two clouds because there are differences there. So that's kinda what we've seen here recently is some folks kind of pivoting a little bit to do I need to look at alternative options when it comes to hybrid cloud and do I wanna run a primary environment one and then a secondary and a different cloud?

Bill: But there are layers, right? So I mean, if you have a cloud flare issue, it almost doesn't matter that you have two clouds if you have a DNS issue. Correct. It doesn't matter if there's two clouds. I mean, so I mean, we have to think about resiliency on a lot of different scales, don't we?

Charles: Yeah, that's correct. So like when you're looking at resiliency, I think a lot of folks, and I spent a lot of time talking to people, is like, you can plan around what you're planning around. And what that means is like if you're expecting everything to go down, you can plan around that. But like to your point around CloudFlare, if you're using that front of your DNS, have you planned for resiliency around just that particular thing?

Like, where do we go next and how do we keep adding additional resiliency at those different levels? I think the challenge that folks have is when they look at like a disaster recovery or resiliency, again, they're thinking everything's completely down. I've got a crater in the ground, what do I do?

And reality is like even with the recent outages, that's, that was not the case for organizations, at least for what we publicly know that were impacted by any of those three. It was, I have sub components of those systems are down and what does that look like and how do I respond to that Operationally,

Drex: sometimes that can almost be a more dangerous situation, right?

Because some things are down, some things are up, you're not exactly sure what's working in what unit or in what building in your organization. And so I don't know, is the tendency to just say, we're gonna take everything down until we figure this out, because we might actually, you know, people start to rely on the things that are up and they're even sketchy.

Do I therefore take other potential tier one systems down to bring that one back up online? And so kind of like the due diligence of having a known list of applications, their impact to the business and clinical side of the house, and what that means, I think is critical. It's also like who is gonna actually make those decisions?

So. One of the things that's challenging is when you get in these situations, people start panicking a little bit if they don't have the documentation in place, and it's also, do I even have the knowledge one to make the decision? Do I know who should make the decision? Am I comfortable making that decision?

Right? So it's like putting all those components in place to where I actually am comfortable that I understand the systems that are in place, their impact to the system on if there is an issue. And then what's the criticality of those? Who's making the decisions? What the operational impact looks like?

There's all these like nuances that go into place and it's easy to go down a lot of rabbit holes in these conversations. But I think there's value in actually having the conversation of what's important, who's making the decisions. That way you have a level comfortability going into these situations.

Well, well, does

Drex: that come out in the form of like a playbook or something? Are these tabletop exercises that you run, is this just sort of, well, so ongoing conversation, you're figuring out what's prioritized and what

Bill: I sort of wanna ask the same question because you know, you talk about documentation and whatnot mm-hmm.

In my history, and I'm old now. Look at this gray hair. I mean, I'm old in my history, I've come across one organization that had good documentation around. Yeah. Continuity and resiliency. And that was MasterCard. And you know, and I worked in banks beer banks and bombs, right. Anheuser-Busch you know, banking and McDon Douglas and others like that.

Where are we at today? I mean, like, if I were to walk into a health system and say, Hey, you're having issues. Show me the docs. Let's let's figure this out. Where are we at?

Charles: I think you'll find not much has changed for a lot of organizations. And it's also, you know, an opportunity when we're working with folks to move to the cloud, it gives you that ability like, Hey, I can automate the failover of the technical components.

But I think what we're seeing is a lot of folks don't have the operational rigor in place. So to your point, like if you walk in and do have a playbook, a lot of folks don't have a playbook. They, at best, they have tiering of applications for when it comes to Dr. So that way they can prioritize which applications they're bringing up.

Where there is a potential disconnect between leadership, believing they have a DR strategy, but reality is you have a failover you can do, but can you actually bring that system back up operationally? And get that to your clinical end users in an efficient manner.

Drex: And they're prerequisites. They're prerequisites in some of this stuff too, right?

Like so even though they're all tier one, some of those systems have to come up before others, correct. And you see people bring up a system and then say, have the, ah, we're gonna have to take that back down again. You know, like there's a lot of wasted time in that. In the heat of the moment decision making.

Charles: Yeah, correct. And even, going back, we touched on DNS earlier, like, basic things like DNS records, right? If they're not propagating quick enough or like you forget to update one and now you're, you know, spending an hour troubleshooting why something's not connecting to a third party system.

And then going back to bill's question around playbooks, like if that documentation's not there, you don't know how to execute on that. Like, that's where you start to fumble and that's where you see organizations struggle to come up in a quick manner, in an efficient manner, that act they can actually give clinical and business users access back.

It's not in my data center. I put it in the cloud. It should be fine. It should be stable. They've got resilience and everything built in and we just keep being. We just keep being fooled. I mean, not fooled, but it just, we keep having these revelations of like, oh no, AWS went down and, hospitals across the country went down.

So how does that, how do we find all of those choke points? All of those places where we're all really dependent on the same thing. And I mean, some of them are blindingly obvious, a lot of them aren't. How do you guys look at that?

Charles: Going back to your, actually you referenced a little bit or go about like third party systems integration there and like what that flow looks like.

Through some of the work we've done with folks like actually mapping out the connection points between their systems and kind of where they're going, both internal and external. You can vet out some of those components. I think one of the things that I like to remind folks, it doesn't matter if you know you're on-prem or you're in cloud, you're in a hybrid situation.

For what it's worth, there are like. Modern systems today, if you deploy them as designed because it's the way they're designed, there are still single points of failure. And so, knowing what those are and knowing the criticality of monitoring those, I think is important. And so we do emphasize that a lot with our customers, especially when it comes specifically to you know, like some of the network connectivity between their environments on-prem and into the cloud.

But also just within the actual like application ecosystem. There are components where you can't have, you know. Dual VMs up and running that are immediately able to fail over. There are components that are like that. So kind doing that data mapping, looking at the data flow and actually identifying those is really important.

We're talking to board members and CEOs. It's like, how can you know? And you know, what advice, where do we start on that with that group to say, look, the next time your IT organization's in the room, here's some questions you need to ask them. Here's some things you need to ask for potentially some testing you want to do to make sure that.

What they're saying is accurate. Where would you guys start? Drex, I'm starting with you. We keep asking him questions. I'm curious where you would start with this.

Drex: If you kind of go back to the very, very beginning, I think there's a really interesting, you know, movement that's happening right now where a lot of organizations are putting cybersecurity and risk specialists onto the board.

Bill: board, I've seen that board.

Drex: I have been, I've presented to that exact board. And so I think having, you know, a true technology leader from another company on the board having.

Cybersecurity and risk leader from another company on the board is a really good place to start because they start to be able to ask those kinds of questions and they can really tell when somebody is kind of turning up the BS meter. But those are the kind of questions. When's the last time, you know, tell me about the, our resilience plan.

Tell me about resilience around the electronic health record. I know we have this connection with Epic. Our EHR runs. In Epic, in the Epic Cloud, do, what's our backup plan from that? What happens if Epic goes down or AWS or whatever. And I think it's really interesting, like if you can get into that conversation even about things like CloudFlare, this is a tool that we've, you know, acquired to.

Fortunately, it wasn't a security incident, but it still had this major impact across the organization. So it's sort of like. Knowing to ask all those kinds of questions and then being able to sort of sort through it.

Bill: I mean, you know what was interesting? I was talking to Chuck Christian yesterday Franciscan.

Mm-hmm. And he was talking, we were talking about their cloud strategy and he was saying, oh yeah. We you know, we completed a complete failover of our EHR to our other data center and ran from there for about 30 days and then we did a complete failback, and I'm like. Okay. How many health systems do I know that have done that?

Drex: Oh, I know tho those kinds of questions are but,

Charles: Yeah. With Drexel's comment too it's asking that like how often are we failing over? What was the fail over time like?

I'm actually familiar with Francisco's environment, so I know the answer to that. I'm not gonna say it, but like, how quickly can I get that environment up and running and actually back it? I think with, you know, some of the organizations we've seen, they do the bare minimum to keep kind of their insurance requirements or the board's compliance requirements in mind.

But actually take to that next level and like pushing them on, okay, oh, you, you did a failover, that's great. How quickly did you do it? Were we able to validate basic workflows? You know, can I do e-prescribing? You know, something that's fundamentally. Like a requirement, can you do those things? And I know there's been some folks who've had success with that and the ability to automate some of the failover of the technical components.

It's been great. But again, like going back to a previous point around operationalizing it, do you have the components to get people in quickly enough? And, you know, direct, you touched on the security aspect of it. If you have a ransomware attack, that's, even worse in my opinion.

Bill: Maybe it's 'cause I'm not talking about security as much, trucks, maybe you're taking all the unh hacked stories and whatnot.

But I, I noticed that one of the stories you sent over was phishing operation, attacking at least 20 healthcare organizations dis disrupted. And I'm curious, you know, it's still our biggest vulnerability is humans. Hopefully we'll replace all of us with AI and we won't have this problem anymore, that we'll have a different set of problems.

But, as long as we have humans where that's still the thing. And I know that phishing has been. The number one way that people get into organizations from a tax standpoint. Have we made progress here or is this still the, is this still the number one way that people are getting the health systems?

Drex: I think that you know, phishing attacks are relentless. I think the kind of what we call spear phishing, like super hyper customized emails to individuals.

Drex: trip, blah, blah, blah. I was reading something the other day that you know, somebody had figured out how to spoof a, you know, their chief operating officer's email.

And the title then of the email was The, eh r's down. Can you know, can you click over here, give me an update, or something like that. Like they know that our brain is kind of wired to people that we know or we love. When the message comes from them, all of our like, oh, I'm gonna be careful and I'm not gonna click links.

I don't know. All that stuff just gets trashed and you immediately do that. So that sense of urgency turns out to be really important. The other thing I talked about on the two minute drill this morning. Was this idea of, or yesterday, I guess this idea of there are a lot of bad guys now who are reaching out to individuals inside of organizations and recruiting them to give up their username, password, MFA capabilities so that they can get into the network.

The phishing operations that attack 20 healthcare organizations have been disrupted. The, I love those. I mean, I love it when a bad guy gets their, you know, their just dues, but. The problem is sometimes when we take this stuff down, like in a few hours, it's back up on other servers somewhere else.

Like they take it out, but it doesn't, it's just one battle in a long grinding war, I think.

Bill: I go to a bad actor, they give a hundred grand. I give them my credentials, my MFA and that kinda stuff. I mean, what's the ramifications for me? I know I could, I almost in positive I lose my job. Right? That, that just has to be a.

Drex: Oh yeah.

Bill: A base.

I'm imagining that law enforcement's gonna be at your door and you're gonna be in handcuffs pretty quickly. You're helping to perpetrate a crime on an organization. Yeah. Yeah. You're going to jail and then you're going to trial. So. Yeah, don't do that. I'm not saying do, I'm saying it's a terrible idea and I know it's life changing money, but life changing money when you have to live someplace that doesn't have extradition agreements with the US is not, that's not a great, don't do that.

It's a bad plan.

Bill: Don't do that. Oh, man. Um, You know, outside of the normal things we have software for this, we have practices to simulate this internally and whatnot. Are we seeing any any new kinds of methods to to, I don't know, protect us from Phish?

Drex: the reason they're fishing is they want the idea and once they've got the idea, they go into the network and they do all these bad things.

They go to this app and open it. Why are they trying to do this thing now? You can either quarantine that identity, or you can quarantine that advice, that device, and you can start to do an investigation. The challenge is like, it's a lot of work and there are alerts like this that may happen all the time and there may be a very legitimate reason that the person's going someplace they wouldn't normally go, but you gotta hold up the show and then go find out.

So, the tools are there, they're just. You know, there's a lot of, there's work involved in it.

Bill: Charles, I'll give you the last word on this.

You know, I won't name names, but you know, there was a health system that had spoken about how they had picked up on somebody who's trying to social engineer MFA reset for a admin user, right? So like. You can put all these tools in place, but if you don't have the appropriate training and support processes in place that you're actually validating these things are happening, like you're still at risk.

And that's, I think the main thing is at the end of the day the person sitting on the side of the computer or answering that phone call is your biggest risk period.

Bill: It is. Last question for all of us, a quick one to close. What are you looking forward to eating the most tomorrow? 'cause we're recording this on Wednesday, the day before Thanksgiving.

Charles, we'll start with you. What are you looking forward to the most?

Charles: I'm very basic. I'm a stuffing and mashed potatoes guy, so like that's my favorite thing on Thanksgiving. You know, if maybe if somebody was fry a Turkey tomorrow, I might get into that. Fried turkeys are good, but fried Turkey, definitely something.

And mashed potatoes.

Bill: Yeah. I love the videos on YouTube of fried turkeys where people drop them in and they become an explosives

I was so paranoid. Yeah.

Drex: The other thing too is like, do it 300 yards from the house. Do it out in,

Bill: they like do it in their garage. So like, who, who hasn't thrown in their garage? Like what are you thinking? What have you seen? The ones where the thing becomes like a

Drex: projectile.

Bill: Like,

Drex: like, oh, it's amazing. It gets in there and just, it really does.

A lot of it I think is 'cause the turkey's wet and it just kind of explodes once it hits the oil. Well. I've never

Bill: had fried Turkey, but I hope it's worth it. It's delicious.

Charles: It was delicious. Yeah.

Bill: Well, Drex, what are you looking forward to?

Drex: Probably apple pie or pecan pie. Those are my, that's my, I am a dessert guy, so

Bill: yeah, I am as well.

Drex: think the company that decides that actually is a thing and people like pumpkin and we're just gonna do it all the time, I think that's big money.

Bill: That's big. It's it's a marketing ploy. Is that what you're telling me? Yeah. Yeah. Gentlemen, always great to hang out. Charles, thanks for coming on the show. Appreciate it. Thank you guys. Good conversation.

That's Newsday. Stay informed between episodes with our Daily Insights email. And remember, every healthcare leader needs a community they can lean on and learn from. Subscribe at this week, health.com/subscribe. Thanks for listening. That's all for now.