This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

UnHack the Podcast: Children's Hospital Code Dark Plan with Laurie Campbell and Rick McIntosh

Drex DeFord: I'm Drex Deford, president of Cybersecurity and Risk at this Week, health in the 2 29 Project. Our mission is healthcare transformation powered by community. Welcome to UnHack, where we navigate healthcare security challenges together because cyber safety is patient safety.

Let's get started. Hey everyone, welcome to unpack the podcast. I'm Drex Sand. Uh, I'm really lucky today I have some great folks from Children's Hospital, Colorado here. Uh, say Hi Rick.

Rick McIntosh: Hello,

Drex DeFord: and Lori.

Laurie Campbell: Hi.

Drex DeFord: , We should probably start, we're gonna talk a little bit about resilience today.

Rick McIntosh: Sounds great and, uh, happy to be here. Thanks. Thanks for the invite and the opportunity to kind of share our experiences. But I'm Rick McIntosh, CTO at Children's Hospital Colorado. I've been in the IT industry for 30 plus years, and, uh, 17 of those, almost 17, uh, here at Children's. Um, prior to that was, uh, a reseller integrator, um, so on and, you know, Southern California and Nevada before coming out here to Colorado.

And, uh, just really, uh. Hit by the mission of what all the children's hospitals do. And so, um, that's why I'm still here 17 years later.

Drex DeFord: There's nothing like being at a children's hospital. I was the CIO at Seattle Children's once upon a time. So it's definitely a great mission. Yeah. Lori, tell us, uh, tell us a little bit about you and your background.

Of our code dark initiative, um, helping us to keep our doors open during an extended downtime and be able to create, um, and provide, uh, great patient care.

Drex DeFord: Lori, this feels like your, your job role is one of those things that like when you do a really good job, people just give it, keep giving you more stuff.

Has that turned out to be the kind of the case a little bit?

Laurie Campbell: That's typically what happens. Um, I, I've acquired a lot over the years and continue to acquire more and more systems, um, and areas so. Um, but I love children's. I've been here. My daughter had care here. Um, and so it's just, it speaks to my heart to be here.

And I was like, I wasn't at Seattle too, but I totally get it. Right. You're a, mm-hmm. You're a. You're a community anchor and so mm-hmm. Um, it's really important that you're there all the time for patients and families in the community, which kind of takes us back to this conversation around code dark and resilience.

Rick, um, kind of start me on the path here. Uh, how long ago was it? What have you been doing? You know, you think about like, I think at some point. You know, in our, in our history, it was like, oh, it's theoretical. Maybe that we would go down and be down for a long time. But it, at some point, something must have hit you all and you know, you said, no, this is actually a thing that can happen and we need to start planning for it.

Tell me a little bit about that story and how you then got into the, into the project.

You know, we've always known that, you know, healthcare information is valuable, more valuable than credit cards and so on, and people are always trying to steal that data. But, probably in the four to five year ago timeframe, that's when we started hearing stories about, um, these, you know, ransomware people encrypting EMR databases and holding it hostage and, you know, extorting money and so on.

And um, and we all know that technology in healthcare is. So critical., We went from, you know, 10 know 20 years ago where technology was just, we were a shared service. It was kind of a nice to have and people relied on it. But, you know, in the last 10 years, technology's become a strategic.

Weapon or tool for the organization. and particularly in the last 10 years, um, like no care

Drex DeFord: happens without information services today.

And of course that's what makes that extortion, you know. Doable is because you can't function without your EMR or like, you know, your supply chain, your ERP system and so on. And so, going back probably about three years ago, I want to say, we realized, and it's a difficult admission to tell yourself.

But It's more of a when than an if. We all like to think we have great defenses and so on, but it's kind of the whack-a-mole game that we all know, you know, I mean, the, the, you know, the bad actors are morphing their technologies rapidly. We're always playing catch up and so on. And so when you really start thinking about it from the context of it's a win and not an if question, then we have to start being prepared for when it does happen.

Clinical and nonclinical. Right? I mean, payroll, supply chain, I mean all these things that are not clinical, but absolutely crucial to an organization continuing to function and being able to provide, you know, good patient care safely and so on. And um, and that's when we decided. in It that we really need to facilitate these conversations around what happens.

When technology's gone for an extended period of time, not an outage or a bug or something that causes a server to crash and you recover in a few hours or whatever that may be, but an extended period of time. You know, the 30, the 45 day window and so on. And that's when, and and Lori will jump in here in a second, but I wanna say it was around two years ago or so, maybe a little bit over at this 0.2 and a half, where we decided it was on us to really facilitate those conversations.

Drex DeFord: Yeah.

Rick McIntosh: And um, and that was met with some, you know, big eyes and some shock looks like that can happen.

And it's like, yeah, it is happening. And of course by then, Lori Child's others, you know, it's all hitting the news. Yeah. It's like, oh wow, this, this really could happen to us and we need to be prepared.

Drex DeFord: It's interesting how the, you know, just even as I think back, um, the business continuity thing, you know, we used to always talk to the departments about, you know, what's your plan to go to paper?

What's your plan to operate? Should the systems go down? And a lot of times my clinical and business department struggled with, just like I, and I think part of it was like, I don't know what you mean. And so at some point it really became. Okay. You all obviously have done this. We're gonna take the ball and run with it.

We're gonna teach you what it means and how it works and mapping out your workflow. Lori, tell me that whole, like how did, how did you get started and how did you convince them to go along with it?

Um, clinical and business units and operational units, and it was really interesting the, we talk about it that as we went into the departments, it would take us. 3, 4, 5 meetings to be able to allow those departments and units and clinics to wrap their head around the fact that we could be down for 30 to 45 days.

Mm-hmm. And the first response is always, no, well, we'll just have to shut the doors.

Um, in the beginning it was really tough. It took a long time for people to realize, no, we're gonna have to stay open and we're gonna have to, figure out and determine what we need to do to provide safe patient care. During that time,

Drex DeFord: how, how'd you do it? Did you go sort of like department by department by department?

Did you prioritize the department somehow and kind of said, let's start here and we'll just kind of work our way through,

Laurie Campbell: about that. Yeah. So originally my vision was that we would take a patient all the way through their continuity of care. Oh, from, for instance, the a kid that had been in an accident was a trauma hit one of our network of care sites.

They're air flighted to, an shoots and that Anschutz, they go through the ed, they need services such. Pharmacy and radiology and they need surgery. And so taking 'em through all of those and then, you know, into med surg for, care for that. And then of course, for discharge. And what we realized when we first started that was, that was a huge scope.

Really the most. Mm-hmm. And so we scaled it back to that and those were our initial, um, departments and units that we worked with.

Rick McIntosh: Well, and in that, in that first year we did, just to pile onto that, also include a few of the core nonclinical

Laurie Campbell: Yes.

Rick McIntosh: Departments that are also very important. I mean, if you think about what goes on in a hospital and, and how critical the supply chain is, um, the revenue cycle, you know.

I tried to pick off all those important key areas that, um, that we knew had to continue functioning and learned with those, and then, you know, eventually working our way through the entire organization, which I don't think we're quite done yet.

Drex DeFord: So I like the idea of taking the patient journey and kind of.

Starting there. Mm-hmm. And then you kind of had to back up and say, wait a minute, if we try to do all of this at the same time, it's too much. So you've really looked at sort of like Lab Rad Pharmacy, the things that every unit and every clinic probably uses. You've kind of started there. Is that mm-hmm.

Am I kind of getting that right?

Laurie Campbell: Yes, that and also, um, an ICU unit as well, and the ed, so the entry, the two entries into the hospital for patients as well.

think through this whole new idea of being down and how are we gonna run the business? How are we gonna register patients? How are we gonna file claims? How, how'd that work? And Lori, feel free to jump in on this too.

Rick McIntosh: Sure. Yeah. And Lori was deeply involved in all that work, but it, it's the same conversation, whether it's clinical or nonclinical of just this whole idea of, of, you know, people wrapping their heads around the concept of no technology.

Right. Okay. What if, we have. No billing and no, no way to record. We're doing all this on paper and most people in healthcare. Aren't necessarily old enough to remember the day. Yeah. I don't even

Drex DeFord: know

Rick McIntosh: what to do.

Drex DeFord: I don't know what

Rick McIntosh: we're talking about. Paper. And so it's, it's a similar conversation with different, you know, some of it's not life safety or, patient safety type stuff, but it's that same conversation of how do we function?

Um, but um, same thing with the payers of, okay, we're not gonna be able to bill you yet. We still need. Revenue. So, how do we work out this system where just keep paying us what you paid us before and um, and then we'll audit it and true it all up 'cause it's gonna be reasonably close. You know, those kind of things and I'm way oversimplifying it.

But those are the conversations that need to take place and it takes, you know, our, our, our business stakeholders, you know, having good relationships with, you know, those suppliers and those vendors and so on. Make all that work and have those plans in place.

Drex DeFord: So, so did it work? Were those, did those plans kind of come together?

You've got some of those kinds of agreements in place now?

Rick McIntosh: Yeah.

Laurie Campbell: Um, the, the one thing that we did do is we created a repeatable process.

And so it doesn't really, like Rick was saying, it doesn't really matter that it's business or operational or clinical. Um, the repeatable process is that we come in, um, we talk to you about your workflow and, and that's, that's tough, right? Because people wanna go to the technology first and we say, okay, nope, there's nothing.

And, and it's funny because they would go through the what ifs. Well, but what if I, there can't be any what ifs. So you have to say, we are down, we're down, we're in a code dark situation. Uh, we may have ransomware, extended downtime, and you're not gonna have any technology for, we don't know how long. And so we have to build systems.

Okay, fine. You write an order. Okay, that's gonna be a paper order now. So you're gonna write an order and that order's gonna go to the lab. How's it gonna get to the lab? Well, we're gonna need a runner to get to the lab because our tube system is not working. So you're gonna run that to the lab we borrowed this, um, I think it was from Laurie. Uh, they had a bouncer at the lab. So they, they read the, the order, and if they can't read it. Send it back to be rewritten. Um, so they have a lab, lab bouncer. What the idea of

Drex DeFord: bouncer. That's some big tough guy with tattoos standing in their lab door.

Drex DeFord: Mm-hmm.

Laurie Campbell: One of the challenges that we had is, you know, we're talking, let's talk about lab. Right? It's shared service, and then we also have. Um, other areas, areas that use the lab, for instance, an inpatient unit, for instance. And so one of the challenges we had was keeping people on track because. When you get in and say, okay, what's your workflow?

It's like, well, okay, I'm gonna write this order. It's gonna go to the lab. What's gonna happen? Well, we haven't talked to lab yet, so we're gonna stay with you. So let's, let's just say, okay, that order has gone down to the lab. They've taken care of that the results have come back.

Now what are you gonna do? Well, I wanna know what. What is happening in the lab? Well, you'll know that eventually, but right now we can't do everything at once.

Rick McIntosh: Uhhuh.

Laurie Campbell: So we start first with that unit or clinic. Yeah. And, and work through that procedure. And then we go to the next one. So maybe we did PICU for instance, then we go to lab and then we come back and we True up.

Drex DeFord: Do they update their workflow? I mean, as their workflow then changes over the course of a year because of a new system or a new process, they have to update those workflows again.

Laurie Campbell: Exactly. So, um, And the, the whole industry is now aware and, and working on solutions to try to keep you up and running somewhat. Mm-hmm. And so whether it's a new. Product that we're bringing in, or one of our vendors said, Hey, we can keep functioning if we do X, Y, Z.

Drex DeFord: Mm-hmm.

Laurie Campbell: And then we now can write that into our procedure, that we can have that system up and running pretty quickly, um, with, with our, um, changes.

So it's very iterative.

Mm-hmm. So good on you. Definitely. We're doing that

Rick McIntosh: and there's, there's a lot of nuances to it too because there's, you know, there's that whole negotiating that goes on of like, okay, there's a downtime. Well, will we have internet access or can I use a spreadsheet on my computer? Or, you know, you know, will I have a cell phone?

Or those things? And so then, you know, the complexity can spiral.

Drex DeFord: Mm-hmm.

Rick McIntosh: Big time because there are, remedies for those things, right? We have plans to distribute, like clean laptops and so on. We have plans to use our das system and hotspots for internet access. We have plans with, um, like, you know, the cell carriers that'll come, drop a truck with antennas in your parking lot and give you additional bandwidth.

Of those. Um, but you start with the worst case scenario.

Drex DeFord: I think it was Mike, I think it was Mike Tyson who said something like, everybody's got a good plan until they get punched in the face.

Rick McIntosh: Exactly. Punched in the mouth. Yeah. That was his

Drex DeFord: punched the mouth. Yeah, have, have I wanna go from the absolutely worst case to some of the other cool stuff that you've done.

Um, in my head, I want to use the term minimum viable hospital, but I know that's not necessarily exactly, um, what you're doing. It's sort of alludes to this idea, Lori, that you were talking about, where there are some things that can function. 'cause we have technology to be able to run those things and other things will be on paper.

That's the. Sort of playing it by ear in the heat of the moment, in the heat of the battle. But at least you've thought through those scenarios, you've got ideas about how all that would work. So tell me about some of the other tech stuff that you've done to maybe lean into should you have the code dark happen?

And that's where we've come up with some creative ideas of how do we, how do we help? Uh, the departments for, for instance, supply chain is a good example. They're never gonna be able to manage the supply on a piece of paper.

Right? They have to have something. So then we give them, um, a, uh, clean computer, basic clean computer with maybe excel or, sheets or something like that, that they can use, that they can at least, you know, bring inventory in and distribute it and know where it's going.

And so, um, we came up with Sneakernet. Taking images off the modalities, putting them on and prepping a laptop. So a whole operational piece is logging in the image, identifying the patient, getting, you know, taking the patient's chart with it. And then we give a radiologist a laptop that's ready to go with the images already on it.

Drex DeFord: Exactly.

Laurie Campbell: Right. Better than the first phase. And then the second phase is, is then we look at, okay, how do we get them in a room? Get them back to their reading room, in their monitors, in their dark room and, um, give them a work list. And so that's creating a, you know, a closed loop LAN that is just isolated into that one room.

And then somebody, one person can be uploading images into a work list that they can be reading from. So that was one of the creative things that we came up with, um, for radiology.

Drex DeFord: It's a huge amount of innovation. , Rick, I don't know if you guys are watching the Pit right now.

Laurie Campbell: Uhhuh.

Drex DeFord: Yes. I

Laurie Campbell: love

the

Drex DeFord: pit.

So everybody, I mean, I can't tell you how many notes and calls and stuff that I've, um, yeah. Received. You all probably have too.

Laurie Campbell: Yeah.

You're making a trade off about speed versus safety or transparency versus control or uptime versus, you know, I wish it was perfect. I wish we had more integrity. How do you deal with some of those conversations as you've gone through building these plans and, and getting them up and running? Rick, I'll start with you.

Rick McIntosh: safety is not really an option, right? I mean, that's always top of mind. I mean, patient safety is first and foremost a hundred percent of the time.

Um, so we're, we're not gonna compromise that to speed anything up. It usually drives a conversation around what can we do different, um, if there's, if, if there's a need, , and there always is, but how do we get technology to these different business units faster?

So then, you know, working with some incident response firms and so on, you know, how do we contain, and then how do we start utilizing aspects of the environment that aren't yet impacted? Or how do we repurpose, um. Like for example, most organizations have internet pads out of multiple data centers, right?

So, you know, how do we utilize, um, our network infrastructure, um, utilize our different wireless SSIDs and so on, to then maybe create secure environments where we can restore some services faster and so on. And, that's gonna be dependent on, you know, each different environment, um, you know, in terms of your listener base and so on, but.

Drex DeFord: It makes great sense, Laura, are there examples of that besides radiology that you'd, you'd point out?

That, that you've, you've figured out ways to kind of bring services back faster to some. It's pro. It's probably all part of the evolution. Yeah. But of the plan,

Laurie Campbell: as a manager for clinical ancillary apps, I'm responsible for all the different applications, the clinical applications that are out there, um, that are ancillary apps.

And so for us, trying to figure out how to get them back up and working and running. So really working with either the vendor, which is, which is interesting because when we started this. We were the only people calling the vendors and saying, what can you do if we're down? What? What do we have? Yeah. And a lot of the vendors just.

Is tasked to go out and do certain things. We have to flip things to local or something like that.

Drex DeFord: Sure.

Laurie Campbell: Um, and so we have a whole, a whole dashboard, um, and tasks within that zero to 24 hours that my team has to accomplish, uh, to get us to be able to continue to function with some of those systems. Does that answer your question?

Drex DeFord: It does. It's the amazing discovery process in all of this, I think, where you start to look at workflow and surprise that there's all this other stuff in workflow that you didn't know about. Then you start to look at systems and you realize like, uh, this system is actually dependent upon six other systems to actually work.

Tell me how your third party partner ecosystem is also playing into this analysis of. What's important? How do we plan? How do we plan when one of those partners is down? Not that your network is down, but there's somebody really important to the workflow offline.

Rick McIntosh: That's actually a two part process, right? Because when you onboard technologies, you also have to have that conversation.

Drex DeFord: Hmm.

Rick McIntosh: So we have a, we have a process by which we, you know, we onboard technology to the organization and whether it's an application that's on-prem running in our data centers.

What are our alternatives? So we've had some of those conversations already. Now, if that happens to be because of, you know, bad actors and, malware, ransomware, and so on, changes our response a little bit, right? I mean, one of the most obvious ones being is if, although in most cases we don't see it a lot anymore, but there are some vendors that still require VPNs to function and so on, and we're gonna terminate those connections really fast.

Mm-hmm. Um. Just to be safe. I mean you have to. but um, but a lot of those conversations around just what to do without this vendor, if they go dark, have, have occurred now. With that said, Lord, please. Elaborate.

Laurie Campbell: Yeah. No, I think you did a great job. I think again, on the, on the front end side of that is, you know, are we gonna drop the VPN if we have one?

And so then we start looking at those downtime procedures of how are you gonna function if we bring the system down and what are you gonna, or,

Drex DeFord: or a backup vendor. Do you look at a backup vendor in some of these cases too? Like what's your plan B if plan A fails?

Laurie Campbell: Yeah. And you can, and we can do that definitely if, if there's a vendor that we know.

Um. We can engage a different vendor and say we need some assistance with this. Um, so definitely you can do that. I do think there's a couple of areas that have done that as well.

Drex DeFord: I'm gonna try to close 'cause I feel like I could talk to you guys forever about this. Um. Looking back kind of at one investment, the technology, the process, the people, the one investment that you think kind of made the biggest difference in helping you make progress on the resilience journey, what is it?

Rick McIntosh: What's the investment of time? Um, I mean this, you know, it's, this is not like a capital investment or, any technology that, you know, or tools or anything that we bought. It was really a, a, an understanding and, uh, the organizationally, the commitment to say, you know, not only are we gonna take a number of, it, people and so on, and project managers and others, informaticists and, and all the people necessary to help document all this and so on

but there's time of every, every department in the organization to really have these conversations and it's, , we we're, we. We'll be going on three years this year of doing this work. And you learn a lot about the organization and the process too. I mean, we've learned a ton, um, about, a lot of the, dependencies and nuances of, of how a lot of these different departments, business processes or clinical, how all these things function.

But it's a huge investment at time, um, um, but well worth it.

Drex DeFord: Yeah. Laurie.

Um, and then we got a year and a half. Then we, uh, actually got a formal project when some people did. To be dedicated to it. So executive leadership is key to this, and it's from the top down for the organization and everybody has supported this work. We've not, you know, it's rare that we go into a department that says, Hey, we can't help you.

Sometimes they're busy. Sometimes we're in respiratory season, we have to move things the blocks around a little bit. But the support is, has been amazing. Uh, for this.

Drex DeFord: It's amazing when you have the top down support, how much. Attention and momentum you can actually get. I'm gonna, I'm gonna do, this is just a personal thing I like to do 'cause it's kind of fun and interesting and I do this as sometimes at the summits too.

What's your, what's your walkup song, Lori?

Laurie Campbell: I would say believer.

the teams, they believe in this, they, you know, maybe we're a little hesitant to start with, but really the organization believes in this project and this initiative. Uh, and I'm just super proud to be part of it.

Drex DeFord: All right. I like it. I like it. Rick, what's your walkup song?

Rick McIntosh: That's great, Lori. I don't have anything that meaningful. I would probably wanna do, like, uh, DM x's, you know, X gonna give it to you simply because, not 'cause the song has any meaning whatsoever. But it'll get the whole room going, right? That's right.

Drex DeFord: Oh, good. Thank you guys for, uh, for being on the show today.

I really appreciate it. Lori and Rick from Children's Hospital Colorado. Uh, thanks for being on unh, the podcast. Thanks for joining on UnHack. Remember, we're not alone in this. Every healthcare leader needs a community to lean on and learn from. Join our community at this week, health.com/subscribe and share this not only with your security crew, but with your entire leadership team and staff.

Together we are stronger.