This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

This episode is brought to you by TIDO. TIDO provides managed integration , EHR, data migration archiving, workflow automation, and 24x7 monitoring to ensure seamless data exchange and improved operational efficiency.

Dealing with capacity challenges in data or integration projects and operational optimization issues, leverage TIDO's AI driven solutions to reduce costs and enhance patient care continuity. Ready to elevate your healthcare IT strategy? Visit ThisWeekHealth. com slash TIDO today and discover how

: innovative solutions can transform your organization

I'm Bill Russell, creator of this week Health, where our mission is to transform healthcare one connection at a time. Welcome to Newsday, breaking Down the Health it headlines that matter most. Let's jump into the news.

Vik Patel: Vik? I'm good. It's it's Diwali today, by the way. Happy Diwali. Oh, happy Diwali.

Bill Russell: And obviously Drex, the Ford is here otherwise known as uncle Drex as we call him.

Oh, don't do it.

It's just gonna continue. And then Sarah Richardson back from a, an amazing summit 2 29 summit out in Napa. So it's good to see you back in town and ready to go. And hopefully we won't have a a cad appearance on the show again.

Sarah Richardson: They're all asleep right now.

Wow. This is, you know, this is what happens when we [00:02:00] consolidate all this stuff. I mean, there's a lot of things running over here. Root cause let's see, this NBC news, I don't know. We'll see how close we feel this one.

Vik Patel: Are they blaming? DNS.

Bill Russell: While the issue originated from within the EC2 internal network and impacted core AWS services, including Dynamo Db SQS and Amazon Connect, AWS identified the root cause as underlying internal subsystem responsible for monitoring the health of the network.

Load balancers. Yeah, it could be DNS. That's a pretty, I don't know, drex. That feels like a pretty broad.

Drex DeFord: Broad

Bill Russell: statement.

Drex DeFord: My gut tells me like somebody probably did something that they weren't supposed to do. Somebody clicked a button from on to off or off to on, and there were a lot of unintended consequences and it took a while to figure it out.

This is probably like a change control thing that didn't work the way it was supposed to.

I'm trying to see if there's any healthcare. I'm sure healthcare was impacted, but I don't none of those

Vik Patel: lot of people screaming were the Roblox and Fortnite users, obviously, you know, they're everywhere. You know, if you go on Reddit or anywhere else, even Reddit was affected,

Sarah Richardson: you're hosting, I mean, it could have, it could affect. Claims processing your data collection, insurance verification. Just some of the critical infrastructure, although I had to, you know, I had to go down my little path of, well, if it happened in Virginia, is it not? Maybe it's tied to the government shutdown. .

Bill Russell: Here's what it says for current status. Well, AWS initially said the. DNS issue. So there you go. DNS issue was fully mitigated at 2:24 AM Pacific Daylight Time. Customers continue experiencing increased error rates when launching new EC2 instances as of 1:30 PM Eastern time. AWS reported seeing early signs.

They put it out. So, I mean, unless it's obviously a cyber attack of some kind, but they generally will let you know exactly what's going on and not just their clients. They let the world though. I mean there's no, if you go to

Drex DeFord: health.aws.amazon.com, you will see every outage that they have sort of like a running live dashboard that they update with all the stuff that's going on.

So Drex, do you know any health system that has that? No, I do. I do not. I mean it's kind of interesting. There might be some version of this where they send out a regular email update. But yeah, I dunno about the dashboard. Yeah,

Bill Russell: what does healthcare take away from this?

Some of 'em are hosting with Epic themselves, which which is an option as well. You know what do we take away from this If you are a healthcare leader, Sarah, I'll start with you and then Vik, I'll come over to you. Sarah if you're a leader, because you were moving an awful lot of your stuff to the cloud.

Sarah Richardson: We actually, we moved everything to AWS, we did it two ways though. We did the multi-region AWS, which isn't completely soundproof. I mean, you've gotta think about multi-cloud as much as multi-region. However, AWS does have data centers all over the world, and so when we did a full scale move to AWS activity, we did multi-region as one of our primaries, and we did have some backup services with Azure.

Bill Russell: So this is one of those no dust statements, but architecture still matters even if you're going to the cloud.

Like you can't just go, well, we went to AWS and you know, clearly they have all this stuff worked out. I mean, it's even more so you have to think about it.

Sarah Richardson: It's very specifically, and depending how you have it, how you have it designed, that architectural vulnerability like AWS is architected with thousands literally of microservices that are communicating, I think continuously with one another.

So when you have a foundational serVike, like we had a lot in their Dynamo DB and the Kinesis experience had problems in the past that can propagate to other services. And so you need to have an API gateway that checks NMO for configuration. Lambda function needs to write logs. CloudWatch has to have database storage for metrics.

Of connections and some of them you don't have control over all of 'em. You only have the awareness factor. You've gotta know where to shut it off as much as you need to know where to connect it and be able to turn it on effectively.

Bill Russell: Vik what's your takeaway? Healthcare leaders, what's, what should their takeaway be from this outage?

Well,

Vik Patel: first of all, Sarah, like do, are you on the Amazon AWS support team? Like,

Sarah Richardson: no. But when I went to their services, I had to make sure I knew what all of those, you know

Vik Patel: a lot about it. I mean, that's amazing. But yeah, you know who's who's kind of laughing today, it's probably the people who are like, I am not going to the cloud.

Like the lab instruments and other vital signs, instruments that you need by the bad side, you don't need all those going to the cloud. And then coming back, right? Like you, you probably need an architecture which supports a lot of that integration. I'm talking about integration in this case, connecting to something OnPrem instead of making that hop every time to the cloud and back because in this kind of situation.

If that happens now, you just affected patient care within the hospital, right? Like, that's terrible. That was a terrible architecture. So those are the kind of things that I think you should pay a lot of attention to. So, I mean, yeah, every time this kind of event happens, I think it's a, it's an amazing opportunity to sit back and look at what's in place.

Bill Russell: Drex, you know, there's so many things we're supposed to take a look at and I'm not sure we're looking at all those things. I like the architect concept.

And it is almost reverse Dr if you will. And in, in California it makes sense. because if we had. Our disaster was an earthquake and it could take out, I mean, it could take out everything quite frankly. And we were also serving Northern California and West Texas from there as well. And you didn't want to take them out just because Southern California got taken out.

Drex DeFord: I think there's this interesting sort of, path that we've been on through the pandemic, where we pushed a lot of stuff out of the data center and into software as a service And we don't really understand what's happening with the software as a service company and who they're using as fourth and fifth level, you know, service providers like AWS or anybody else.

And so. It's hard for us, and I mean, part of the reason we did that is that we didn't wanna be in that business, so we moved it out to someone else. I think the same thing when we move workloads, we do other things with companies like AWS. Part of that is that we don't wanna be in that business anymore.

And now the world has changed. and Network latency is way better, and we've got technology that can handle that, and we wind up in this word situation that we do things because we can, not because we necessarily should. The same thing applies to those contracts. When you read those contracts and you look at architecture, make sure you've got penalties in there for downtime and all those things.

The problem is that when you're down, you may have the penalties written in, but it doesn't really help you in the heat of the moment when. Your EHR is offline for six hours because something's happening at AWS. I mean, we try to shift the responsibility somewhere else and we try to contractually obligate the risk to someone else, but I don't know if you can really do that.

And he's like, Hey, why don't I talk to somebody, I think we can do this on prem. And I was like, really? I mean. If you have $500 million, you know, to build the same infrastructure so I think that's where, you know, once I kind of walked him through what it would actually take and if Microsoft would even

allow that kind of licensing, first of all, right? Like the Azure licensing on-prem. I mean, I don't even, I haven't heard anybody do that. You gotta build the scale

Drex DeFord: and everything too, to be able to keep, you know, you've got a lot of stuff sitting there idling locally. It's part of the advantage of the cloud is that you have, that you can surge when you need to surge.

And

So that's where, again, going back to the architecture, you know, what are the things that you can still get away being OnPrem and should help you with patient care, but the ones where you do need to use the cloud. How do you do it intelligently? So if when something like this happens, you're still not impacting patient care.

Bill Russell: So Vik let's get into your world. So Tido really focuses in on the data aspect of well data in an AI world, data in a cloud world, if you will. What do we need to be thinking about when we're moving to the cloud and our data and being able to, I don't know, do what we want to do?

You know, I what does data in the cloud world and the AI world look like?

Vik Patel: No, that's a great question and I'm glad you kind of capped it a little bit vague. So, I will go to one of our latest solutions around Tito Cortex ai where. We are using our expertise that we have built in-house with our Mitre ai, the proactive monitoring of interfaces and applications.

So we are partnering with organizations to help them implement their AI applications. Right? Like we are not trying to be Emmy or whatever, you know, all the other products that, that Epic and the other EHRs are doing. That's not what we are trying to do. We are just partnering with the organizations, bring our data scientists, bring our technical team and work with them.

To the analytics, all that stuff, but sending data to the cloud, it's like, oh yeah, let's just send all our EHR data, you know, to the cloud for processing and then maybe use it in open ai, which would be terrible if you just send PHI in there. So one of the things that we are actually doing is we are working with organi with three organizations today where.

So you can still have all the data that many of these applications, especially the analytics and the reporting side of things that we'll need. They don't necessarily need to know who that John Doe is. You know, you need to have the data that you can use to make sense of it and have some actions on top of it.

But why not use de-identified data? Right. So I think that could reduce the risk largely. And you know, we are creating, a pipeline to actually do that on-prem or wherever they have their data before it goes to Databricks or before it goes to the cloud and blobs.

And then, you know, that's where it's then formatted for the AI data lake. But I would suggest that, I would say don't expose all that data. I mean, that's taking on a lot of risk once it's out there. And once it's consumed by some of these applications, you have no idea where it's going to end up.

And he said the magic word, he said risk. And as he's sort of describing that, I'm sure that was part of the conversation you guys had.

Drex DeFord: There's so much to that, right? Like are you gonna build your own LLM model? Are you going to try to leverage an existing LLM model like OpenAI and then.

There's obviously levels of risk in that conversation around that first decision. And then how are you going to manage the data that goes into that Part of it certainly ties back to, I mean, I think when we go back to the. Fundamentals. The fundamentals are, do I know what data I have and where it is, and am I sure that it's actually correct and clean and usable and, you know, all of the things that go along with the data conversation.

Vik when you're. Talking to health systems who are kind of in that position, how do you help walk them through that or think through that challenge? Because I think for the health systems that were in my CISO event, there, a lot of them are still struggling with that issue.

You know, there's all the other clinical data too, but just to kind of keep it simplified, that's where you may have to adjust how you make use of data, which data you trust more, and then allocate more points to the more trusted system. So you have to make those kind of adjustments, but having, you know, you just need the right team.

Who can understand that and make those adjustments. So I think that's a great example. Drex in terms of, yeah, like that's part of the process when you are putting all the data together that your future AI applications will use.

Drex DeFord: Yeah, I think that's really interesting, right? This idea that because we have so much crappy data in so many different systems that maybe AI can help us sort that out.

Maybe that's the great use for ai,

Just across the board you had great data pipelines and whatnot. I'd love for you to weigh in here.

Sarah Richardson: Yeah, the what was most important was getting the, just the foundational team in place. So we had the whole like data fabric, so that was just a title or ti something people called it, but higher leaders who knew how to do data, people who could do architecture, who could do the governance, who could do the reporting, the intake, the normalization.

That, well, we couldn't de-identify it for married reasons, the way we were structured with the health plans, but creating a space and only allowed for that and all be a clean space. Like we had to create owners of every single pipeline. People knew where things went, and then we did work with the health plans to be able to pull their own reports.

The hardest part of all of that. Wasn't actually the architecture and the technology around it. It was getting the CFO and the business owners to agree to the cost of ongoing care and feeding and maintenance. It's not like you clean up your data once and it's done, and then you go ahead and just have this beautiful warehouse with all these reporting features on top of it.

All that lineage and quality and tracing that information is really key. In addition to the security aspect of something goes wrong, what needs to get turned off first, and so that was an entire team. I always went back to the board and to my C-suite peers to explain not just the efficacy and value of the program, but why we had to constantly fund it and what that meant for us organizationally.

So. Hire people that is their job.

Bill Russell: When I came into healthcare again, 20 10, 20 11 ish timeframe, one of the first things I did was stand up, get data governance program. And because every other industry I'd ever been in had a data governance program, I just assumed that healthcare would have a data governance program.

And then the thing I think that still surprises me here we are a little less than a decade later. Or over a decade later, and we still hear people saying, well, you know, we're gonna stand up a data governance program. I'm like. Really? Wow. Like, how can that still be a thing that would be like saying, we're thinking of hiring our first ciso.

With direct on the thing. I have to say, you know, if you de decide to, you know, what is your risk profile look like? How are you gonna, you know, data at rest data you know, you get encrypted at rest data and transport and all those things. You gotta build all that stuff out, which.

I'm sure most health systems have done 'cause they have a good security program. But on the data side, you know, data definitions make all the difference in the world. I mean, let's just take the basic most simplest thing and most health organizations have taken care of this, but when I came in, I remember the first request I got was from the reporting team and they're like, Hey, we need five more people.

And I wanted to be the good CIO. And I'm like, oh yeah, of course you need five more people. Let's get you five more people. And then the next year they came back and said, Hey, we need five more people. Until I, you know, and at that point, of course, you don't have to do this to me twice, then I'd sit there and go.

That was my first report. That was like. Are you kidding? None of these reports are getting used.

Drex DeFord: Why are we doing, we have 25,000 reports in the inventory and 22 of them are actually used on a monthly basis or something. Yeah,

Bill Russell: we had a great conversation around simplicity as a strategy. For it, like, just as a core design principle for it.

And in our CIO meeting, it was really interesting to hear. It's like, look, complexity is it is sort of a gravitational pool in healthcare and part of the role of the CIO I'd love, we'll close on this. I'd love for each of you to sort of. Discuss this from your own perspective. The gravitational pool is towards complexity.

I'm trying to figure out who I want to give the last word to. Vik. I'm gonna give you the last word. So Sarah, we'll start with you.

Sarah Richardson: I will put this on my banner that I fly behind a plane or anywhere else. To me, a hallmark of how well we are functioning as an IT organization is when other people inside our organization can tell our story on our behalf because we are there to serve others.

And so if you're the chief financial officer. You can tell the quote unquote it story in terms of how it's benefiting your organization. We're doing the right things.

Drex DeFord: You've heard me go off about this before, but I mean, I think it has to be an anchor in all the things that you do,

It's more likely to have uptime and ha not have some crazy downtime that you can't figure out, and you have to talk about it as a DNS outage because the environment is simple and so. The simpler you can make everything and you're right. I mean, I think the environment pulls you toward complexity every day.

And part of the job of the CIO is to figure out, to be the champion of simplicity that has to be part of the job.

Bill Russell: Vik, last word on this topic.

For the care teams. Right. It should be like no brainer to them. It should be, like we don't, they shouldn't be calling into the help desk when there's an issue. We should know about it and I'll, you know, talk about how our mi ai does that very proactively finds things. But I think that's the difficult part, like simplicity.

If it was that easy, everybody would've done it. So I think, again, I don't think it matters how many applications at the end of the day, you can have one application, but you can still make a mess of it still won't be simple. So I, you know, from that,

Bill Russell: where you were going, I wasn't gonna agree with you, but from that perspective, I agree with you.

Even with one application, it can get messy.

Bill Russell: this is all real simple to me. It is. You know, the only reason we need data is to answer questions. Right. That's what we needed. So people are asking questions they need data. And my experience this morning was, I need to know how many partners were gonna be at our city or Philadelphia City Staff Roundtable next week.

And so I started it in, in Google Sheets and it wasn't there. Then I went over to Airtable and it wasn't there. It probably was in both of those locations. I just couldn't find it there. And then I and then I ended up looking at a different Google sheet and found it over there and. How that experience probably plays over and over again in every organization of, they just want answers to questions.

If you want to know where I got the answer. It's here, and here. It's these 10 websites that I went through. And so, you know, again, I, maybe I'm oversimplifying, but data is about get, getting answers. And at the end of the day the best way to do that is to know what your data is. What questions it can answer and be able to find it and funnel it into the, in the right direction.

I think that's what you guys are doing. Vik and I think we're in wild agreement, which our production team is not gonna like, they like it when we argue and disagree with each other, but I think in this one we're pretty well in lockstep with each other. Hey Vik, thank you for coming on the show.

It's always a pleasure to hang out and discuss the news with you.

Vik Patel: Thank you so much for having me.