TownHall: Implementing A/B Testing to Study Medical Alert Effectiveness
Episode 2616th March 2023 • This Week Health: Conference • This Week Health
00:00:00 00:23:58

Share Episode


This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

Today on This Week Health.

It's that subjective experience of using the E H R that's probably more associated with burnout and wellness

so I, think it's a great if we can show that the notes are this many characters shorter but what I think really matters is what that note writing experience is like. Do providers feel like writing the notice faster

Welcome to TownHall. A show hosted by leaders on the front lines with interviews of people making things happen in healthcare with technology. My name is Bill Russell, the creator of This Week Health, a set of channels dedicated to keeping health IT staff and engaged. For five years we've been making podcasts that amplify great thinking to propel healthcare forward. We want to thank our show partners, MEDITECH and Transcarent, for investing in our mission to develop the next generation of health leaders now onto our show.

Hey everybody. I'm Jake Lancaster and Internal Medicine Physician and the Chief Medical Information Officer for Baptist based outta Memphis, Tennessee. And today I'm, very honored to have Dr. Navani, a clinical fellow and pediatrician at Stanford

Nevi, welcome to the program.

Hey, thanks so much for having me.

Yeah, we're gonna have a really exciting conversation today. We're gonna talk about some note redesign that you've been working on as well as this pseudo clinical trial for alerts. But, first can you just tell the audience a little

bit about your background?

Yeah. So, as you mentioned, I'm a general pediatrician.

I actually came to clinical informatics through a slightly non-traditional route. I was an electrical engineer at first, having studied at Stanford. And then through that process, really got interested in medical technology research. And this, light bulb just kind of went off. I was like, I think I want to go through the medical route and work on this stuff as a, physician, as well as an engineer.

And so I got my stuff together took a gap here and Applied to medical school and then throughout that process in medical school and, residency, got really interested in health, health it, and using medical data to improve healthcare delivery. So that was kind of a natural fit to go into clinical informatics fellowship after I finished my peds residency.

have you on. So, starting in:

I mean, a lot of people have seen the studies where notes in the United States are about three times longer Than other systems around the world that are on similar EHRs. So like in Australia, their notes are about a third, the length of our inpatient node in the US and a lot of that has to do with some of those prior regulations.

And so starting in:

changing in, healthcare.

Yeah, so pretty excited about this project. We've been revamping our note templates going division to division at our hospital.

And I wanna acknowledge the, team that's been working on this, cuz I, I joined as a fellow, but this actually had been in place before I, joined. And so just to take a moment to acknowledge that , two of the other pediatricians on the project Araj, Anno, Shri, and Rachel Goldstein. And then we have a great analyst, Julia Hahn, who helps us build our note templates and Rosalia Sandoval, who is our project manager.

But like you mentioned, the motivation for this project really came about because of those e N M changes. . But then also on the pediatric side, there was an additional motivation that came with the 21st Century Cures Act. And so as a pediatric institution, we were getting ready to share our notes with our patients, but our patients are children and so they have to have their parents involved in their care to consent to care.

And so typically at a pediatric health institution, the patient portal is configured so. The patient or the parents or the guardian of the patient have access to that medical record. And so we knew that if we were turning on our note sharing particularly for our teenage patients, their parents or guardians might be seeing those notes.

And at the same time, we wanted to share notes because no, I, I believe in open notes. Our institution believes in open notes. It also was mandated by the 21st Century Cures Act, but that there's a tension with. Along with the minor consent laws, which they vary state by state, but there are these laws that dictate that adolescent patients should be able to receive confidential care around certain sensitive health topics like mental health, reproductive health, substance use.

And so we wanted to make sure that the notes that we were sharing wasn't going to infringe on that privacy for those patients. And so the, those were the two motivations for this project. Because we had noticed that a lot of the note templates were pulling in parts of social history and medications, for example, that may hint towards some of those health topics and might actually infringe on that, on the privacy of those patients.

And then at the same time, We saw a huge opportunity to clean up our notes, make them more readable cuz we could get rid of a lot of that stuff that previously was needed.


one is on CSO priorities for:

We have Eric Decker within our mountain, Shauna Hofer with St. Luke's Health System out of Boise, Idaho, and Vic Aurora with Hospital four Special Surgery. And we are gonna delve into what are the priorities for security? What are we seeing? What are the new threat? What is top of mind for this group? If you wanna be a part of these webinars and we would love to have you be a part of them, go ahead and sign up.

You can go to our website this week,, top right hand corner, you'll see our webinar. And when you get to that page, go ahead and fill out your information. Don't forget to put a question in there. one of the things that we do, I think that is pretty distinct is we. like for today's webinar, we had 50 some odd questions that we utilized, in order to make sure that the conversation is the conversation that you want us to have with these executives.

So really appreciate you guys being a part of it and look forward to seeing you on that webinar. Now, back to the show. 📍


Yeah. We've had

some similar projects at the system level. we've been on Epic our EHR for, a decade almost.

And we've probably have not revised our templates since the very beginning. And so they're very outdated and there's a lot of fluff in there that's no longer required. and what I found was that after the wall changed, almost nobody adjusted their templates on their own. Maybe a handful, like, just our really motivated physicians might do.

But it was so easy just to continue with your old template that the really only way to get. Change was to go for me, I started on the outpatient side and I would meet with individual physicians and adjust their templates one-on-one. And so your approach to doing it department by department, I think is, really great as well we're starting to look at our inpatient note templates and finding the same thing where most people have not adjusted.

So tell us about How it has worked, going department by department. What I found on the outpatient side was, was some were really open to the idea of revising their templates and, took it up, and then some said, no, I'm kind of, happy where I am. Leave me alone. what's been your experience so far?

Yeah, I, think I can echo that. , we've got some groups where the adoption of the new note template is very high and some groups where it's still under 50% and, we've encountered Definitely a variety of opinions about note templates. Note templates I feel like are really personal or can be really personal, it can be a very unique part of the workflow and there can be a lot of heterogeneity there. But we decided to go division to division like you had mentioned, just because we thought it'd be faster. And we were trying to think about what is the right group or size of group to tackle this problem.

And like you said, I think individually going through and updating notes might get better adoption. But it would take a lot longer and. so we also are wrapping this up with the huge educational campaign where we are teaching providers about the adolescent confidentiality issues with sharing our notes, as well as the E&M changes.

So this is like an educational campaign plus note revamp all in one project. and the way that we've decided to engage with each division is through a core group of champions. And so we meet with the division chief first and set expectations for what this project is, what we hope to achieve, and then ask them to identify a few champions within their group.

And then we have working sessions with those champions where we learn about what their workflows are, what their needs. are what they use the note for. And then through those working sessions, we, build a note template for them that's based on a best practice template we have in mind.

And we just kind of modify it for what their specific needs are. And then we let it loose, have, an observation period where we wait for comments and feedback and then loop back again with the division. So, ,


the change management structure we've been using. There's some lessons we've learned along the way.

Sometimes the champions don't always represent the workflows That the broader group uses. Sometimes there's some, representation in the champion that doesn't represent like, the full group. So, we are learning about how to engage multiple viewpoints and keep the process efficient at the same time.

That's great.

Have you had any results You can. .

So we've done a few divisions mostly anecdotal results so one of my recent contributions to this project is coming up with a rigorous evaluation for our progress so far, so we can exactly quantify how, much of a difference we've made.

We've gotten some very positive anecdotal feedback from some providers who've shown us a bef there before note there was several screens long, and then the after note, that's only , two screens long or one screen long. Some metrics that we're looking at measuring is note length, but also read rates in our patient portal.

We're interested to see if the shorter notes. Engage our patients more to actually read them. Now that we've taken out a lot of the filler and then we're also u going to use signal data to look at documentation time. And we recently designed a survey, so we've given this to the last group that we worked with some before and after surveys that look at the subjective.

Experience of writing the note, because I think that's, really important. As you likely know, there's research showing that really it's that subjective experience of using the E H R that's probably more associated with burnout and wellness as opposed to the actual quantitative measurements.

So I, think it's a great if we can show that the notes are this many characters shorter or if providers spent like this many minutes less on a. But what I think really matters is what that note writing experience is like. Do providers feel like writing the notice faster, even if it's not that much faster?

And I think that goes further than the actual quantitative measurements. What you can get from signal.

No, that sounds like a great project and hopefully one I can mimic at, our institution. So tell me about what you're doing with alerts and, these pseudo randomized


The alerts I, I find that really interesting.

We just completed our first pseudo randomized trial of an alert and have started a second one. And we're trying to formalize how to incorporate this into our clinical decision support governance. But yeah, the basically. the way that we set this up was we had a discharge medication alert that we had created, and we wanted to evaluate whether that alert is actually effective or not.

And I think this is, a powerful method and it can be done natively in the E H R. We use Epic. . And so what the experiment we set up was that we had this alert that fires for specific type of discharge medication prescription, and I'll go into that in a little bit. But basically if the provider ID was even, they saw the alert and if the provider ID was odd, they didn't see the alert.

And then we could compare some process metrics between the two groups to see exactly how effective that alert.

And so you can do that natively within the, I guess the BPA configuration where you can say provider ID equals even versus odd, or would you have to create a special rule for that?

How, how did that work?

Yeah, so our rule A actually does string matching. We look at the last character of the provider id, and then if it's 0 2, 4, 6, or. That that's, so we made two BPAs and one of them fires if this discharge medication criteria are met. And then the second rule is if the provider ID ends in 0 2 4 8, then we made another BPA that has no visual component.

There's no interruption at all, and it has that same first criteria. And then the second criteria are if the last character, the provider ID are 13 5 70.


And so, you've done that for the discharge medication. Have you seen any

results so far?

Yeah, so , what we were trying to do is reduce the amount of free texts that was being used in our discharge medications.

And the few reasons that , we were interested in this problem are. . Well, when you use the free text as opposed to the discrete computerized provider order entry fields, it can bypass some of our clinical decision support tools, like dose range checking weight-based dose checking. And so that's one issue, but there's also a health equity issue in it as well because we employ a automated translator of the prescription.

And that only works on the discrete order entry fields. Mm-hmm. . So when a provider free text enters in the prescription sig, it cannot be translated by that system. Like it can be if there were discrete fields used. And so we wanted to increase. The rate of translation of our prescriptions for our patients who prefer a language other than English.

And also we want always improve our medication safety practices. And so those are the two motivations for this. We employed this alert to fire when a provider was writing a discharge prescription using free text instead of using the discrete fields, and then it actually fired and interrupted the workflow if their ID was even and, didn't show up, didn't interrupt the workflow if their ID was odd.

And we looked at a few measurements, but the main one is just how many encounters ended up with a free text prescription at discharge between the two groups. And we found that in the group with the alert, it was a little under 10% less. So like in the mid 80% versus mid 90%, which was actually not as low as we had expected.

So we found that the majority of the users were still overriding the alert. And, still 80 some percent of these prescriptions were being sent.

tuations a family can face in:

At this week, health, we're going to give back. We are partnering with Alex's Lemonade Stand all year long. We've got a goal to raise $50,000 from our community, and we are already up over $12,000 and we are asking you to join us. There are two ways that you can do that. One is you can just hit our website.

There's a banner at the top that says Alex's Lemonade Stand. Click on that and you can give directly at that lemonade stent. another way that you could do that is we have been doing drives and we've been doing drives all year long in January and February. Our drive for March is something that the team came up with and I'm really excited about and it is, we are going to vibe.

With the team and we're bringing captain. Captain is my producer's service dog and Captain will be with us for the entire event. You're gonna see us around the event doing interviews, and here's the drive. The drive is get your picture taken with Captain you and a bunch of your friends with Captain. Get the picture taken, go ahead and post it on social media and Twitter linked.

And tag us at this week. Health. When you do that, we're gonna count the number of people who are facing the camera in the picture, and for each person in the picture with Captain facing the camera, we're going to give $1 to Alex's lemonade stand. So with your help, we hope to raise a bunch of money for childhood cancer at the VI event, and we hope that you'll be a part.

It's gonna be exciting. We have some partners that are a part of this with us and we really appreciate them stepping up to help us fund this. We hope to raise a ton of money. Shore Test is a phenomenal partner. Order is another one. And Artis site, great companies, great products. Check them out and we hope that you'll participate with us and we look forward to seeing you at the Vibe event. Now, back to the show.


  📍 No, I think that's really good information and a good way to go about it. I get asked for new alerts all the time to try to improve a quality measure, try to do X, Y, and Z related to patient safety or something else.

And what we do currently is we run it in the background to see how often it fires to try to get a picture of whether or not it's gonna be a nuisance, alert or not, and tweak

it. ,

but we don't have a great way of showing whether or not it was effective in achieving the desired outcome versus kind of standard of care.

So, I think that's a great way that we might adopt going about evaluating new requests for BPAs to see if it's worthwhile. We can show, hey, look, this had a 10% effect. Is that enough? Or is there something else we could try besides an interruptive. .


Yeah. And another, I think power of this method is that it can help you tease apart confounders.

Particularly in, I think in, in quality improvement. This happens quite often. We kind of throw the kitchen sink at a problem because our objective is to solve the problem, right? It's not to generate research about the problem. , we're trying to fix this operational issue around our medication safety.

And so at the same time, there are actually other. modifications we were making to the ehr, we were adjusting some of the order entry field options. We were updating some of the preference lists, and this is happening at the same time as the alert was employed. And I could see a situation where an alert is proposed as part of that solution.

It goes into effect, but actually another intervention was more effective and it makes it look like the alert was effective and that. alert Perpetuates. But this method allowed us to see that, you know, and we did see that both groups towards the end had slightly lower rates. And it helped me at least that was a sign that, hey, maybe it was another intervention that we did that was driving these rates down.

Because when you actually compare the groups, there wasn't that big of a difference between them.

No, that's great. And I really do think that that approach should be standard, going forward is, definitely. Do AB testing to see is this thing actually working before we put into production versus what we're all doing right now is going back, years later and finally cleaning up all the alerts that have gotten put in over the years.

Like I said, we've been on for a decade and. A new alert every, month or so, adds up quickly. So, that's, that's a great approach. Thanks so much for, for coming on the program. Any, last words you wanna leave with the audience?

Well, I'll, I'll say one more thing about the alert.

I think it, it can be This is something we're thinking through as we formalize the process. So I hope that other health IT leaders can, also think about this and problem solve. But we, we decided to, basically pseudo randomize on the provider id. But , there are so many ways that you can do this AB testing and.

At least for now, , we're kind of deciding on a case by case basis, but you may have seen some examples where the randomization is done based on the patient mrn, for example, or the clinical site. Mm-hmm. , any of these variables that you can access in the B p a criteria essentially are fair game for randomizing, some sort of intervention, and there are pros and cons to these different approach.

For the provider stratification. If you have two providers taking care of the same patient, but one of them happens to be in in one group and the other happens to be in the other group, you can get a little bit of contamination. But then on the other hand, if you stratify by patients and you have a provider taking care of multiple patients that are in the two different groups and they see the alert for one patient but not for the other, it still might affect their behavior taking care of the other patient.

So there's contamination there as well. So it. There's a lot of thought. I think that goes into setting up these projects. And what we're trying to do is formalize it so that we don't have to create a whole research project every time we want to take up an alert. And that's something that we're working through creating that standard process that makes it easier to set these types of experiments up.

And we're doing our second experiment now that's actually run on the patient m r n. And so I think we'll learn some lessons from that as we try to come up with a way to incorporate, standardize this into our, our CDs. Mainten.

No, I think that's great and it would be great actually if, the EHR companies would build this natively within the app, so you don't have to do these behind the scenes rules.

But you know, maybe we can get there in the future. Well, thanks again for coming on, and thank you everybody for listening. Have a great day.

gosh, I really love this show. I love hearing what workers and leaders on the front lines are doing, and we wanna thank our hosts who continue to support the community by developing this great content. If you wanna support This Week Health, the best way to do that is to let someone else know about our channels. Let them know you're listening to it and you are getting value. We have two channels This Week Health Conference and This Week Health Newsroom. You can check them out today. You can find them wherever you listen to podcasts. You can find 'em on our website this, and you can subscribe there as well. We also wanna thank our show partners, MEDITECH and Transcarent, for investing in our mission to develop the next generation of health leaders. Thanks for listening. That's all for now.



More from YouTube