This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

For choosing to invest in our show. My name is Bill Russell Healthcare, CIO, coach and creator of this Week in Health. It a set of podcast videos and collaboration events dedicated to developing the next generation of health leaders. If you're not familiar with our show throughout the year, we do a new show every Tuesday, Tuesday News Day, where we take a look at the news that's gonna impact health it, and every Friday we do an interview with an industry influencer.

A couple times a year. We forego this pattern to cover events this week. Health events are shorter interviews conducted from the floor. We drop a bunch of these episodes during the event week and let you pull up the ones that you find most interesting. Even though Chime and Hys are not gonna be going on this week, we're gonna be honoring our interview requests and we will be bringing you a bunch of shows.

So we're gonna be dropping a bunch of shows this week. This week, health events literally has as its mission to bring the event to those who are unable to attend. And this year that just happens to be everyone. Our first interview is with Joe Petrow, the CTO of Nuance. Uh, we visited with Joe last year in the booth.

And he introduced us to a lot of concepts that they were working on around ambient clin clinical intelligence. And it was really exciting and I thought it was one of the best, uh, presentations at the show last year. And so we decided to catch up with Joe again this year to see what kind of progress they've been making on ACI and to, uh, explore some of the other topics.

Thanks. Thanks for the invite. I appreciate it. Well, I'm, I'm looking forward to this conversation. Last year was really exciting. Uh, we met at hims and, uh, you know, voice, you guys were, were showing off the ambient clinical intelligence and listening. Booth, and I thought it was one of the most exciting things that was going on at the show.

Um, give us an idea of, let's start with, you know, what, what will you be showcasing this year at hims? Yeah, sure. Um, you know, at this point we're still being a little bit, uh, vague on it for, for obvious competitive reasons, but, you know, at a high level, we're basically showing the next generation of, um, of that demonstration that you actually saw last year.

So. We're coming out with a, um, a general, uh, generally available version of the product here in the next month or so. And so that's what we'll be showcasing. It's going to be, um, you know, at a super, super high level. It's gonna be a lot less structured, uh, a lot more real world. Uh, the audience is gonna participate a little bit, so it's, um, it's exciting.

Uh, in the patient room seems to be sort of the hottest thing going. Uh, talk about what, what you've learned over the past year. So you, you've put it out there in pilots. Yep. And, you know, what are you learning or what are you finding as people start to interact with the technology? Yeah. We've got, we've got a number of, uh, uh, clients that I would call early adopters.

Uh.

I think what we're basically learning is that this, this is spot on in terms of the, you know, hitting the center of gravity of the pain point. You know, that physicians are feeling what we're, what this whole thing, what ACI has really targeted at is attempting to remove the screen, you know, between the we, between the patient and the physician.

And, you know, for a long time now we're dragging medical one we've always said. We, uh, we turned the chair around. There's this famous crayon drawing that's been in, um, in jama, which a, a little girl actually drew of, uh, her. Her sister, who was very sick, um, and, uh, she drew a picture of her, of her sister kind of sitting on the exam table and her mom and her other sibling sitting next to her, but with the physician turned around sitting in a chair typing on a computer.

Uh, and that's really what this is targeted at. We're trying to turn the chair continuing, turn the chair around, kind of extending on the Dragon Medical one, um, you know, proposition of, of relieving physician burden and trying to bring back some of that joy of care to the, to the physician and. From a patient point of view, you know, improving the experience for ourselves, uh, ourselves as well.

So that's fundamentally what it's, what it's targeted at, and it, it seems to be spot on and it, you know, the, it's been overwhelmingly positive. The feedback that we've gotten from the, uh, physicians that are on the system. Are, are you learning anything in, in regards to, uh, you know, dragon Medical one is, you know, microphone pretty close to the physician generally in a, a quiet area.

When you're doing this in sort of a room setting, you're probably learning, I would assume a ton of things in terms of, uh, positioning the microphones, picking up the video, picking up, uh, visual cues and those kind of things. Are, are you learning anything in those, in that, in those areas? Yeah, there's, um, a number of kind of mundane things that are, it's just important to kind of work through it, you know, as.

We're in the ropes, of course. It's a, it's a noisy environment. Um, so we're, we're pushing the state-of-the-art in terms of what we call signal enhancement and noise filtering. And we've, we've leveraged technology that we brought over from our, our other divisions. Um, you know, some people don't realize it, but you know, new nuance has been in the automotive space for a long time.

If, if you've got a car and you're talking to what the odds are, you're probably talking to, uh, to 65% of the vehicles out there. Um, and that technology has allowed us to really advance the state-of-the-art in terms of signal enhancement and noise filtering because of road noise, as well as, um, wind noise, as well as kind of inter-party noise where you've got kids in the background kind of yelling and so forth and you're trying to control your car.

Um, so. We always expected that that was gonna be a challenge because it's, it's a noisy environment. It's a multi-party environment. You know, you have a doctor, you could have a number of nurses, you could have a number of FA family members. So those kind of stuff that we actually expected. Um, there's some more mundane things like, you know, one, how we actually, um, have the patient opt into the experience is important.

You know, that what, what happens is this is fully disclosed to the patient so that the patient understands what's going on and understand. How this is actually gonna help their experience. So how you do that, you know, what forms you put in front of them, um, you know, what signatures you get from them and how you actually explain it.

This kind of an art form to that. Of course that rests largely on, um, on our clients. Uh, not so much on nuance, but that's an important step, uh, because with all the crazy stuff that's going out there in terms of privacy and so forth, we wanna be very, very clean on that. So we've learned a lot there. And our opt opt-in rates are still very, very high, which is awesome.

Um, because that's, that's what we want it to be. 'cause we do really, truly believe this is gonna be good for everybody. Um, there's other little things, like there are times during the episode of care where you don't want the machine listening. It might be a very, uh, intimate conversation that's going on between the patient and the physician.

And the patient might say something like, you know, can we shut, can we shut this down for.

You know, you kind of maintain the context of that. So when you come back in simple stuff, like how do you make sure that all of these various kind of snippets of, you know, audio and so forth are kind of stitched together. Seems like a simple, mundane problem. But, you know, doing that in a way that, you know, maintains the continuity of the, of the dialogue, um, so that we can make sense of it.

You know, those little things like that. Um, but there, there haven't been a lot of, you know, very big discoveries. This is a hard problem. We, we know it's hard. We kind of walked into it knowing it's hard and, uh, the problem has been actually nicely yielding, you know, over the course of the last, uh, last year.

Yeah. You know what I'm, I, I'm hearing the industry, I, people are ecstatic about Dragon Medical one. The people who are using it, uh, you know, cloud-based solution, it's, they are, uh, the physicians are figuring out, has moved pretty. Pretty far along, pretty rapidly compared to what I was looking at maybe six or seven years ago.

And, uh, I mean, I'm hearing really good things. Your accuracy rates on that are, at least, I, I think they are hovering in the high nineties, 96, 7 8%. Um, are you able to get those same kind of accuracy rates in, in, in the, uh, uh, you know, the, the ambient clinical environments? Yeah. Yeah. So, um, lemme just explain.

So the answer is no, not yet. Uh, and this is part of the, the problem yielding, uh, to us. But lemme kind of explain how we're, how we're creeping up on this. So, I joined Nuance about 12 years ago or so, and when I, when I got here, the out of the box accuracy on on Dragon was, was okay. It was like 88, 80, 80 9%.

But the fact is. You know, 88, 80 9%, that's still one in 10 words, you know, that you need to correct. So even though it, it's, it's, it would be a good grade on a test maybe if you're in college. Uh, not such a good grade, if you're stream, if you're streaming speech. Right. And there is no scale here like there is in college.

Um, and over the years we made that yield at. What we call 15% relative error, uh, word, error rate per year. We basically chipped away at the problem, and now as you say, it's extraordinarily high. Basically, we've reached a level of what we call human agreeance, which fundamentally means that if we have a human being listening to the conversation and we have the speech engine listening to the conversation, the speech engine and the human are gonna agree in terms of, uh, what they heard.

A, um, number one, conversational in nature. So it's, it's sloppy the way humans are sloppy in terms of the way we speak back and forth to each other in conversation. Versus if you look at Dragon Medical one, dragon Medical one is a very controlled, formalized, um, form of clinical documentation delivery. So physicians tend to speak in complete sentences, complete thoughts.

You know, they're trying to do like a soap note. They're trying to do a progress note. They know, and they're kind of pre contemplating and premeditating what they say. And so, you know, patient presents with complaints of severe headache. Patient is on 150 milligrams of X, Y, z. Patient has history of headache.

Basically a very kind of formalized prescriptive type of a speech. But when we're talking back and forth, if you are the physician and I'm the patient, it's sloppy. We're talking about the kids, we're talking about the kids' games, we're talking about, you know, the score of the last football game. We're talking about some ing, my knee.

It's very sloppy. It's back and forth. There's lots. By its very nature, conversational speech has a lower level of accuracy than that formal speech. That's one, that's one step kind of down in terms of accuracy level. Um, then we have to diarize the speech. So you have to take what you said and what I said and separate it so that you can track both, both of those kind of threads, and that is what you apply the speech to text to.

You apply it to the patient thread, you apply it to the physician thread or multiple physicians and possible family members. There's another accuracy rate that applies to that. And then once you turn it into text, you start to harvest it for information so that you can summarize it, because that's what ambient clinical intelligence does.

It summarizes the conversation into clinical documentation that gets embodied in EMR. There's another level of accuracy there. So the challenge with ACI. You might be multiplying several high accuracies together, but there's this inertia, this pull towards a lower level of accuracy. So the question is, what level of accuracy do we need inside of ACI so that the physician accepts the result and they just want to edit it.

Okay, so it's easier for the, for this thing to listen and then for the physician to look at the summarized result and actually edit it. Will they be willing to do that? That's a. We discovered this tipping point in the old days with transcription because, you know, we went on this journey where transcriptionists were involved.

They would do the typing for, uh, for physicians. And even with transcriptionists, we knew we had to get to a certain level of accuracy. The magic number where the transcriptionist were kind of a relatively lower paid person, lower paid knowledge worker than the physician happened to be 85%. Once we got to 85%, it was a lot faster to edit it than it was to type it.

Even though it was 15 words a hundred, and what we learned in that process is we learned that formatting the document was actually far more important than the speech accuracy. And then as speech started getting higher and higher and higher, then we flipped it over to the physician. And when we started, when we kind of hit that trigger point of 88, 89, 90 position started wanting to edit.

We don't know yet 'cause we don't have any experience. We have a bunch of opinions 'cause we're highly experienced in this field. We don't really know yet where that trigger point is and, and that that is what we're basically, uh, working on. But the problems yielding nicely from a technical point of view.

Accuracies are continuing to go up. You know, we, we we're not hitting any blocking points at, at this point in time. So it's really, really exciting and we're kind of looking forward to going through the same type of journey. Uh, and I appreciate you going into that level of detail. As you know, the listeners for the show are primarily health IT professionals, and that's the level of detail they, they really, uh, wanna understand.

'cause the hype around this is amazing and you guys are tackling the hardest area to tackle. And you know, so people are saying, Hey, you know, we could put, uh, you know, we could put a, a Amazon Echo in the, uh, patient room and we get this. Yeah. But you're just asking for a blanket or asking for a nurse to come up.

That's a completely different level of accuracy that's required and, and risk tolerance, quite frankly, that's going on. Um, so you guys signed a, you made an announcement with Microsoft earlier this year. Uh, is that about tapping into AI capabilities or what, what is that about? Yeah, that's a super exciting opportunity, um, for, for us.

So just to tell you a little bit about the history. Um, Microsoft approached us like two and a half years ago at, at uh, and Peter Lee, who runs their healthcare business, um, had approached us. He came, he, they came to me and they were talking about like different ways that we could work together. And you know, to be perfectly honest, whenever kind of a big platform player approaches you, you're um, you're not exactly sure why and you're a little bit skeptical.

Uh, and, and so we had a healthy level of skepticism in the early days. But what ended up happening just to kind of fast forward through a, a multiyear journey. We started to realize that Peter's team and Microsoft legitimately were trying to get into the business of improving the situation in healthcare and, and really improving the situation with the, with the, um, physician, uh, in particular.

And we talk about all different types of, all different kinds of ways that we could potentially work together. Um, you know, we've got something called the AI marketplace, which is a, a diagnostic and radiology. So we can, we do some image processing, we've. AI image algorithms for the detection of certain disease conditions, so forth.

So we looked at a whole bunch of stuff, but we kept coming back to the ACI problem because they had a project called Empower md, which was similar to ACI, a little bit different, but you know, the spirit of it was right. We started to realize they're thinking about the problem the same way we are. Um, and we realized that if we could create a partnership that was a legitimate partnership, it wasn't just some kind of like quote unquote Azure licensing deal.

But this could be really good for us 'cause it could help us accelerate, um, our ACI efforts, but it could also, you know, get Microsoft into the market as well, the way they, they desired. And so this kind of spiritual alignment between the two companies and the somewhat friendships we developed with Peter's team, that's the thing that kind of fueled it.

And so what the partnership is all about is it's about bringing the, uh, the net horsepower of Microsoft and the way we think about Microsoft as an enterprise class. Uh, r and d company and solutions company, bringing the full horsepower of them in terms of ai, their research, um, all the various things they have available in the market, as well as privately marrying that up with the nuance experiences and history in terms of, um, clinical documentation expertise, our partnerships with the EMRs and so forth, and attempting to accelerate the AC problem.

Complex and difficult deal to, to contemplate and think about for two, you know, uh, companies like ours that, that both own our, our territories, but we got through all that. It had support all the way at the very top CEO level at Microsoft as well as the CEO level on our side. So, um, it was hard to do, but we think it's very much gonna be worth it and it will help accelerate us.

Yeah, and I

expect. On voice and really conversational technologies is that you are going to alleviate the clinician burnout and the clinician problem. I checked my 87 year old, uh, Father-in-Law into a hospital last night, and there was literally, uh, in, in the EE, uh, we started in the, uh, ed. And there's literally a person dedicated to the keyboard.

They were sitting in front of Epic, they were doing all their stuff and, and then nurses and doctors came in and out and I just, I, after a while, I just said to her, I'm like, Hey, if you don't mind me asking, you know, what's your, your level? And, you know, she's, she's a nurse practitioner, but she really hardly ever left the keyboard because it was required.

I mean, somebody had to put all this information in and there was no other way to get the information acceleration of. Uh, you know, is really important. And last year we talked about, you know, you're doing this in orthopedics, but in order to accelerate this, you almost need new vocabularies to move forward.

Um, are we going, are we gonna be able to, uh, accelerate this over the next 12 to 18 months, or is this gonna be a more of a five to six year journey kind of thing? Yeah, and this is where I could easily fall into the hype cycle, so I I won't do that. Um, you know, the, the reality is this problem's gonna yield over time, and I think it's gonna yield in pieces, right?

So the way we're doing it is we're, we've already got over 1.5 million, uh, cases of, of, of care in, in our AI training library. And that spans, you know, five different specialties. So we talked about ortho last year. We're also attacking, I'm just reading off a list here, podiatry, dermatology, ears, no ear, ears, nose and throat, as well as ophthalmology.

Um, so I think that the data collection piece of it, for us at least, because we know how to do this, we've been doing this for a long time, it's kind of the easier part of it. The vocabulary part of it is also the easier part. The hard part about this is. You need to be an expert in each one of these specialties.

Like, so for example, I just went, did one of these executive physicals a short time ago, and I went through several different care settings. So I, I dealt with a dermatologist, I dealt with, uh, an ophthalmologist, I dealt with a general care person, cardiologist, et cetera. And if you look at their notes and you look at the way they behave in the treatment room or the exam room, everyone is different.

Like the ophthalmologist hardly said anything to me, right? So. They were just doing measurements. They have the thing on your face and they're, they're making the adjustments. They're looking in your eyes and so forth. And he really didn't say anything until the very end, and he barely summarized it. I mean, he didn't summarize the measurements for me, so he, he barely verbalized.

And so the ACI experience in the ophthalmology setting and what we capture is gonna be different than say the dermatology setting when I was in dermatology. This physician actually had a scribe. The scribe walked in and she sat behind a curtain at the back of the room and the ophthalmologist came up and exam, examined me in everything he pointed to on my body and on my skin.

He described it, he described it in dermatological terms, but he also summarized it for me in kind of layman's terms so I understood what was going on and I, whether I should be worried or not, and each one of those care settings was different. So the . The finesse here, or the spec man ship or the art form to this is going to be each one of these specialties we encounter trying to figure out what has to be in that summarized document, what has to go from the summarized document to the electronic medical record.

And so in that respect, it's brand new territory. Um, and that this is where the subject matter expertise and the specialties come. And, you know, this is kind of how we're chipping away at it, Joe. I know we're at the end of our, have one more question. Yeah, yeah, sure. Uh, last question is, um. We, we've, we've focused a lot on storing the information in the EHR.

Are we starting to look at storing the information elsewhere and making it available for research and, uh, looking at training and those kinds of things in, in other, uh, I don't know, technologies, data technologies? Yeah. Um, there are companies that are looking at that. We, we tend to, um, operate in a very, very narrow band around the uses of data.

Um, we take a very conservative approach. We've basically . Labeled ourselves as stewards and custodians of the data, and we have a very clear conversation with our clients that we're borrowing the data for a short period of time so that we can train our model so that we can deliver a, uh, a very specific service back to you.

We're not gonna derive value from the data. We're not gonna try to resell the data. We're not gonna try to build products that you're not aware of. So we have something in our contracts called the data usage clause that really, really limits. Our scope. Um, with that said, there's a lot going on out there, some of it a little spooky, you know, as consumers, um, you know, uh, just in terms of the way our data is used and our health data is used, it, it's not making me very, very comfortable.

And I think there are companies that are starting to, you know, get into the business of potentially having secondary monetization of, of healthcare data. Um. You know, if a company tries to crack it, um, it'll probably end up being a partnership with one of these, you know, big IDNs or whatever, trying to come up with a way to, um, you know, anonymize the data and then create kind of a honey pot type of a, a resource for AI training.

Um, that is not the business, you know, that we're in, and we've kind of intentionally tried to stay away from that. The other thing is, is that the EMR for us is very central, uh, and it, it, it makes our world go round. So we have a very open relationship with the EMR and we try to push everything back to them.

Joe, thanks for your time. I will, uh, I, I, I'll look forward to stopping by the booth at hims. I will, uh, likely air this prior to the HIMSS show so that, uh, people will get a, a feel for what you guys are gonna show, even though you were kind of cryptic with us early on. But, uh, I understand the marketing aspect of that.

People standing over here off camera. By the way, that.

You. I really wanna thank Joe and, uh, the team from Nuance for making 'em available for this, uh, interview. A lot of great things going on in voice conversational technologies, ai. And, uh, it's exciting to hear how focused they are on making that a reality. Uh, we want to thank our founding channel sponsors who make this content possible.

Health lyrics, Galen Healthcare, Starbridge Advisors, VMware, and Pro Talent Advisors. If you wanna be a part of our mission to develop health leaders, go to this week, health. Dot com slash sponsor For more information, if you wanna reach me, you can always shoot me an email build this week in health it.com.

I love your feedback, feedback on the show. Um, I've, I've a couple of you have sent, uh, ideas for future shows, people I should interview and those kind of things, uh, keep the feedback coming. It is extremely helpful. Uh, this show is production of this week in Health It. For more great content, you check out our website this week, health.com or the YouTube channel as well.

Thanks for listening. . That's all for now.