Brian Lancaster on Getting to a Single Pane of Glass in the DC
Episode 18728th February 2020 • This Week Health: Conference • This Week Health
00:00:00 00:35:54

Share Episode

Transcripts

This transcription is provided by artificial intelligence. We believe in technology but understand that even the smartest robots can sometimes get speech recognition wrong.

 Welcome to this Week in Health, IT influence where we discuss the influence of technology on health with the people who are making it happen. Today we get very practical with Brian Lancaster, VP of IT at Nebraska Medicine. My name is Bill Russell Healthcare, CI coach and creator of this week in Health.

It I set of podcast videos and collaboration events dedicated to developing the next generation of health leaders. This episode is sponsored by health lyrics. I coach health leaders on all things health. IT Coaching was instrumental in my success, and it is the focus of my work at Health Lyrics. I coach CEOs of health systems, CIOs, uh, CTOs, startups as well.

want to elevate your game in:

Yeah, we, um, really went all in on kind of an open space concept, um, really first geared towards, we're running outta space, but it also has helped us with, um, collaboration and, and communication, things like that. It's interesting when you, when you take in an IT organization into that environment, how have you found they react?

I, I'll tell you my story after I hear. Because my story's not a good one. ? . Yeah. So, um, so that's an interesting question. The, um, we, we were all over the place from people who were excited, people who really didn't care. They'd come do the work, whichever environment they had, and then also people that were really concerned.

And, um, unfortunately I think the people who were really concerned were the most vocal. Um, so we did a lot of storming, norming, and. Uh, forming. So meaning we had open, um, um. Sessions where they could talk about their concerns and, you know, everything from, wow, I'm gonna have to talk to people I don't like.

Um, which is probably something that we need to work on regardless of the space, um, as well as to, you know, stinky food and stuff like that. Um, so we had some concerns up front. Um, we also coupled the space with a pretty, um, aggressive work from home, um, policy. So kind of adopting the mentality of work is what you do, not where you go.

Um, so we, we really have this, um, interesting, um, kind of outcome in terms of people who need heads, downtime, can go work from home if their environment is conducive to that and work at another university location for heads, downtime. We also created kind of focused areas for heads, downtime, and then open areas really centered around teams and, and after the implementation.

I think we've had great results and it's something that the rest of our organization is looking to adopt. Um, for basic reasons around just run out space. Yeah, I'll tell you as, as ACIO, the, the thing that I got in the most trouble for was always moving too fast and not laying the foundation. And so I come into healthcare and I notice that, um, the, the first thing you notice, it just sort of cracks you up.

We have this four story building for it. And all the offices are on the outside of the building, so, you know, the, the cubicles are on the inside with no light. And then all the, you know, anyone with a title of director or above has taken all the light away from the staff members. And I thought, man, this is horrible.

This, you know, clearly we could do better. So we took one of the floors, we took everything out. I mean, completely gutted it, opened it up. I thought people would be clamoring to get to that space. It, it's change, you know, people just naturally struggle with change and uh, and all the things you sort of, uh, talked about.

It's like, I've never worked in an open environment. How does this, you know, how do I get work done? Do I put headphones on? Is it rude to have, I mean, we have to answer, we have to, uh, walk people through it before you just throw 'em into it, I guess. Yeah. Yeah, absolutely. And we found it's really an individual choice.

What works for me may not work for you. So how do we allow for people to find out, you know, maybe the open space or working from home doesn't work so we can pull 'em in and give 'em appropriate space. Um, so we've been learning that as well. Yeah. But change is always the, uh, um, you know, I was, I like you.

I, I love change and I move probably faster than I should. And, uh, you know, I think most people just get concerned with, with change. We, we had some individuals who were coming to the same cubicle for 40 years and, you know, they thought that cubicle was theirs. And, and now they, they see that it's, what they do is so much more than that space.

So that takes time. Well, I wanna, I wanna get nerdy on this episode with you. Uh, we had a prior call, um. But before we do that, um, Brian, tell us a little bit about, uh, Nebraska medicine and a little bit about your role. Yeah, so, so we're located in Omaha, Nebraska. Um, and basically we support the surrounding region with our two hospitals and 40 clinics.

Um, so roughly at eight, 800 beds or so, um, $1.8 billion in, in, in revenue from an active health system. So we're not, certainly not the largest, uh, organization. Um, but Nebraska Medicine is really the, the clinical enterprise. So with our, uh, research and educational partner at the University of Nebraska Medical Center, um, we share a mission, which, which I think is pretty exciting and, and pretty inspiring.

Um, we, we aspire to lead the world in transforming lives to create a healthy future for all individuals through our premier education. Innovative research and the quality of care we provide. Um, so we're really known for quaternary care, so specialty care. And, um, right now we are getting some national attention for the coronavirus.

So we have one of the largest biocontainment units, um, in the country. And we are, um, basically just received the, the patients coming in from the, the cruise ship. Um, so we. Quaternary care, the, the infectious disease, um, transplant, um, cancer, et cetera, those types of things. My role is I have IT responsibilities for both the clinical enterprise and the academic enterprise.

Um, so it's uh, an interesting role 'cause I get to go from really three different businesses from clinical research and educational. Yeah, that's a, that is a fun role. I, um, you know, I love talking to, here's what I've found. I, you know, I, I do talk to some of the larger health systems and they, uh, you know, they're pushing the envelope in a lot of different areas and whatnot.

But I'd like talking to, uh, people at your size organization, you tend to be, uh, a little bit more practical, a little bit more pragmatic. Uh, I, you know, I used to say that I would go to the . At my house system, I would go to the data center and, uh, you know, my card key, my swipe key didn't work at the data center.

They wouldn't let me in without a, without an escort as the CIO. But you know, as we get to, you know, certain size health systems, you have to be more, uh, in the weeds with a lot of this stuff. Um, clearly you can't be in the weeds with everything, but that I, I, I like that and that, the other thing I found is there's just a lot of 1 billion to $3 billion health systems out there that share a lot of the same problems.

So I'm, I'm looking forward to this conversation. Um, I, I hope we get a little nerdy on this. We'll see, you know, we'll see where it goes. Um. Uh, you know, so the, uh, you know, we met a while back, but what happened is I was sitting in an executive briefing with VMware and they showcased a solution that was so cool that I asked if I could talk to the system that was doing the work, and they put me in touch with you, which was awesome.

I, it was essentially. And, and, you know, this is the direction I want to go. You know, what's, what's cha what's changing in the data center? What's changing, uh, in how we deliver the, uh, services to the desktop? What's changing in the way . That we look at security, uh, and, and I was, I was sitting in this executive briefing and they, they showed me how, um, I guess it was you guys, so how you were closing down at the, the attack surface of the ports of legacy applications and even the EHR that we oftentimes don't even think about.

But, uh, but they're very real and, uh. And, and I, I looked at that solution and I said, I want my listeners to hear about that, uh, hear about that work. So can you give us a little background on that? Yeah. So, um, great question. So, so really this gets into our story around, um, virtualization and, and like any organization, we started with, uh, compute and server virtualization.

Um, and, and actually we had a, a large project child 16, to move from one data center to, to a brand new data center. And, and one of our intents of that project is really to adopt as many automation or virtualization techniques as possible. Um, and, and really as part of that, that effort, we went from a handful of virtualization to roughly about 70% server virtualization.

ove a lot easier. Um, then in:

So logical switches, routers, firewalls, load balancers, VPNs. All within that connected workload. So it's, it's using software to, to deliver the networking resources. Um, they're promatically created, provisioned managed, and, and it really creates a lot of opportunities. One of those opportunities is referred to as microsegmentation, uh, but there are other use cases too, so, so I'll talk about microsegmentation a bit, but in my opinion, network virtualization is a key component to having a cloud strategy.

Um, why is that? Because it, it also supports stretch layer two. So in essence, I can take a workload and move it from one data center to the next without having to re IP it. And if you've ever done this type of work, you realize having to try to manage all this stuff gets very complex between two on-prem data centers.

Why this ties to a cloud strategy is if I can virtualize all that and use software to manage it, and I can do stretch layer two. I can not only move it from one of my data centers to another data center, but I can also move it to Azure or AWS or your cloud provider as, um, as, as, as you see fit. And to me that creates elasticity.

So I don't have to get tied to the terms of AWS or Azure if I consume those from a, a native perspective. Um, so hopefully concept makes sense. Um. If get the concept, I think you can start to see how it could help with micro, which I'll get into in a bit, but I'll pause. Yeah, no, that really makes sense. You know, we talk a lot about, uh, vendor lock-in with regard to cloud strategies.

And we talk about, you know, one of the primary things you're trying to get from a cloud strategy is agility, right? The ability to move out, do something, move it back, or, um, or to move away from the old SunGard model of. For disaster recovery and business continuity of buying all these resources that I, I don't know if I, I don't know if you remember the SunGard model, but you used to used to pay every month for these resources and they didn't even guarantee that they would be available.

Emergency. Yes. Which, which always made me scratch my head, but, but that agility is really what you're paying for the ability to move things around. The virtual network is, is, I mean, to not re IP back, back when I was doing this, we still had to re IP a lot of stuff, and so over the last four years, this has really, uh, really moved forward pretty, pretty much since, since I stopped playing with it.

Yeah, I mean, if you, if you get into networking, it's, it's really complicated to try to track the flow from an application server to a web server, to a database server and do. 8,000 different systems. So if you're not using software, you know, you're bound to have in inefficient processes and create error and, and error results in downtime.

Um, so with that backdrop, if you want, we can go into microsegmentation. Um, absolutely. So, so microsegmentation, um, basically was, was always a thing, right? So meaning how do I take a system and make sure I only have enough, um, access to that system as, as possible? You could do that on a manual basis. Um, and in fact, we, I was just having an argument with one of our, um, computer scientists, um, faculty members who still teaches a lab today.

On how to build, um, a micros segmented, um, system manually. And, and I, I was telling 'em that that doesn't make any sense. Anyone who would do that, um, would use software to do that. So you should be teaching network virtualization because in essence that manual process just wasn't operational feasible. Um, because you would have to, um, look at each packet that leaves the virtual machine.

Transverse the network. Evaluate it, enforce it to make sure you have a choke point within the firewall, and you're not gonna be able to do that at scale. Um, so it really wasn't feasible. So now with network virtualization and the combination of hypervisor based and kernel distribution, firewalling platform based automation, we can segment it, um, using.

Software using virtualization. And, and really to do it, you have to have detailed network visibility and, and really understand what these packets are doing. And to be honest with you, we, we didn't really understand that. And again, that has clear impact to productivity. It also has a huge info security issue, so we don't know what is.

Our systems are connecting to how they're using that data, and we could be exposed for, for a threat. It also simplifies underlying physical networks, meaning now we know exactly the, the things that are connected to our epic environment. Um, now we can make sure we lock down everything else. And, and it's also really designed for automation.

So how do I take a, a, a new system, make sure I understand what it connects to, and automatically deploy the network components for it. Um, and then obviously it's implied that info security benefits of this, but forensics, we have richer context, better troubleshooting. Um, in essence, we know this port has never been pinged before.

Now we, we know that there's activities and volume of activities we can turn it off or, or, um, investigate it. Yeah, no, it's, it's, uh, it's in, so I'm gonna dig a little deeper here. I mean, the, the two things I loved about this were, uh, the visibility and, and the automation. And, uh, you know, that the argument, as you say, with the, uh, with the professor.

You know, that is the way that a lot of health systems are still doing this. They're still manually setting up all this networking, and they're still manually setting up at systems as well, and, and, and it, it doesn't really scale. And so when you become, you know, when, when you go to add another health system or acquire a health system or grow in that way, now all of a sudden that really comes to light because now you're trying to do those manual processes.

Across, maybe even double the size as we're seeing in, in some of the mergers in Chicago and, and, uh, in other places. So there's, there's the, uh, automation part and I really wanna dig into that further. But before we get there, I wanna talk about visibility. So, one of the things that happens is you, you do start to see all the ports, right?

You start to see all the traffic that's coming into your EHR. All the requests and all the things that are going on, and you're like, Hey, wait a minute. Why, why are all those ports open? Or why are we communicating with all these different things? And you're able to, uh, to really from a software standpoint create those choke points and say, no, no, only this information through these ports gets to, uh, gets to my critical systems.

So talk about that a little bit. Yeah. So, so really the, the first task of our project was really understanding the sequence of, of, of the packets and making sure the packet belonged to certain network session, um, between two end points. And, um, I think when we started the project, we were working with a technology that was acquired by VMware, so it was called Arkin.

Now it's, uh, vRealize operations. It allows you to visualize. All the sessions and the packets, so we really have a clear understanding of our environment. And then basically when we understand it, we, we did, um, we basically mapped our, our portfolios across all protocols and services, providing kind of that clarity, if you will.

Um, and then we, we quickly saw that, you know, we had way too many open ports and then we kind of locked it down. Where, where we were, and this was in thousand 17, and they've added some additional features, which I everyone will benefit from, but we basically kind of. Turn off ports ourselves. Um, and, and unfortunately as part of our, uh, lesson learned, if you will, um, we've turned off some ports.

It impacted a, a, uh, a, a small department, but still impacted the department, uh, 'cause they couldn't connect to that resource. And obviously we, we fixed that. But that first part was kind of doing the, the, um, port flow and then kind of mapping that back to the what we could activate or deactivate. Yeah. So let, uh, so let's talk about automation a little bit.

Uh, we, uh, you know, so one of the things we ended up doing in our cloud journey was we standardized our builds. We, we looked at AWS and we said, you know. For the entire world. They're essentially saying, Hey, you can buy this many images. And we said, well, if they could do that for the world, we could probably get below our 120 images that we have within our health system and we narrowed it down to five.

So we, you know, so we moved to five images and then my team was so happy the day they came in and said, Hey, hey. Look through, uh, through, uh, an echo device. We can now provision servers in the health system. I, now, it was just a test and they were just sort of showing it off. But I, the, the, the reason I talk about that is the level of automation.

I don't think the level of automation that's possible, um, people are taking advantage of or even realize is out there. I mean, we really can program that whole stack at this point and automate a ton of things. Yeah, so, so absolutely. So, so part of this project, we created a set of standardized templates for our electronic medical record for other organizations to use to take the template and kind of give you a good start for what type of networking should be in existence for that.

For those, those environments. Um, and then also they, they had proactive monitoring tools. Now, um, the current version of the system has basically added AI to it, where you can basically say, okay, I'm gonna, I'm gonna monitor your, um, traffic for a period of time. Then I'm gonna basically, you know, give you a report that says you should turn on or off these ports.

stion versus where we were in:

Um, now we can actually, um, see that real time, how the AI makes the suggestions of what should be turned on and off, and then basically accept those, those rules. How, how, how are you guys thinking about your cloud strategy? I mean, you're not thinking of all in cloud strategy like we're. We're getting outta the data center business, we're moving it all to the cloud or, or is that how you're thinking about it?

Um, so, so I would love to get outta the data center business, but unfortunately with some of the systems I support, um, that's impossible, right? Because a lot of those systems are still physical, um, you know, uh, cache Citrix, a lot of physical parts to those, those systems. So from a data or a cloud, um, strategy or a data center strategy, I'm really trying to reduce my footprints and first and foremost is adopt as much, um, software as the services I can.

So we we're moving from loss into workday and things like that. Um, so that helps, but that doesn't really, you know, get me out of the data center business. So the, the next piece of it is, is looking at, okay, how do I continue to, um, make sure I'm cloud ready And, and first and foremost that gets into be fully virtualized and employ cloud techniques to my current environment, which is also referred to as private cloud.

So now if I have a private cloud environment, I am kind of preparing for the ability to move workloads to the cloud if that becomes feasible. Um. And, and that's a big component. So cloud first, where cloud's not available, then virtualization. And then what, what I would love to get to the point is providing my cloud engineering team one pane of glass that they could manage workloads across all data centers.

So it could be my private cloud as well as the public cloud. And, and I think when I can enable that I have a true cloud strategy because, because I don't personally feel adopting software services a cloud strategy because I still have, you know, a data center. Um, and, and that gets into ensuring that as I start to move workloads to the cloud.

Then I can basically have the same, um, hypervisor, the same technology that I'm using for compute network and storage virtualization available to me and my my by my cloud providers. And that's where we're at now. So it's still, um, in the emerging strategy, um, 'cause really the only cloud we're consuming is from software to service vendors.

We do some bursting for our, um, researchers. And we also do some cloud environments for a kind of a development environment, a non-production testing environment. Um, but again, that's a, a trivial amount of our workloads. How, how are you thinking about Dr. So. Uh, Nebraska. So your disaster is tornadoes. Do you have other disasters that you have to consider?

I, um, the biggest disaster of recent time was a, a flood. So I don't know this time last year we were, um, under a immense amount of flooding, um, that really impacted, you know, the. The region, if, if not just the state. Um, so yeah, um, tornadoes, um, um, floods, but then I think where we, um, do our business continuity drills anymore, it's all cyber.

Um, so, um, so I think cyber would be a, a piece of the disaster as well. But physical disasters would be tornadoes and, um, floods. Are, is, is cloud a part of that strategy or are you thinking Um, so not really at this point because we, we run active Active, um, and then I haven't seen a, a DR solution per se that will allow me to run, you know, especially from an ethic perspective, active, active, um, instead of a, a, a true DR strategy of I want to have a static environment where I can, you know, have a.

I go recover that information. That's interesting. I mean, active, active, you're talking about fantastic R-T-O-R-P-O times, but active, active is generally considered pretty expensive to, to stand up and maintain. So that's why systems of your size generally don't have that in place. But you guys are, I guess, fortunate enough to have that.

moved from our data center in:

So, yeah, so it's not, you don't do well, I don't think you do. Do cost containment. Um, and you certainly do, don't do active, active for cost containment. It's really about making sure the business doesn't get impacted. Yeah. And you, you don't go through that. Uh, you know, the , the thing I always had to go through is, you know, is, is this gonna be an extended outage?

Or, I mean, you'd have to sit there as ACIO and predict whether to do DR or not because the, the move to DR was always. Um, cumbersome to get there, but it was also cumbersome to get back. Right. Well, so an active, active is, is sort of , sort of nirvana for A-A-C-I-O. You don't have to sit in that chair and, and make that decision.

It just. Yeah, so it's, it is interesting 'cause we, we have started exploring is active, active the right strategy due to the cost, um, aspects as well as, um, it can be challenging from a cyber perspective. Um, so meaning if I have an infected, um, system and I go in running active, active, I, I keep propagating that.

System. So how do I have a, a clean copy, which then gets into, you know, having a longer set of, of, of, of backup systems that are static, which in, in fact are, is dr. You know, um, so it gets, we are kind of thinking about is that, is active, active is really the right strategy anymore, um, due to the cost and, and due to some of the cyber risks.

after network virtualization:

And I think that's been really interesting 'cause we can now predict when, um, storage is running out and we can. Basically have systems that allocate additional storage, um, on, um, on demand or, um, automatically. So I think that's been, that's been promising. The other big thing we've done, um, is we we're, we're providing infrastructure as a service and platform as a service, as a self-service to our clinical researchers.

Who are doing, um, really development, um, that was done mainly because they're, um, in essence running servers and, and whatnot in an uncontrolled environment. So providing them kinda compute resources on demand, um, has allowed us to address a pretty large cyber event as well as get alignment with a group that really wasn't ever aligned with it, which is your, your research community.

Um, then what the big impact of all this has been on, I think is the changing role of it. So if you automate a lot of this work, and I'm sure you've seen some of our ROI, um, which is, you know, in the $33 million per year of virtualizing all this, which gets into. We had a, we had over 3 million ports and went down to 500 ports for our electronic medical record environment, and you can associate a value with that, but it's also efficiency now.

Now you've automated storage, networking, and compute. What we used to do from a IT perspective is now changed. We've actually introduced, um, what I call it architects, which is kind of a hybrid between an enterprise architect and a solution architect who works with the business to understand their problem, but they come back with a kind of a IT approach plan for solving that business problem.

And I don't think we would've been able to do that if those individuals are still doing their traditional IT jobs, which. Is also complex because your best server engineers are probably not your most client, uh, facing individual. So how do we, you know, retool people, align people appropriately, things of that nature.

Yeah. Actually, you gave me three jumping off points and we have a couple more minutes, so I'm gonna, I'm gonna take 'em if that's okay. The, you know, so it's, so we started hiring full stack people, right? Because you, they almost have to be full stack. At this point where they're, uh, we used to have a, a group people that did storage.

Then we had a group people that did networking. Then we had a group, people that did servers. Um, and then we had the analysts over here and, and we started hiring full stack people towards the end of my tenure as ACIO. Uh, because the, that's the way they needed, they needed to understand and, and see that whole thing.

So. Um, let's start with single pane environment. Are, are you, are you pretty close to a single pane environment in terms of, uh, being able to see network system storage and cloud from a single pane? How, or how far away from that are we today? Yes. So we do have that environment, um, and it is basically enabled through, um, a VMware product, VMware Cloud Management on AAWS.

So it basically uses the same infrastructure that, um, we've deployed here. So my hypervisor's the same, and we have some workloads running on AWS, for example, which are the same pane of glass as we have OnPrem, if that makes sense. Are, are even OnPrem, are you? So, I mean, we had, we had a storage environment and it had its own set of tools.

We had a compute environment, it had its own set of tools. We had a, you know, network environment. So, you know, we had a Cisco environment. We had, uh, whatever, we had a Dell server environment. So we had those tools in the Microsoft tools, and then we had, uh, the, uh, EMC and, and, and those kinds of tools. So we had, I mean, single pane of glass, we had lots of pane of glass.

Has, has that really consolidated for you guys? Yeah. Um, and that's really what I, I mentioned, um, vRealize, um, operations that gives us that insight across everything that we've virtualized and made full transparency. We, we haven't gotten a hundred percent, so we still have other tools, but 75% or so, we, we have that in one pane of glass.

So we are moving towards that, uh, direction as much as we can. Yes. Yeah, that's huge. Uh, do you guys do any development at all? I mean, you're a a academic center, you have university, college students, there are, are there, is there any middleware or patient engagement stuff? Are are, are you doing any development there?

Yeah, so, so on the university side we do a lot of development and, and actually we're, we we're retooling them to be a DevOps shop. Um, 'cause they were doing, um, power builder development with SASE as the backend, which if you, if you know those technologies, um, they've basically been end of life for a while.

So we're, we're converting those to, um, a, a, uh, Sal server environment running and Azure. And then also, um, PHP on the front end. Um, so we do mainly on the, on the academic side for some, you know. Administrative functions like IRB, couch payable, things of that nature, parking, um, things like that on, um, so the clinical side we're, we're pretty early in, um, kinda mobile and web technology.

We're, we're doing, um, a digital front door strategy, which is basically consolidating, um, multiple different experiences. Um, so today if you're a patient of ours, you have at least three different mobile or web experiences and they don't work together. So you have MyChart, you have find a physician, and you have a retail pharmacy app from, uh, McKesson.

And those don't play nice together. So we're basically creating one app that's accessible on, um, droid or iOS as well as the web and point to do new experiences. And, and I think some really cool things that. None of those would really, um, introduce for us that will provide a really good consumer experience.

Like we're a complex academic medical center with lots of different buildings. It'd be really neat to have an app that would say, Hey, where, where should I park? Um, park at parking lot B and take door two. And then when you come back out of your appointment, say, where did I park? And have that voice command tell you, oh, you parked in parking lot B spot two.

So we're working on some of those types of things as well, and that does tie into our cloud infrastructure and, and trying to get to a, a true DevOps, um, type of situation. So, man, well, you, you clearly did not disappoint. And if, if you're open to it, I reserve the right to, uh, call you up at a later time when I get stuck on some technology topic that I want to talk about 'cause, um, because this was a great conversation.

I really appreciate it. Um. Always happy to help, especially if you wanna get nerdy . Exactly. So, uh, is there a way that people can follow you or, or see some of the stuff that you're doing? Yeah, so LinkedIn's the best case. I'm, I'm not too active on Twitter. Um, so LinkedIn and just Brian, Brian Lancaster, c Nebraska Medicine, and, and you should be able to find me.

Fantastic. Uh, thanks again. So, uh, special thanks to our channel sponsors, uh, Starbridge Advisors, VMware, Galen, healthcare Pro, talent Advisors, and health. Erics for choosing to invest. Developing the next generation of health leaders. Please come back every Friday for more great interviews with influencers.

And don't forget, every Tuesday we take a look at the news, which is impacting Health It. This show is a production of this week in Health It. For more great content, check out the website this week, health.com, or the YouTube channel. If you wanna support the show, best way to do it. Refer peers, send them an email, tell 'em you're listening to the show and you're getting stuff out of it.

And you're learning stuff and getting really nerdy, geeky stuff out of it, that would be great. Really appreciate it. Uh, thanks for listening.

Chapters