Willie Hicks, Dynatrace’s Federal Chief Technologist recently appeared on the Federal Tech Podcast. It is such a great interview we wanted to make sure our Tech Transforms audience got to listen. Enjoy this crossover episode with Federal Tech Podcast!

Episode Links and Resources

Ep. 42 Vulnerability Management for Federal Systems

Federal Tech Podcast

Willie Hicks

Today for Tech Transforms, we have a special treat. Willie Hicks Dinertrace's Federal Chief Technologist appeared on the Federal Tech Podcast with John Gilroy. It is such a great interview, we wanted to make sure our Tech Transforms audience got to listen. So enjoy this crossover episode with Federal Tech podcast.

This is John Gilroy from the Federal.

Tech Podcast, and this is Willie Hicks from Dynatrace.

Today we're going to talk about observability and federal technology. Hit the music, Claude.

Welcome to the Federal Tech Podcast, where industry leaders share insights on innovation with a focus on reducing cost and improving security for federal technology. If you like the Federal Tech Podcast, please support us by giving us a rating and review on Apple Podcast.

Welcome to the Federal Tech Podcast. My name is John Gilroy and I will be your moderator. Everybody wants to know what's going on. They want to have visibility into their network. So does the federal government. So today we're going to maybe chop it up into small little pieces. We're going to talk about hybrid networks, network visibility for federal information technology. We're going to drill down to some specific suggestions made by an organization called Sisa. And hopefully in 30 minutes, when you're done, you'll walk away with at least some concepts about observability, what companies can help, and what CIS is doing to help our listeners maybe understand a little bit better. But first, I have to introduce my guest. He is. Willie Hicks, federal CTO. From a company called Dynatrace. D-Y-N-A-T-R-A-C-E. Willie, how are you doing?

Well, and great to be here.

Well, we set the stage. Now we got to do the hard part. Setting the stage is always easy part. During COVID everyone's online to handle that, to have It staff. So they switched everything to the cloud to hybrid cloud, public cloud, private cloud. And what's happened is some people are looking at it and going, well, we're vulnerable because a lot of these situations weren't as carefully vetted as they should have been. And maybe it introduced vulnerabilities in the systems that could not have been there if they're more careful. But we survived COVID, and now we have to consider the whole idea of looking at vulnerabilities. And maybe for the benefit of our audience, you give us just a little 22nd nutshell description of Dynatrace and the topic we're going to talk about today.

Yeah, okay. Well, first let me just say it's again. A pleasure to be here again with you. This is my second time. We are an observability company, and what that means is that we provide agencies with the ability to understand their applications, their application landscapes, their software, and to understand it from what we call the full stack standpoint. So from the end user all the way to the desktop, all the way to the database. And that allows users and agencies to get unprecedented insight into their security posture. Their application performance, how their users are interacting with the applications, and so forth. So all of this is tied together with AI. And so it's all kind of predicated on this idea of automation. So we try to do this super simple and make it as easy as possible for our customers.

So, Dynatrace, we can have this conversation walking down the streets in St. Louis because big companies have problems, and we can do this in Rio de Janeiro. Everyone seems to talk about this. So let's drill down, and let's talk about how companies fail, how organizations typically fail with this whole concept of observability.

Well, when I think about how companies fail, I think about it from a couple of vantage points. What I see with agencies, what I see with companies, they just try to do it themselves, kind of. They take the DIY approach. They try to stitch together multiple tools. This is typical in many agencies, where you've got this tool deck. You've got a lot of tools out there for collecting logs, for collecting metrics. You now got the cloud, so you're bringing in metrics from the cloud. And so for observability to work, you've got to have all of these data sources kind of combined all this data into they call it a lot of things now data lake houses, data lakes and so forth. But you've got to be able to collect this data, to understand this data, to analyze this data efficiently. And what I see a lot of times is agencies, they try to do it themselves. They sometimes then throw in the middle, maybe a correlation engine or some way, and then they'll throw some dashboards on top of that. And ultimately, what they find themselves doing is just still back in the war room, trying to figure out how to solve a problem. And sometimes I also see kind of the opposite of that, where they're not bringing in multiple tools, but maybe they're focused on one. Just logs. They're just maybe scraping logs. They're just trying to use that to understand not just their security posture, but performance. And when there are problems, they're always digging through logs to try to figure that out. And honestly, those methods, especially as environments have become more and more complex, they just don't work anymore.

So we got a big problem here. A lot of people in the cloud, a lot of people the observability, it's true with companies. It's true in federal agencies, okay? So Dynatrace can help companies with a platform that makes it easier and saves them money to have this good observability into what's going on in their system. But this whole problem of what's going on is not just isolated to the commercial world. I mean, the federal government recognizes it. In fact, CISA, just in November last month, they came up with a binding operational directive saying, hey, Willie, we know we know what you're going through, buddy. And here's a few ways we can help the suggestions. And this has been kind of a big deal in the industry as far as data management. It's kind of a big deal, isn't it?

It's a huge deal. And what Sisa came out with, what Sisa is talking about here from this binding operational directive around vulnerability management, third party vulnerabilities, very important. It's a subset as a part of the observability landscape. So this is something we need to obviously always have our eyes on. And a lot of this came from, you may have heard of this thing called Log for J. And I think probably everybody in the country at some point has heard about Law for J. And a lot of the vulnerabilities that have really appeared, not just in the last few years, this has been going on forever, but it's becoming more and more evident to, I think, the general populace that cyber criminals are taking advantage and exploiting weaknesses in the system. And a lot of those weaknesses are introduced through open source, through vulnerabilities. I'm not saying open source is a bad thing, it's a fantastic thing. But the problem is a lot of times what we see with agencies, what we see with our commercial customers, is they get hit by vulnerabilities that are well known. And so you've got these known exploits, you've got these known vectors into your application, but they're not patched. And so this is where CISA is really kind of, I should say, put the hammer down, but really saying that we have got to do a better job at vulnerability management, we've got to do a better job at continuously monitoring for those vulnerabilities and remediating those. Because honestly, I think of this as low hanging fruit. If there's a known vulnerability out there, it should be patched, it should be isolated and we should make sure that we are closing off all the avenues to our adversaries and to cybercriminals and so forth so they can't attack our systems. Now, granted, it's kind of a cold war. There's going to always be this back and forth. We close one door, they're going to try to find another one. But we've got to make sure we keep on top of these, if that makes sense.

I think that's a good so you're talking about this excellent blog that was released by, I think, Eric Goldstein. The interesting thing is, and I think you're right, you are 100% right, that this needs to be a public private partnership. I know that's kind of always talked about and maybe overplayed, but this truly needs to be where industry, where vendors are really stepping up and providing being able to interface with things like you mentioned, the vulnerability exchange and the Kev. I think it's the kev. The known exploited vulnerabilities catalog. So there are these catalogs, there are these databases. You're right. CISA and other organizations, they always are keeping these CVEs and these vulnerability and exploit definitions up to date. As soon as they come out, they go into these databases, they go into these repositories. And really for one main purpose is, well, obviously they're out there, they're known, but then they can be automated. So you have to have I think a key part of this article was also how we achieve automation by publishing these advisories in a common framework. So I think they called it the Common Security Advisory Framework, or CSAP. So being able to publish these in machine readable format so we can automate so we can automate the scanning, the remediation, all those other capabilities. And I think where industry really needs to play a part in this alongside all of these other topics, but there's something we haven't mentioned yet. It's a part of it. But being able to deliver S bombs, or what we call as an S bomb is basically a bill of materials, a software bill of materials. So industry, when I deliver, when Donna Trace delivers software, I should deliver a full account, a full account, a full bill of materials, of everything that goes into that software, every piece of third party software, every open source instance, whatever might be in there. Because this also needs to be in kind of in that machine readable format. So when agencies run into another log for J, they can quickly scan and say, okay, this is all the software we have in house. These are all the vulnerabilities. This is what we need to go and remediate. We shouldn't be just like with Law for J. I remember when it happened. I remember meeting after meeting being canceled because everybody's off trying to isolate what systems we have out there that have law for J. So it was all hands on deck. We have to make this better. We have to be quicker at this because the longer it takes, the longer we're exposed. Does that make sense?

Yeah. I come from a humble background, a blue collar background. I had neighbors who are factory workers and refrigerator repairmen and longshoremen. And I never forget how you doing? Oh, I'm overworked and underpaid. This was a blue collar lament 40 hours a week going to a factory. Overworked and underpaid. That phrase may apply to the federal government. And I think a lot of federal systems managers are going crazy. They're understaffed. They have a tremendous amount of work to do. And really what I view this as is transforming the vulnerability. It helps them manage their vulnerabilities better. They're overwhelmed. They have to do something. I think automation is going to be the key to this. They have to automate it's just a whack a mole. That's 20ft long you're running and hitting the moles and you're not getting them. So I think this is another key aspect of it. I think it's having observability what's on your system, know the vulnerability, the weaknesses on there, but also this is going to help our people keep safe.

Yeah, John, you're exactly right. And I come from a similar upbringing. Granted, I'm a little bit further south. My father was a landscaper and I worked with him. And then he owns little side businesses and things like that. But same thing. Overworked, underpaid. I see that in the federal government. And on top of that, I am always proud of the work we do at Donna Trey's and the work I do for the federal government. Because often these civil servants are, like you said, underpaid. But also and overworked but also. They're doing this out of sense of patriotism. They're obviously not doing it for the money. They're not doing this for fame. And so I think it's our duty to find ways to help augment their works, I think with automation, with AI, I'm never talking about replacing people, I'm never talking about replacing the security operations centers. I'm not talking about replacing the NOC workers. I'm always talking about augmenting their capabilities. And observability does that. It helps bring in all these millions, sometimes billions of data points. But on top of that layer, artificial intelligence, AI. So you don't have to spend hours sorting through data, sorting through logs, sorting through vulnerabilities, sorting through all of the security information. You can actually isolate where the problem is, isolate the root cause, sometimes automatically remediate it. And then if you trust your AI to do that but at the bare minimum, you can then take that data and instead of spending hours and hours sorting through this yourself, take a few minutes and then you can move on to your actual key tasks, the things you should be working on.

And prioritize too, because everything doesn't have the same value. The military guys say if you defend everything, you defend nothing. It's an old military adage from 500 years ago. Same is true. I mean, what if, let's say, System A Gilroy software has got the vulnerability and your agency doesn't use Gilroy software? Well, you don't have to worry about it. Do you? I mean, you can prioritize where the 40 hours a week you're in there, you can prioritize and go to the right system. Maybe the Hicks system has got a vulnerability and you happen to use the Hicks system. Okay? Forget about Gilroy. I mean, the whole idea of prioritizing. How else can you be efficient with the limited time that we have?

You are so right in that. And prioritization, especially in any of these cases with performance issues, with security vulnerabilities, especially when you have a major kind of going back to log for J, when you have a major outage, when you have a major vulnerability, a major issue that you need to resolve, you need to find those most vulnerable systems. You need to find the ones that are with offer J. Everything was a level ten. We got to get everything remediated. Well, maybe not. Maybe I should focus first on those top 20%, 40%, whatever it might be 50% of systems that might be forward facing, they might be constituent facing, they might have access to the Internet. They might have sensitive or confidential or secret data sets, data sets that I need to protect. Those are the ones I need to focus on first. There might be 20 or 30% of those that get law for Jays on the system. It's not actually even being used. It's in a dormant module not being used. I can handle that later. So to your point being able to actually quickly isolate that and prioritize completely, that is completely right. And using another military idea, situational awareness is another kind of key part of this. And that's, I think, what we're talking about, this idea that the military this is kind of one of their mindsets being able, knowing their environment, knowing the environment that they're working in, knowing all of having as many data points as they can because this is how they make decisions. This is how they make really quick tactical decisions, is because they have the best information they can have at a given time. I think it should be the same way with our systems. You should have as much and the best data that you can have to make those decisions and to make them quickly so you can isolate problems and move on.

If you enjoy this conversation, you may want to listen to an episode. Episode number 32 I did of a guy named Dr. Stephen McGill from Sonotype. He talked about the S bomb, about the software bill of materialism. And that gives you just a different flavor, different aspect on it. I'm going to turn the tables here and instead of examining the number of hours that a federal CIO puts in the office and doesn't, let's look at the vendors. And one could read this like an attorney could read this and go, it looks like they're turning that spotlight to the vendors and saying the vendors are going to have some responsibility here for telling the agencies they work with about known vulnerabilities, for vulnerabilities in the system. Now, no one wants to talk about their dirty laundry, but I think that's the case that's happening here.

Right? It is critical, yes. As industry, we don't want to expose ourselves like, yes, we've got vulnerable modules in our software. But I think what everyone needs to realize, industry and agencies, is that this is the world we live in. We all use open source. We're all using different types of different coding techniques and different ways. Now, we should be using very safe coding practices. We should be using the best practices out there. But ultimately, things like this happen. I give a perfect example, is the law for J. Almost every company out there was using it, every agent. It was everywhere. And what we found out later is that this vulnerability had been there for years, but no one had discovered it. So it just kind of proliferated. So industry can't be just held accountable. If no one knew. Now, if we knew and did nothing about it, now, that's a whole nother story. Now, I think where it becomes negligence is when we try to hide things, when we try to cover it, when we try to protect our interest by kind of not exposing that we had this problem. Well, we're putting agencies, we're putting our commercial customers, we're putting everybody at risk, and nobody wants to do that. So at Donatreys, we are very open. We publish all of our vulnerabilities, we publish our timelines. We're very open about that. We act upon these things quickly because they do happen. One question that often comes up with S bombs, though, is that if we put out, there kind of everything that's used to make our products, all of our secrets, well, we don't have to put down to the code. Level, but all the modules, everything we're using. Then there are sometimes there are people ask, well, do we want our competitors to get a hold of that? There are ways for us to kind of control that, but ultimately we need to do what's right for the country, what's right for our customers. And that means kind of honestly being open about all of this.

Willie, I'm listening to you carefully taking notes, and every time you say S Bomb I spent 25 years doing live radio. I always had S Bomb, I'll tell you that much. And never got in trouble with the SEC because of it. So that's a light hearted approach to S Bomb.

I don't mention that going through the airport.

But you don't want to say that word either. I'm an expert in what? Italy? Hey, Willie, it's football season and there's some silly team in town with a crazy name. And we all know that when a defense looks at, let's say, the Chicago Bears, and they say, oh, this is what they do, then this is their Tennessee. And this. And then they go to the game, and all of a sudden, they tossed it to the split end. It's a new play. And so my point is that there are known vulnerabilities. You can scout a team, you know what they're going to play, then there's unknown vulnerabilities. This is one critique of that. I don't know if it's a fair critique or not, but we can say, well, these are the known vulnerabilities. So you think it's a fair critique or not a fair critique?

No, that is 100% a fair critique. And there are kind of two ways I look at this from the known standpoint. The known ones we should just be addressing. I mean, we know them. We know they're out there. They should be addressed. They should be addressed quickly. The unknown, that becomes a little bit more difficult, obviously, because they're unknown. So there have to be other approaches. There has to be. And I think this is where you start looking at your different types of security tools, your seams and all the different types of tools you're using. And this is where I'm obviously a big proponent for AI and machine learning and different types of approaches where, okay, so we don't know that there's a vulnerability out there, but there are probably heuristics and different things that we need to be analyzing for behavioral patterns, user behavioral patterns, machine behavioral patterns. This is where, again, observability becomes key because understanding how machines normally should work. What should I see from a CPU standpoint, from a memory standpoint, network communication, what's talking to what if I see something very simple, this system might have been compromised. That is not a known vulnerability. I saw no scanning. I saw nothing. But all of a sudden, I started to see traffic that I never saw before. And it's very sporadic traffic. It looks like you got to be really you have to look at almost every transaction because how this system is working is sending just a little bit of data at a time. It's kind of almost trying to mask what it's doing. You need to be able to see those types. And I don't claim to be a security expert. I'm an expert in observability. And that's kind of key to, I think, observability, which I think is key to security. Being able to see the small changes, to be able to understand those small changes and then to act upon them. Because it might be a normal this might have been a code change, and this activity is completely normal. But we need to analyze it.

Dynatrace.com. That's Dynatrace. No, I in. There is A-Y-D-Y-N-A-T-R-A-C-E. Well, simon sinnick wrote a book. He said you got to start with why. And I think we spent the first part of this discussion with the why. We did a lot of Y, and now we got to transition to how and when you talked about vulnerabilities. I think traditionally the how was, let's say Gilroy software would see a vulnerability. Then I'd put up a PDF, and then maybe a federal CIA would go to my site and read the PDF and take action. I guess technically, legally, yeah, I announced it, it's buried, but it's a PDF. Somewhere in the transition I see in this document is that, hey, let's make it machine readable and be able to be distributed in a more efficient manner than some PDFs Gilroy.com that no one reads. I mean, this machine readable capability really, really puts, you know, puts a lot of muscle into this, doesn't it?

100%. And kind of the old way of doing this definitely was I go out monthly to a database. I go out and vulnerabilities are out there. And over time, things got a little bit better, there's more automation. And I look at it this way too. I think that where things are going is that I think industry, when they deliver new software, it should be just every piece of software needs to have that software bill of materials. It needs to be in a format, JSON format or whatever format is that. It'll probably be JSON and whatever the standard is, that could then easily be translated into as part of the agency's automation process. So that's uploaded, and it's constantly just whenever there's a version change, whenever there's an update, there's a new S bomb. So that's part of it. Then we're tied into also the known Exploited vulnerability catalogs, and we're tied into the different vulnerability databases. So we're pulling these fees, these sources automatically. And you have whatever method you use, if you're using agents, if you're scanning, you're using scanning software once a week, once a month. We should always be looking, we should always be observing. And when something changes, when something becomes anomalous, when it becomes not where it should be, that should be flagged, and we should have an analyst or an AI algorithm that can actually quickly look at that isolate, does this look like it's been compromised, and then act upon it. Ultimately, I'm really big on automation of that. I believe that what we see commercially and from agency standpoints is that the idea has always been kind of perimeter defense protecting. You got your web application firewalls, you've got all your different perimeter tools to really try to prevent people from getting into the environment. With COVID what we saw is the perimeter actually extended out now to the home. So you got a whole nother aspect. So you've got to factor that in. I think that with observability, with this idea of kind of always monitoring, I think we pull the perimeter in even closer to the application. So there are a lot of platforms out there that can do basically runtime application protection. So while maybe having an agent at the application that's constantly monitoring it for performance and all these other things but also do I see a command line attack, a SQL injection attack, a typical type of pattern that tells me that this application has been compromised. I should be able to stop it at the application. So I need to maybe tighten that parameter up as well. So that's just another idea.

Yeah, I think all the people listening this should take and go to system and read this. I think it puts a lot of the concepts we've been talking about observability and the whole machine readable language and software bill of materials and understand what you have. And it gives you the tools to actually accomplish some of the things they've been targeting. And I'm really glad we had John talk about it. You've been listening to Federal Tech podcast with John Gilroy. I'd like to thank my guest, Willie Hicks, Federal CTO Dynatrace.

Thank you.

Thanks for listening to the Federal Tech podcast. If you like the Federal Tech Podcast, please support us by giving us a rating and review on Apple Podcast.