I discuss with Lukas Hlavicka how ordinary security incidents can escalate into full-blown crises when organizations lack preparation, reliable backups, and secure deployment pipelines. We share stark war stories, from backups that were never actually taken to compromised software deployment systems and a ransomware attack that exposed six months of bank fraud. Lukas draws on over a decade of incident response experience, including leadership of a national CSIRT and work in the private sector. We describe common failure points: unknown assets and third‑party links, intertwined IT and OT systems, missing evidence, unclear roles for decision and containment, and compromised CI/CD. We stress clear, practical steps: maintain offline incident plans and backups, map assets and dependencies, secure and monitor deployment pipelines, and practice response with defined authorities. This episode targets any organization that thinks “it won’t happen to us” and seeks concrete lessons to prevent incidents from becoming crises.

Takeaways:

• I learned that many incidents become crises because organizations do not know what assets they own or how they connect to third parties.
• We must keep offline backups and verify that backups are actually taken and restorable before incidents occur.
• I saw attackers abuse cloud managed security products and deployment pipelines to gain full network access quickly.
• We found that unclear roles and slow approval processes delay containment and allow the attacker to spread further.
• I observed that organizations often destroy evidence or change systems in ways that hinder forensic investigations.
• We recommend practicing incident response, keeping plans offline, and retaining institutional memory to avoid repeating past mistakes.
• I noted cases where ransomware accidentally revealed long running fraud and showed the need to follow financial trails in investigations.
• We must monitor CI/CD and software supply chains continuously because compromise there can implant persistent backdoors.

Hello everyone.

Welcome back to another episode of the Security By Default podcast.

I'm the host of the show, Joe Carson, chief security evangelist at Segura, based here in Tallinn, Estonia and joining you with another awesome guest and another fantastic, very interesting topic.

Always bringing interesting ideas around, how to get clarity out of the chaos in the world that we live in and really allow you to learn and help you on your career or journey that wherever it may lead you in the cybersecurity world.

So I'm joined by a guest that has a very, a very, very experienced background.

So Lukas, welcome to the Security by Default podcast.

Since it's your first time on the show, can you give the audience a bit of a background about how did you get into the industry, what you've been doing and what you're doing today.

So if you can give us your origin story.

Thank you.

First, I would like to thank you for privilege to be in here.

So I would like to say hi to audience.

I started as the system engineer, then incident responder and was head of analytical department.

And then we came and created our own cybersecurity company in Slovakia east where I am currently serving as cto.

We are focusing especially on the incident, risk response, forensic, basically let's say special cyber operations and also helping our client to basically solve their problems in cybersecurity world.

Fantastic.

That's really interesting.

For me it's one of the fun areas.

It is one of the most stressful areas in our industry, working in incident response because you're always looking at a lot of the victims, helping them, looking through the incidents.

And sometimes in those situations you're faced with hard choices.

And I think in recent years through the increase of things like ransomware and the geopolitical situation we're in and DDoS attacks, it really means that all organizations have to have a plan.

They have to have, you know, practice.

They need to know what they're doing.

And this topic of course has taken some of the interesting scenarios is that when things go wrong, what choices you make can lead from a simple incident that can go to completely disaster and crisis situation.

So take me through what's some of the first things, you know, when organizations, you know, first experience an incident, what's some of the common things that they start off with?

What's the common kind of like simple mistakes that already can it start the snowball effect rolling.

Basically there are quite a lot of areas I like my job because there is quite a lot of the.

The dopamine and adrenaline and this kind of stuff.

When the incident started usually there is a lot of suspects incident happen.

And basically the.

The cases where we are involved usually is the basically most serious incidents from the ransomware, sabotage, exfiltrated data or sometimes even the life of the people which are connected to the.

To the.

To the.

To the devices are at stake.

And the mistakes which really creates the crisis is usually the bad steps which was done before the incidents that the organization was not prepared from the.

We have for example even nowadays we run the incidents in the organization which is not defensible.

Basically we solve the incidents, we clear the attacker.

But because of there is very specific type of the organization.

We just created the infrastructure and cannot be shut down.

And the organization is not defensible.

We basically need to play a little bit whack a mole with the attacker.

Because we removed the attacker from one side.

We deleted.

And so.

And we said we found that we have new.

We have actually all.

We found new connections to the external networks which is quite old.

But it was one of the supplier which was.

Which is not even the organization not exist for six years or seven years.

But the connection is still there and the public IP and the services is publicly available still.

So we have another vector which we need to discuss and to contain and eradicate and recover to the.

To the again to the production.

So basically the most common cases is that there are missing that the organization is not defensible.

So the organization doesn't even know what they have how they are interconnected with their third party.

Especially in the multi organization multi data center is everything from the New York through the Frankfurt through the capsicum store through the.

Through the Hong Kong and basically everywhere we doesn't know how we are connected with the third parties.

We doesn't know what we have.

And basically it's something happened.

This is most challenging parts.

There are actually way how to do that.

But we can.

We can maybe speak about it later.

Then there are also possibilities that basically where we have especially the integration with IT and OT systems.

And there is not possible to distinguish between security and safety networks.

For example, we have cases few months ago where basically the safety network was interconnected with the standard IT network.

And so we cannot basically disable the connection because basically the same connection which actually was used for the connection between the Organization and their supplier was used for actually the connection of the safety systems of the chemical factory.

Which basically if you would disconnect it you doesn't know if it will going to basically blow up.

So this was actually funny.

So interconnection IT and OT network especially the safety networks.

There is also the missing the evidences.

Sometimes you doesn't even have access to the part of your own infrastructure.

For example, we found a way where the.

Where the attacker was able to compromise part of the infrastructure.

It was a multinational retail company.

And basically what was happening is that the organization not anymore had access to their own switches, routers and firewalls because it was managed long time ago by the some kind of third party vendor.

And some of these devices was not passed to the new vendors.

So nobody has access to those.

So that was quite funny actually that we needed to do offensive parts of our engagement that we get to the specific firewalls.

So we.

So we was able to basically help the client to defend their infrastructure.

That there is also the undefined roles that okay, we are buying this kind of software as a service and but who is managing it this kind of stuff.

There is also a lot of parts that okay, this is.

This is this is we paying this kind of part of the infrastructure.

Storage as a service, containers as a service or anything as a service.

But we doesn't know who really actually is able to help us with that because we do it with the cloud service broker which basically provide us with service.

But when we needed okay, but we have problem then there was three or four days until we get to the people who actually was able to do something about it.

So this is kind of the.

When the.

When the crisis it's happening.

And also if you doesn't have precisely know what your crown jewels actually are.

And if you had it basically backup for example.

Very interesting story happened few actually it's already months ago where basically the organization didn't know if they.

If they have backups and say okay, but our vendor was told us that we have backups.

So we come to the vendor and we need the backups to.

To recover your organization.

The organization.

And they said but we was never asked to do the backups.

And I say we.

We send the.

We send the proposal.

But it was never approved.

So the backups was never done.

And the organization was thinking that okay backups and we have it in daily basis.

So you doesn't have backup.

So we need to start the negotiation with the attacker and make sure that what kind of we can we can recover what kind of.

If we need to basically pay the RAM until we find the the until we receive the the or if we receive the recovery the decryption key.

So this is something which basically if there would be standard processes, if there would be communications organization know what have.

How dare they have it.

They would be able to basically have the incident would be very easier to solve.

But usually why the incidents became the crisis is that the preparation is not enough.

Basically they sometimes have very expensive tools, for example EDRs, XDRs, AI based and you know, Palm, but nobody use it or they are in default state.

For example.

In some of the cases I even found that basically the security product itself was actually way how attacker was introduced into the infrastructure.

Because well, we bought these new shiny firewalls and so on.

And we say okay, you can integrate it with with our cloud solution and so on.

And you can everything managed from the cloud.

Great.

But then basically attacker came because why they would use two factor authentication to the network as a service, right?

So the attacker came, they actually find the password because the administrator used same password on some website which was compromised.

And then actually attacker just do the credential stuffing connect it to the to the network to the network as a service.

And he connected his own machine as part of the network in the server side of the network.

And the solution was actually very good to that.

So it allowed the attacker to connect directly to the same network as was domain controllers.

So he was able to connect there.

So basically some incident really quickly escalated to crisis which needed to be done.

Then in a lot of cases, even when this is not happening, the organization usually has quite a lot of problems in the reward that they doesn't know who can authorize.

For example disconnected part of the network.

And basically it happened in the actually same the factory where they ask okay, we need to disconnect this part of network.

Because this is how the attacker is spreading through your infrastructure.

And they say okay, we need to find out who is able to approve it.

And while we were looking who is able to approve it, we waited hours.

So we said okay, we do this kind of firewall rules, this kind of containment measures and so on.

But basically there was infected about 20 another servers which needn't to be infected if we would have possibility to do what it was possible to do.

But we was told do not touch it.

We are not allowed to touch it because that is responsible.

We doesn't know who our sister company in another state.

So this kind of stuff is quite a lot of problematic Also we see in quite a lot of cases where the.

When the.

When the administrator starting to do the incident response, then they.

They destroy the evidence.

Sometimes not willingly, sometimes even willingly because they okay, I did something wrong, I destroy evidence.

So nobody can see that.

But we see that you destroy the evidence, right?

And also we also quite lot of see that.

Basically when the incident is start happening, for example, we see the attacker is lurking in the environment.

He is trying starting to exfiltrate.

Then we basically tipping of the attacker.

He doesn't know what actually was done, but we disable his account and so on and say okay, execute order 66 and wipe the backups and start encrypting.

This is when the real incidents start to become the crisis.

And also there is quite a lot of problems in the cases of the solving of the incident.

Especially if there is multinational companies involved.

Then basically there is information sending right and left and these information are usually not either correct or basically is what I think I will send us information and basically it will happen.

If you know, if you remember a ping of the deck.

Basically you send so much information that it will kill basically the recipient because basically he doesn't know what to do.

Or actually even the Smurf attack is actually applicable here because you receive the information of from every every site and you doesn't know what to do.

Another part of very interesting why the incident become crisis because you have for example you find the infection on the some of the server.

Let's say on the web server or publicly facing server or anything like that and what is actually happening the server is compromised and you say okay, I clear it.

I do not know how the organization how the attacker got there, but I but I reimagined and I'm happy and basically I could if I investigate it by the attacker or how the attacker got actually there on the server then I would be able to find that okay, that attacker basically had privileged accounts or something like that or exploit vulnerability which I didn't know it was present there.

And basically if I only recover or even if I basically solve the initial.

This initial vector and I doesn't know how it actually got there really what was the first point of attack then?

Basically I'm allowing the attacker come back.

And when the attacker come back usually I say okay, I will prepare.

And this is this kind of cases which I'm very unhappy about because basically this is the incident which could be.

Could be avoided.

We had the cases actually in the June where the organization it was actually the second it was actually the second incident because basically we was performing incident response there a year ago and what was actually happening was that basically we was monitoring the endpoints and we saw that basically the software management solution deployed anomalous system and it deployed actually malware immediately.

We stopped basically the spreading and basically block and so on.

But we found out that attacker actually was able to compromise the software deployment toolkit which was basically.

And created new profiles.

So anybody who basically connect there was downloading a new package and install malware in the system and say okay, and how it was good because this was, this should be internal tour.

I've seen, I've seen, yeah I've seen those, those scenarios a few times.

One of.

One of my favorite ones was there was a telco that had an initial incident and what they decided to do was let's go to the backup.

They restored a backup and within two weeks later the incident reoccurred again because they actually had backed up the actually the infection.

So, so the infection was in their backups.

They just never sanitized them before recovery and never said let's.

When was the initial access?

When was the initial incident happening?

They just went to that recovery scenario, got the business, got everything running and then of course the reinfection was introduced because the backups actually had the infection contained in it.

And I you mentioned about the.

I remember a similar scenario doing.

It was a.

There was a Windows 7, Windows 10 upgrade.

This has come back quite a while.

And during that upgrade process there was a massive new software deployment rollout that was about to happen.

And what the attackers had done was they had embedded their piece of malware into the software deployment.

So you know, the organization during this upgrade was going to deploy the malware for the attacker themselves.

So sometimes, I mean they are very creative.

Yeah they try to, you know, embed themselves into the existing organization's workflows.

So absolutely it's it so and some of those lessons, you know, you hope that organizations make it more visible from, from the lessons learned and that's what I love about this particular topic is that as long as we get, you know, enough knowledge out there that these, you know, similar mistakes and, and, and incidents that you know, could have been avoided can, can get to the knowledge of others any other.

I mean what else from.

From the software deployment?

One that's always.

It's very creative when the attackers get into your software supply chain.

Actually I even better because you know, nowadays it's infrastructure as a code and my most beloved part, how the attacker got there when you Compromise CI CD pipeline and basically you add a repository where basically every time you add a new version you add basically backdoor.

And usually this is not kind of the stuff where you have some kind of ransomware or this kind of stuff.

Usually this is the stuff where basically you have the backdoor and attacker go there, receive the information and live there.

And when you analyze it and you see there is even the years there, it's actually okay.

And you say okay, this is a very funny story and I will 100% told it in the next podcast which I, which I will be in.

So that is very, actually that's unfortunately.

And what we see that even when the people, when the organization will learn that they shouldn't do this, they do the same mistake six, six months later.

Usually as you know that organization has six months long memory against the bad stuff.

Then usually they say oh, it was not so bad, it could be we survived.

So let's, let's spare some money, let's lower the budget and then the incident will happen again and again and again.

Basically I was thinking same as you, that if we put enough information to the public they will learn, they will try to learn from the mistakes of the other, other organization.

But unfortunately what I found out is that even if the organization will the experience, lived experience of the mistake, they are able to forget it about year, year and a half later because when we do the, for example all these or this kind of stuff year and a half again they are on the same level as was before the incident.

Yeah, yeah, I've seen, I've seen those scenarios happen more than I wish, you know, they would occur is that they quickly become victims again.

And a lot of sometimes is that they see the cause as the IT and security team so they rotate the people but they didn't change the culture and, and the investment into IT as well.

One thing I've seen also quite often I don't know if you've seen is during incidents as well, two scenarios.

One is the instant response plan was also encrypted with a ransomware.

So yeah, so, so, so, so their ability to respond is also like okay, we don't have an offline copy and we don't know what to do because, because our instant response plan itself has been encrypted.

Always funny scenarios.

I mean it is crisis and it's painful when it does happen, but when you think about it, you know, it is iron.

And then another situation as well is that when you, when you start the incident, you're doing the forensics evidence gathering.

A lot of times you find that, you know, you're dealing with an attacker who's launched a campaign, but you'll always find multiple evidence of maybe other multiple attackers who just, you know, we're in the environment but just decided, you know, that then wasn't the right time.

Or they may have not had enough privilege to do as damage as they want.

And they were waiting to do that lateral move or elevation.

So sometimes you do find, you know, multiple attackers in the same environment.

It's just one was more malicious than another.

Actually I have one one story where the attacker, the organization was actually saved by ransomware actor.

You say how it is possible the organization, it was actually the small bank.

It was connected to the SWIFT system and this kind of stuff.

And basically they was hit by ransomware, became where there was hit by ransomware and started to solving the incident and so on.

And in the forensic we found out that the attacker was there six months before and slowly steal the money and move it away to the.

To the other accounts.

And if the attacker would.

And actually the attacker was very, very smart in a way that basically they have very basic controlling of the money.

And there.

So he changed the.

So the summaries was correct and nobody checked anything else but summaries.

It was very small bank in Africa.

Then the ransomware actor hit and basically this attacker was basically quick away because it was encrypted also the system which he used for stolen the money.

So it's actually very funny story.

It's not funny for them, but it was very funny for me to be honest.

That's an interesting.

It reminds me of that one incident that I worked on a few years ago was all very similar, but this was.

There was two incidents.

One was where the attackers had, you know, deployed a ransomware, brought the business to a complete standstill.

And during the forensics investigation, which was permitted to go back five years of logs found two years prior, one of the employees had stalled a crypto mining within the business and had been actually using the server resources for actually doing crypto mining.

And the energy costs alone, because the CPUs of course are running it, you know, over over loaded the CPUs and the energy costs alone ran into the tens of thousands of euros, which was like, okay, so sometimes you find if that incident didn't occur, would they have found that employee who was abusing their resources?

Probably not.

Another interesting one, this goes back into.

So it's supposed to wanna cry, not pet you.

Yeah.

And this was in.

In Ukraine.

They were doing basically they were overloaded with cases.

So the cases they couldn't handle this huge case load that they had.

And they were giving financial support to victims of ransomware.

So these organizations who had ransomware, they were getting financial support to recover.

And one particular organization actually because they were trying to hide fraud had deployed the ransomware to themselves.

Which is, which is very creative when you think about it.

Because at the same time there are hiding, you know, they're destroying evidence of the fraud and, and also getting financial support to recover.

Very creative.

But unfortunately they did.

They didn't get fully away with it because of the case reviews.

I was able to.

To lead it.

You know, it didn't follow the typical attack path because you can see, you know, it was spreading from within rather than from an initial external point.

So there's always interesting cases especially you know the work you've done for.

For the cert because you know you do get a lot of insights over the years from some of the.

Some of the most high profile cases as well.

Well, that was.

That was the time where when I was on a governmental C17 there started ransomwares and this kind of stuff.

But most of the.

Most of the attacks was that somebody compromised something steal data and so on.

And one.

One actually story was that on the.

One of the servers what it was utilized for spreading basically the.

It was small city or small village.

Basically they have one web server where they showed how much you need to pay for the dog and this kind of stuff.

And basically somebody compromised their server and utilized the machine as there was MP3 storage.

So basically and they found the incident in that way that they received quite a big bill for electricity because nobody actually was going to the web server of the small village.

But when you know you sharing the new MP3s, I don't know Shakira or something like that.

Everybody dumbled is it so and they say okay, why we have so much electricity bill?

And they found that they have compromised web server.

That was actually very very funny funny story when I started and or even better they started then business email compromise incidents and there was.

There was this village chief which basically received the email that he need to access and so on.

So he pay all money which they have on the.

On the budget in the fall reach to the some account.

And then that was about a March or something and nine months they doesn't have even sent in the.

In the.

In the budget because they.

They send everything to somewhere else to the.

To the wrong account.

I've seen, I've seen that with.

There was a ship company, shipping company which was basically had docked and was doing the fueling process and for ship, you know, cargo ship, the fuel quite, quite expensive.

So it was, and I think it was like something like €200,000 or whatever, or whatever, whatever currency.

And when they, they got the invoice, they paid the invoice for the fuel and but it's actually been business email compromised.

They'd actually already compromised the systems, had sent the invoice out before it was actually meant to be sent, had changed the accounting information.

So the next day later they got the, the real true invoice and they're going hey, hold on, we have already paid it.

What are you sending another invoice for?

And ultimately they end up finding that yeah, the, the organization was actually doing the invoicing for the fuel had been compromised and all their invoices accounting had been changed and, and the attackers knew how to kind of, you know, their processing and timing as well.

So Casey, you know it's, it's, it's important I think sometimes in our industry following the money is one, one way of motive.

And we talked earlier about, you know, we even look back at Cliff Stoles, Cuckoo's Egg which was a 75 cent accounting error which led to entire, you know, basically cross costs the Atlantic Ocean investigation into why the 75 cents was missing.

So sometimes money can tell us a lot of things about, you know, some, something happening.

It was an expensive, expensive experiment but with an interesting motivation.

So.

Yeah, so, so question you know, in this, you know, we're always moving, you know, fast.

This industry changes a lot.

You know, we've got lots of technology trends, we've got lots of attacks are getting creative, they're advancing social engineering is, is, is accelerating.

How do you, how do you what, what way do you stay up to date, you know, especially in the role that you're in dealing with a lot of incidents.

And how do you step to date?

And also what, what thing would you recommend, you know, any of the audience who wants to prevent their organization from becoming a victim and not going into a crisis.

What would be, what would be your recommendation as well?

First of course this podcast.

Absolutely.

It is a source, a source of knowledge and entertainment for the world.

But there is also the very good conferences which is also very cheap or very free.

For example, one of the best of them is BET defcon.

It's my most favorite conference.

I actually get bought a few few books.

Yep, two two awesome books and absolutely from my favorite publishers which is the no Start yes no Start press yes.

No Start pledge rulers.

So and of course the.

Of course we each time for example look for the new time of the incidents.

We have also our own research which basically one of their.

One of their job is basically to provide us with the links with the new kind of ideas and how to basically enable us to be ready for new kinds of attacks.

Especially here in AI Basically the attacks actually quite faster and quite.

Nowadays even the less prepared and less knowledgeable or sophisticated attackers actually can create the attack or perform the attack on the highest level.

And we actually see even the agent which was on the one of the pivot point and there was the instructions that until compromise do this loop actually Very very very funny.

So the Learn, learn and learn.

Actually there is very good source.

I don't know if you know Cyber.

Yep, yep, I do Cyber.

Absolutely.

I was formerly one of their instructors and mentors there.

Actually this, this podcast came out of Cyber eventually so good.

Cybri was the company that we.

We started the original podcast which was a long time ago and this came out of that.

That Source Vee Co. Has quite few of subscription of there.

So yeah, especially the young juniors can.

Can.

Can work and play.

Absolutely.

It's.

It's been fascinating chatting with you.

Many thanks for coming on and sharing your insights and you know the, the lessons learned and hopefully the audience will you know take away.

You know they're.

I'll.

I'll kind of create some summary in the takeaways in the show notes and also put the links for the.

For the books that you shared as well.

But yeah, I mean it's important to know, you know what we can learn from past incidents, from past scenarios from organizations who've been down, you know the path of being a victim and how the recovery and the post kind of mortem scenarios.

It's really important.

So many thanks for coming on and sharing your story and thank you.

It's a pleasure having you on.

So for the audience.

Thank you.

What's Lucas, if anyone wants to follow up with you or you know, connect with you, what's the best way to.

If they do have follow up questions, what's the best way to contact you,.

Please drop me an [email protected] or basically write me on LinkedIn.

I would be very happy to answer any of your questions and hopefully I will not meet any of the audience in the incidence cases.

But remember the incidence is also dead.

You will meet him sooner or later and better be sooner than later.

Yes, and prepared.

And you know, have the practice and always have an offline copy of your instant response.

Exactly.

This is one takeaway so and we'll definitely make sure to include all of those in the show notes as well.

So Lucas, many thanks for being on the show.

So for everyone out there, this is the Security by Default podcast.

Bringing you information, knowledge, lessons learned and hopefully is something that will help you make your organization safer, make your social sphere, your family better protected and stay safe and secure out there.

So tune in every two weeks for new episodes.

This is Security by Default podcast.

I'm the host Joe Carson.

Stay safe.

Take care until next time.

Thank you.

May the fourth video Goodbye.