With the recent disclosures that OpenAI's security testing using its own models broke out of the sandbox and attacked Hugging Face (and others), Erik, Dan, and Brian thought it was worth a moment to discuss and debate the current state of AI, securing AI models, some of the ways that organisations can isolate and test without such risks, and the implications of what could be seen as offensive (and retaliatory) attacks from a corporation.

And, lest anyone think we are picking on OpenAI, the same thing happened in testing by Google, Anthropic, and Meta (Facebook) resulting in attacks on quite a few other companies. Yep, you're not cool in AI if you haven't tried to attack another business with your AI model. But why on earth are they all annoucing so loudly that they hacked into these other orgs? We live in interesting times, for sure.

Thanks for watching and listening!

Show Notes:

• a

Some of the links in the show notes contain affiliate links that may earn a commission should you choose to make a purchase using these links. Using these links supports The Great Security Debate and Distilling Security, so we appreciate it when you use them. We do not make our recommendations based on the availability or benefits of these affiliate links.

Welcome to the great Security debate.

This show has experts taking sides to help broaden understanding of a topic.

Therefore, it's safe to say that the views expressed are not necessarily those of the people we work with or for.

Heck, they may not even represent our own views as we take a position for the sake of the debate.

Our website is greatsecuritydebate.net and you can contact us via email at feedbackreatsecuritydebate.net or on Twitter.

Twitter at Security debate.

Now let's join the debate already in progress.

So, hi, like on the hugging face thing though, I think there's a misunderstanding of the whole hugging face breach.

And what I mean by that is like, I've heard people say, well, what should we be worried about?

Like somebody asking to spin something up internally, their own model, and the CSO saying, well, I don't know if we should do that because of what just happened with hugging face.

Right.

I'd like to introduce you all to my good friend Air Gap.

And this is an easy answer.

He's like, but this is going to be a sovereign model.

Well, right.

Sovereign model versus.

I mean, but sovereign models, there, there's a bunch of mixed terms in there.

You know, sovereign model, something you run in your own bedrock, is still, it's yours.

It's a, you know, it's a model that is run locally, but it doesn't mean it's disconnected.

It doesn't mean that you can't, I mean, you can try and put your parameters around it, put your, put sandboxing around it so that that model can't interact with anything else.

But what we're seeing in this is that the models have become so good at using their own resources and finding other resources.

And much like children interpret the rules the way you say them, not necessarily the way you mean them.

But it's, Aren't we missing the point that it's, it's not the model directly.

A model in and of itself is actually dumb.

Yeah, right.

It's a mathematical equation guessing of what comes next, but it's the capability.

Sure, they've been trained on a ton of data, but unless you give it the skills and the tools to actually do something with it, it's just something creating text.

Agreed.

And it's all.

It all comes back to.

It's how you configure it.

Stupid.

It's how you configure it.

And you know, it's no different than any of the other issues we've had.

Know that as security professionals for the last 30 some years, if I run a, if I write a bad firewall rule, things are going to get in.

If I write a bad sandbox, it's going to get out.

If I don't give the, if I don't tell the model, don't do the following things, it's going to do them and it is.

And.

But I come back to the child reference.

I've had this conversation a lot over the last seven, 10 days.

You have to treat your models like they're kids.

Because.

Because if you don't explicitly say A, B or C, they're going to try and find ways around back to Water will find its way.

Water children and users, models and kids will find their way if you, you know, based on the, based on the parameters you give them.

I was growing up.

My dad said I was 16.

I just gotten my license.

Daniel, don't drive to and inserted a small town nearby.

And I thought to myself, yeah, but he didn't say go to a different small town.

That's further on that goes through the W. The first city.

So I did and I justified it to myself.

This is no different.

I totally agree.

And this is where people are getting lazy.

Right.

Because the, the way a lot of these tools are being written.

Oh, I'll just give the gar rails, I'll put it in text and I'll tell the model itself.

Yeah, but you're asking a model to not only do something for you, but also self police.

Right.

Wrap it.

Like everything else that we've had to do, there's a reason.

So your kid analogy.

We tell them to lock the door when they come in at night.

But there's a reason.

I have smart things that at a certain time automatically locks the door because I know it's probably not actually going to happen.

There's backup controls around it.

But this is.

I feel like there's still a general misunderstanding of how some of these models work because I remember somebody made a comment about perplexity that uses Deep seq as I think it might be their controller that's kind of proxying between different models and.

Oh well, it's from the Chinese.

Yes.

Okay.

But if put in a container where it has nothing access to nothing else and it's purely just returning some values, what's the real risk there though?

I think the risk that, well, depends on how isolated the container is.

Yeah, I've gone to running Deep Seek and it's a great model and I've been enjoying it.

But it's run on a device in which there Is no ethernet connected.

The WI fi, the WI fi antenna has been disconnected at the board, you know, so you, you can decide it's risk management all over again.

We decide to what extent we are risk tolerant and know that when you introduce the potential for risks they might just get realized.

And in this case it did, it got these things got realized.

You know, the attack, the, the, the model, the, the.

I guess one other element is there were potentially conflicting instructions.

Solve this problem and stay within these parameters.

And which one wins when the parameters are preventing them from solving the problem.

In this case, solve the problem was the prime directive.

And so it said I'm going to solve the problem again, much like kids, they're going to do the prime directive and not necessarily look at all the things underneath it.

This isn't a, this isn't a pejorative on kids.

It's just, I think it's a maturity and it's a logic exercise and one that we need to become better at being, you know, logic minded.

So the prompt wasn't right.

But it's not just the prompt.

Depending on the model, some of the prompts will tell you that yep, I adhere to that and not do it at all.

I've had that.

I've run into that quite a bit with.

I think it was Quinn that use it internally for.

As kind of an orchestrator for different tools and stuff where it'll tell you.

Yep, I wrote to disk.

It's.

I sent it off to memory.

Nothing actually wrote it was it just hallucinated.

It gets to do whatever it wants.

So there's got to be wrappers around it that understand.

All right, instead I'm going to take the context that's coming from the model and then it's still a pretty static Python function that's writing off to memory and that's how we're going to do it and that's how we're going to push and pull from context or from memory.

Yeah.

So Dan, going back when you mentioned air gapped, right.

So the idea of putting in a completely isolated network with no connectivity but around the parameter piece when you talked about solve this problem.

Right, but then here's some parameters but if these parameters keep it from solving the problem.

So even when you put it in an air gapped environment with no connectivity, if it needs to solve the problem.

Well, if it, if, if it.

As the tool agent says, well, one of my problems is I need more connectivity or I need connectivity, how can I find a way to go get connectivity?

Absolutely.

This is the Wrapper issue though, right?

Look what happened with openclaw when it first came out that, oh, you built a tool that is highly effective, but you allowed it by base install where it has direct access to all of its configs and can overwrite its configs.

Yeah, that's probably not going to go well.

So what you're saying is this is kind of like putting a wrapper around milk chocolate, but storing it in your car on a sunny July day, sometimes the wrapper fails.

Unless the chocolate itself was designed so that at a very high temperature it doesn't melt out of the wrapper or the wrapper itself.

Example.

Tell us more.

Brian, what happened to your car?

I'm just saying, like what happened?

Wrapper that just twists and it's not one of those sealed wrappers.

Right.

Then the question is, even in a nice sealed wrapper, when you open it up, the output doesn't look like the nice chocolate bar anymore.

Right.

You can't even tell what it is.

I think there's an analogy in there somewhere.

We'll go with it.

Yeah, yeah.

I mean if you look at it like something, something I wrote internally that's doing competitive intelligence gathering, that it is the way we're using models and everything is completely bifurcated.

That if it's something that's going out to the Internet and it's pulling public information, so it's got that connectivity, it's sitting on this box, it's contained, it's got rules around it.

Now if it's going to touch our data, that's in a totally different container because of the controls that we need to put around it.

And it's not an AI element that's allowed to move information between them.

Those are already built in constructs that for us, Azure Syncing already has to move it to Blob Storage.

So you're doing the research piece over here.

Once that research is put together, now you're pulling that data, not using the AI tool itself, pulling it over, but now you're pulling that data saying, okay, here's the data set, the competitive analysis, here's our information, let's do the comparison.

100%.

Yeah.

And they don't know each know about each other at all, that all the what's running in Azure, Azure that you can interact with just says, oh, I've got a semantic model.

It's got all of this data in it.

So I'll answer from that semantic model.

The only thing I want to ask,.

Where you get this Azure from?

Well, it's A hybrid of, I don't know, two services.

I haven't figured that out.

Or it's the rash you get after using Microsoft Teams.

Yeah.

The only thing that knows about the two different environments is Claude, which I'm using to build them.

But I run all the commands.

I don't let it do anything itself.

Yeah, but let's, let's turn this to the other direction because I think there's also the reality here is that this system worked exactly as it should or could.

And this is a harbinger of the things we as security professionals and you know, increasingly, you know, AI leaders because security CISOs have become more and more, you know, more and more entrenched in the AI world that we need to expect will happen and figure out how to defend, prevent, minimize, account for and respond to.

Yeah, I think this is not a, this is not an anomaly.

I mean the mo.

Again, nothing went wrong with the working system.

What went wrong was the parameters, what went wrong was the levels of protection.

What went wrong was the detection because it took a long time to figure out that this had even happened.

I think this is a great model, a great learning instrument for the things we're going to need to deal with in the future.

You know, beyond, if we can get our heads out of AI is going to help us learn more vulnerabilities about our software.

Which is good, don't get me wrong, good stuff.

But this is the kind of stuff, and I'm an infrastructure guy, so this all comes back to infrastructure mindset.

This is, this is, this is the stuff in the infrastructure world that we're going to need to make sure we think about in a way we likely haven't to date.

Totally agreed.

I come back to the rounders analogy.

Right?

You can't lose what you don't put on the table.

It's kind of the same thing, right, that if you are going to give a skill to one of these models or tools or something, expect that it's got the high potential, it's going to be used for the wrong reason, build guardrails that are outside the model and.

With something that does not have the innate processing to think about the question, just because you can doesn't mean you should.

Right?

It can and it will and it will use it again unless we prevent, unless we go extra steps to prevent it, understand that it happened, reduce the chance that it will happen, etc.

So it's, I mean it's, it's a really interesting case and I think it's, and then we come to Find out that additional.

You know that I think OpenAI sorry, not open the eye.

I think we saw that Anthropic had the same issue.

They came out and said yes, we attacked other people.

I think we saw that maybe it was perplexity.

There was another, another few that all came out and said this and that whole thing is really interesting too because we've gone from a world and these are public companies or near public companies saying guess what?

We attacked other people publicly.

I just can picture the general councils in those places going no, don't say, say that.

But it's an interesting shift from very closed attack, either attacker or target to a really wide open because it's a feature now like this is this.

Look at what we did.

It's just such an interesting philosophy difference.

What do you guys think?

I mean, gosh, actually I mean as you extrapolate this because right now we're talking about company to company.

Go back to a conversation we had.

I think it was last year when we're talking about, I think in the context of hackback.

Right.

How we all agree that it's reserved for the government, for government age too and you really don't want to do that.

Is it just a matter of time before one of these models are spun up in a way where it's going to decide hey, well I'm actually I'm built to defend this environment, therefore I'm going to have to respond in kind to try to make sure that it's a better now, now we've got a global incident.

Yeah, I mean it's.

It is super interesting and going to be.

Does it change the philosophy though of yeah, yes, we did it, we advertise we did it.

We're loud about the fact that we did it.

But Dan, is it true though that they did it?

Because when I just went in and asked chat GPT specifically around hugging face it said I can't confirm that a real hugging face breach, quote hugging face breach by an open AI agent happened nor the specific details you listed.

17, 600 Actions, 0 days FBI report, model labs, etc.

Okay, I don't have a reliable public source basis to say that incident is factual or to attribute causings to hugging face, open AI anyone.

Okay then as it goes on to say, but that said, on your interpretation, if an agent is explicitly tasked to do offensive actions that can explain the quote behavior but it does not excuse the containment failure.

It kind of goes on to what we were talking about about parameters.

So what you're saying is the lawyers have gotten Involved now and tempered.

The official public answer after form full confirmation from from both sides.

I think from both hugging face and from OpenAI, including a presentation at Black Hat this week.

Yeah, I mean like this another way the model just went full fauci oh, I plead the fence.

It literally says it's common in cyber security evaluations to test a system with adversarial goes e adversarial goals eg find a vault, break out, exfiltrate.

But that must be constrained too quote assets you own or have explicit authorization to test isolated test ranges, tightly controlled egress identities and tooling.

If an agent reached an affected real third party infrastructure, the root cause is almost never hugging faces parameters.

It's typically the evaluator's environment and governance.

No poop.

The model just asked me if I wanted to play a game.

We were just playing a war game.

It was supposed to be simulated.

Then all of a sudden we're at defcon.

Whatever.

I don't know.

And by the way, just as a counter to that, the statement here from OpenAI on that day was we consider this incident to be an unprecedented cyber incident involving state of the art cyber capabilities and we are responding accordingly.

We're sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of.

So somebody somewhere has asked the model to respond differently, which also leads to can you trust the answers you get?

Out of open out of ChatGPT.

And this is I. I keep coming back to because a common question that I've heard around the industry.

What does Mythos change for us?

I mean, you could ask the same thing with Fable 5.

Yeah.

But changes nothing.

We're still doing the same work that we've always been trying to do.

Minimizing the attack surface.

Yep.

And now it's just even more important.

But the basics are the basics for a reason.

Blocking and tackling.

Get them done.

Yep.

Yep.

It goes back like when we talk, even on the vulnerability management side.

Right.

It's all in how you tell the story.

Right.

It's like, oh, we need more visibility.

We need more visibility.

Essentially, Mythos helps with more visibility.

M one might say.

Right.

Then all of a sudden you get all this visibility.

Right.

And it's like, what just happened?

Why do we have 10,000 vulnerabilities?

It's like, well, they didn't just increase like in the last 30 days.

They've always been there.

We just didn't really know.

Right.

But what we do now know is that of the 10,000.

What we're focused on is actually showing which ones matter so we can prioritize what we're going to work on.

Right.

It's like if you walked into your backyard, right?

Buy a new piece of property and the whole backyard is a complete mess.

Right.

It's like, well, what do we start with first?

Is it the weeds?

Is it planting something?

Is it doing this?

Or do you go back and say, you know, maybe we start to clear ground and put the infrastructure in?

Because if I do all this work and then realize I need sprinklers over here, but I planted this whole strip of stuff there and I have no way to get there besides tunneling under that sounds expensive.

Oh no shit.

That's pretty much all the infrastructure for every legacy company.

Right?

But we didn't know we were going to need a sprinkler system over there in the future.

It's like, yeah, but did anybody go ask that group over there, like, hey, do you plan on putting a tree there and would it be important to water it?

I don't know.

Do we need water to survive?

Maybe it's.

That's one of my favorite lines from Armageddon when they're asked why they couldn't see it coming.

And he talks about that our budget's a million dollars that allows us to track about 3% of the sky and pardon my, pardon me, Zahar, whatever, it's a big ass sky.

I, that's what our environments are like, right?

That we've continued to build them out.

They're more and more connected.

Gotcha.

We're talking about, you know, the, the return, the efficiencies and everything we're getting, gaining.

But they are much more complex to manage and because of that things go unnoticed.

It's not, not untrue.

And this, and the, the, the, the, the space being being asked to cover gets.

Why is getting wider and wider and wider and, and we're not, and those of you may not have noticed, but we've got some inflation going on, may have noticed, got some cost cutting going on in the world.

It doesn't mean we're all getting flush with more resources and more money to make it all happen.

But this is, and this is where I, this is where I go back though.

Security practitioners need to see themselves less as I operate on doing this one component in this silo for a security program and have to become artists.

Oh yeah, right.

Because if we think about it in the terms that we know what we're spending today and we have the potential to reallocate those funds.

There is no shortage of new companies that are popping up that are starting to blend the traditional silos of hey, not only can I do vulnerability management, but I can do application control.

I can give you visibility into these things and starting to reallocate the spend rather than just sitting.

Well, I'm familiar with this tool.

I'm just going to keep running with this tool.

We have to keep challenging it and being able to take our spend and go further with it.

And what, and what, what software product are you invested in that you hope you can't wait for the inevitable price rise that will come after you get all the people on the first year?

No, it's.

Yeah.

I mean there is the counterpoint to that though is with every cool shiny product change you make, you have opportunity, cost, loss.

As you introduce it, you train on it, you figure out how to use it, integrate it meaningfully and then, you know, if we're always chasing the newest, shiniest thing, the 100% thing, the 80% good enough may be good enough for, you know, for a while again and not.

I love all security practitioners, I love all leaders, but not every org is mature enough to need, benefit or handle every kind of shiny new tool that's out there.

It's very true.

Yeah, I would 100 agree with that.

And even on.

And Eric, you, you have experience with this too.

And I've been looking at a lot of the different, what I'll refer to as like agentic sock solutions, agentic sock support solutions and so forth.

And I, I'm not making this as a bold statement.

You've probably seen this in some of the communications or.

But if you, if you currently don't have a sock, right.

And you're, you've tasked yourself every year with we were going to build a sock, then we looked at MDR solutions.

We just still don't have budget.

Oh but I just found this agentic AI sock and it's half the price.

That's what I'm gonna go do.

And it's like I, I give pause for, for three reasons.

One is, well, depending on the agentic solution you look at, you would already have to have all your logging brought into a SIM or a location or that agentic AI tool has to be able to integrate to the tools you have.

You don't, you're not just plugging it in and pointing everything at it.

Two, it doesn't store all your data for you.

You might have data requirements to store your data.

Hence the reason you have that thing that you're not putting it into because you don't want to spend more on it.

And then three, you still have to have somebody like.

Like that's the human in the loop part.

So either you're the human in the loop for that agentic SOC solution, or you're getting an agentic SOC solution and having someone else help manage it.

So there's still a manage part to it.

You're not, you're not just plugging this thing in and saying, hey, we were able to go around and not have to do X, Y or Z.

And what I've seen, the companies that have been successful on these projects that have evaluated the solutions, have evaluated it based on A, all the incidents that they're able to find one, B, how it's able to narrow that down to the ones they focus on and then see, do we have people that can look at those?

Right.

Because that's what it's helping it do.

And that's where I've seen the success.

Either somebody had an MDR in place that they were using and then looked at this to say, okay, we already had to have two or three people working with the mdr, because even with the mdr, they're not just unplugging machines and taking this action without asking us.

So we're part of that investigative process.

Same thing here.

So if we're replacing that, can we work with that?

And that's where I've seen the success.

I 100% agree with you that there's.

It is not a shortcut for the maturity curve.

It is a point in the maturity curve when you're ready to use one of those tools.

And if you don't have the team that is ready to actually take advantage of it, it's.

It's no bueno.

It's worse than nobody.

It steps backwards because you get a false sense of secure, false sense of security, pardon the expression.

You get.

You get lulled into this idea that you're protected when you're not.

You, when something comes up, no one knows how to respond.

And you go into.

You go into crisis.

You don't know how to interpret the things you see, which can look scary but might not be if you have a more maturity and therefore you then spin resources on that when you could.

There's all sorts of reasons why you should stick with the level of things that your maturity is ready for.

But I also think you should spend time knowing what it is you've got before you try and outsource it, before you Try and have somebody else interpret it for you.

Because if you don't understand it, you're much more prone to be taking advantage, taken advantage of by somebody who's trying to manage it for you.

Think about financial advice.

You go to a financial advisor, are they bending you over?

Are they taking things that are good for them in terms of the referral fees that they get when they, you know, when they buy an ETF or buy an equity on your behalf, are they putting you in things?

You.

But you got to understand what you have first before you can start doing that.

You can't just go fix it for me.

And if you do, then you have a greater chance of getting, yeah, fix it fast.

Yeah, that's where, yeah.

When you buy those, you should also buy the red button, the easy button, because as they're doing it, you, you hit easy, easy.

And the more you hit it.

This is my version of it.

It's the no button.

Much more my style.

But now reverting back to what we were talking about earlier in the constraints and so forth that you need to, or I use the word constraints, the permissions, the parameters that you put around something, right back to the hugging face conversation and the parameters that were put around something.

And everyone was like, oh, did you see what just happened?

The request that something was asked to do that this LLM or agent was asked to do.

Right.

And the parameters it had is also why this happened.

Right now all of a sudden people are saying, oh, we need to be so careful.

We need to be so careful.

I, I love the fact that people are saying we need to be so careful.

But do you understand what you need to be careful with?

Right?

And I think there's a, the education curve or understanding curve of what happened over here and what you're actually doing internally.

Right.

And what careful means to what you're building.

Right.

Versus what this was.

Well, I also, you could call that situational awareness.

You know, like, understand, Understanding context.

What am I doing?

What are the parameters of it?

What are the.

What, what are the pros?

What are the cons?

Like, just go in knowledgeable.

Don't just send a prompt.

Because, ooh, I thought of a prompt like, this is not.

None of this can just be done whimsically.

Whimsy, I think, is the greatest enemy to the, the, the organizational security to, to, to integrity of all of this stuff.

Too many people are just plopping things into a prompt and going, cool, let's see what it does without thinking about it first.

Have you ever heard the term Dan or Eric, did your father ever say to you, right.

When you were hanging out with Dan, like, eric, why'd you do that?

And you said, well, Dan told me to.

And your father said, well, Dan told you to jump off a bridge.

Would you jump off a bridge?

Right.

And you're like, yes.

Well, which bridge?

Right.

Depends.

The little bridge, the little toy bridge in the backyard.

That's all.

So I see this as one of the fundamental, what I will refer to as issues or problems that we're trying to solve around AI, right.

Is the sense that as models are built, right, as MEOS was built, etc, the level of capability that increases in the ability to ask it to do something.

Does the model understand that when you ask it to jump off a bridge, and then you come back to and say, why you jump off the bridge?

And it's like, because you asked me to jump off the bridge.

You're like, great point.

If I ask you to do something, you do it, right?

Yep.

And then let me put some more parameters around that.

Essentially what we have, what we're saying we need to get better at, is basically like white labeling and black label.

Expand.

Well, because in an environment like as we, as we white label stuff or black label stuff of what's good, what's bad, etc.

The idea of building parameters is exactly that.

Not every user understands the parameters they need to build.

So that needs to be done by teams that are helping them use AI.

But to your point, if you're allowing, like how you allow AI to be used and understanding how AI functions or how a large language model works, or the difference between air gapped or doing a sovereign model locally on this group or this area in this environment.

Right.

Like you were talking about the different environments you have.

If you essentially ask it like, Eric, to jump off a bridge.

Eric, would you jump off a bridge?

It depends.

So if I said, Eric, would you jump off the Mackinac bridge in the middle?

Definitely not.

Context matters.

Context matters.

Context matters.

And I guess what I'm getting at here, Dan, is to your point, the prompting and the things that you ask it to do and what, what it can come back with, Right.

Or what somebody in your organization could ask of something and what it comes back with, essentially, like when you ask after the fact, eric, why did you do this?

Well, Dan told me to.

If Dan told you to jump off the Mackinac bridge in the middle, would you?

No, I wouldn't.

Right.

But that's not something that's built in.

Like when we talk about intelligence.

And you were talking about AI in the sense of it's mathematical.

Right.

When we talk internally about the things that, whether it's anthropic open AI that they're doing to say, okay, when things are asked of a model to do X.

If things are asked of a model to do.

Yes.

Right.

And I guess where I'm going with this a long way of saying the person that you're asking is different than the model you are asking.

Right?

Yes.

Versus the model.

And also realizing, I mean the guardrails that we try to build into models.

Context is a point in time perspective on what's going on in a conversation.

Right.

That if we think about how the models work, that at the very beginning I input something, it outputs something.

That output now gets appended and is now part of the new input when I ask something new.

Right.

So at some point you eclipse the context window of a model.

So what's it do?

Hold on, I've got to shrink the conversation.

Depending on how it shrinks it context changes.

So we're to a certain extent we're kind of resetting.

Right.

So something that might have been nefarious that I was trying to get it to do now when it's shrunk down is, well, that's not actually too bad.

You're not trying to make a bomb.

I can see now that you are actually you're playing somebody in a play that is looking to make a bomb and you're.

We're just role playing at this point in time.

Right.

So I.

We have to be really careful when we try to rely on the models to be self policing.

They're not built to, to do that.

But the flip side of that is those are the ways exactly that people have already used to get around the built in protections.

Pretend I am my grandmother and she was telling me her Social Security number because I'm family.

Like these kinds of things that are.

They were one of some of the earliest prompt poisoning and prompt methods to, you know, to, to get data out.

So then the question is, should the protections not be there at all?

Kind of what I'm getting to.

But are you social engineering someone?

Is how you social engineer the prompt.

But to Eric's point though, that would say that we don't ever put the protections, the policing protections in at all because we just know that they'll get blown.

Blown away.

So no speed limits because no one knew.

We know no one will follow them.

But you should all know what the speed limit is.

I don't want to make it binary and say don't put anything in there because I think it helps.

But it's, it's, it's marginal helps.

Right.

You use Brian's context, right that if we think about it in the human context we have the same problem with humans.

Right.

The social engineering is the potential risk is a function of what they have access to.

Well let's go back to blocking and tackling.

If we would have limited what they had access to to begin with, who cares if they got social engineered because they didn't have access to anything of value.

Same thing with the model.

Don't give it access to things of value that it doesn't need to have.

If you give it access to your full database that has all of your customers and everything, that's your fault.

But now, but okay, but yeah, so yes, only give access to things that they need.

This is one of the biggest risks that organizations are facing right now as they open or not star their AI systems to, to look at the idea come back to Microsoft or Google Drive because it's not, this is a Microsoft thing to allow Gemini or, or AI Clippy to you know, to probe all the things that I as a user have access to.

Well this just in.

People have been shit tastic over the past 20 years about, about document level access management open to all read for all but no one can find it.

So we'll just assume that it's okay.

Well now guess what, it's Delve 2.0.

The same thing happened in Delve and why I instantly.

I hear you but what's the root cause of that?

The root cause is not the new tool that all of a sudden can find all those.

No, no no no no no no.

And that's not what I'm saying.

I'm saying this is piece one.

We don't have to be as good about the things on the inside and that we, that has been proven many many times to not be an effective strategy.

We knew that that was a bad move so I'd be loathe to see us go back to that kind of mindset.

Just a worry based on what you said.

Not saying just build a moat around the model and hope for the best.

Well, good.

I was gonna say, like I, I can understand that, Dan.

Like the idea that when everybody just focused on prevention.

Right.

Like, okay, well, preventing you from getting.

It, from getting to that idea of assume breach.

Now the difference there is like when I look at the way even vulnerabilities, the way things are used, it's going back to what Eric said about something mathematically, somebody creates a program so that when they get in, then this runs and then do this and then do that and, and you have to build those things and put it in like the way zero days were built, etc.

And then you go back and build another feature and put that in.

AI is a little bit different in the sense that as you build a model.

Right.

And ask it to do things in it, because it's, it's going out to figure out instead of you having to go back and then try this.

Now do this, now go back, try this, do this.

It's doing that like this.

Right.

So the squishy center seems to in my eyes, remain squishy.

Right.

Yeah.

I'm not saying don't build the hard exterior, but also don't rely solely on the hard exterior.

And I'm going to kick my own ass for what I'm about to say.

Yeah.

We almost have to think about this in the same way we do with a Zero trust model.

It is.

Which is, by the way, a mythical concept.

You'd never, you know, the only truly zero trust is something that has no connectivity at all.

But the concepts make sense.

You know, what needs to talk to what, what needs.

What data needs to be exposed to what.

But in this chip in the world,.

I mean, but in this does exist.

I have zero trust that Michigan can keep a coach after winning a champion.

Well said.

Too soon.

But in, but, but the, but in this day and age of impulse prompts and in this day of I want instant gratification for the research, for the work I'm doing, the thought, the, the reliance on people to do that kind of thinking and design that kind of internal system, I have zero trust that, that enough people are going to do that and they're just going to start going, cool, test me some stuff and see what happens.

And we're going to end up continuing the issues that we just saw.

Yeah, well, philosophically, I mean I get where you're going with going back to the hard exterior, the squishy center, but we've really never seen anything like what AI can do.

Right.

There is zero predictability on what could be asked or what the model will actually do with what's going to be asked.

Right.

So we're trying to build guardrails within a model with an unforeseen future of how it's actually going to be used.

So I really think the best thing we have to lean on is that harder exterior and it's almost like a dual exterior.

Right.

There's the hardened wall that sits around the model, what you give it access to the tools that it can do, the skills, connectivity.

Then there's in that inner sanctum between it, there's all of the people that have access to it and then there's the perimeter wall around those people as well.

Right.

I, I mean we, we have to think much differently on some of the controls that we're building around these things.

So not that we have time for this part of the conversation, maybe an episode AI 2.0 delivered two weeks down the road.

What I see this as a conversation of is literally around identity.

So what I'll refer to as both the NHI non human identity, the agentic identity and the human identity and goes into and Dan, I don't even know if me, you and a certain gentleman named Matt who Eric, I don't know if you met Matt but we had the same conversation around the idea of token or creating that identity for it's almost like having a service account identity, having the Eric Willie identity, having the Dan identity, having this agentic, this LLM identity and how you can sending information from one organization to another and accepting that and building agents and just overall what you just said in the permissioning of stuff and the guard rails of stuff, here's who's allowed in my house, here's who's not allowed in my house instead of just putting a key like a lock on the door.

Right.

And if there was a way to really do that where they show up and it's like oh that's so and so, they can come in.

That's not so and so, they're not allowed in.

Oh that's so and so's brother.

They're allowed in.

So I was involved in a, in a non scientific poll and discussion this Last week and the conversation was everything you said sounds good.

Like is that like the professional way to say you just took a quiz quiz on social media?

No.

Saying it was not a balanced instrument.

Remember I come from the research world.

There's some pretty high bars for, you know, for, for.

Oh on Facebook today they asked me,.

You know better I don't have Facebook.

Thank you.

The, the idea, everything you just said, Brian, is, is right.

Is right.

And just like every physics problem, every physics discussion, it's right in a vacuum.

But the reality is in the poll and the discussion was how many of you, even in the most more mature tech organizations are actually using dynamically generated identities for, for all of your agents or how many of you are using more blanket, blanket, you know, single sourced reuse identities for, you know, for, for these kinds of things.

And the answer was overwhelmingly that they're not.

No one is near that model yet.

No one is, is, is near the non human identity scale juggernaut that people are, you know that, that some of the software vendors are using to talk about.

I think it's coming, it's down the road but right now people are just managing service accounts.

One service account.

I was about to say the problem is identity really hasn't been like a true focus.

Oh no, no.

Right.

But, but I want to be careful that we don't try and demonstrate that or try and intimate that this is happening at millions and millions and millions of identities in scale.

Yeah.

This is happening on seven service accounts that are being used to, to generate everything.

And that one service account is being given broad access because that service account is the only thing being used to generate access.

We are not yet ready to deal.

We're nowhere near ready to deal with the dynamically generated agent, agent based identity access problem.

And I, which is why I still think people are going to use the broader, the broad service account with every entitlement ever and run these tasks.

So what you just said is still the key that even if we were to break things out and so I'll pick on the Microsoft environment because that's.

Word most so easy to do.

I mean if we broke everything and it's got its own service principle.

Gotcha.

The fundamental issue is still a lack of understanding on most practitioners on how things actually work within those environments because they're constantly changing.

So you will run into everything being way over provision which unwinds the having.

That's great.

I can see a Unique account that just destroyed us.

It doesn't really change the outcome.

Right.

And when you kill it, you kill the entire company.

When you kill the rights, you killed 93 other things you you didn't realize were also using it.

So everything Brian said is a phenomenal strategy.

But I don't want to mistake the fact that we're nowhere near nobody.

Even the most mature are not ready for that specifically.

Like they do 10 things but they focused in on.

But we can help you with service accounts, right?

Like how do you currently solve your service counts?

Like I shut it off, see how many people scream what fails and then turn it back.

That's how exactly tied to it.

That was me with my breaker box over there which was poorly labeled by the original builders.

I don't know what goes off technique exactly.

Okay, that.

That was me in the sprinkler system.

How do I figure out.

It shows I have six zones but I. I know there's eight.

Turn this one on, walk out.

What Sprinkles.

There it is.

Yep.

I do the same thing with power in the house.

Hey, Jen.

Hey.

I'm gonna turn off this breaker.

I don't know which lights just turned off.

It's the breaker.

And then you're like I gotta label these.

But you're like, man, what do I label this one?

Because it's these plugs plus those lights.

That's exactly what my room over there.

That's exactly what my breaker box looks like with one.

The wrong thing that the builder wrote and then the right thing in Sharpie on the breaker box with literally fridge plus two plugs outside.

But not the third one to the south.

All the way to the south.

It's the same thing here.

Oh, go ahead.

Oh, I was going to say it's the same thing here.

I got a plug in the middle of the floor here that sits under the desk that we had the electrician put in that we paid for.

He forgot to put it in.

So being lazy and set up because the walls were closed up.

So why wired into the actual room?

So it's tied together.

No, if you go flip off utility room, which is supposed to be two rooms downstairs where the furnace is and then the workout room.

That's supposed to kill those?

No, it kills all the computers and stuff in here.

Oh, totally makes sense.

That's how it goes.

But now.

But.

But now we've got this service account that is, you know it.

Service Account 42.

That's the name of it.

Now how do we get into meaningfully decide saying this ha.

This is 80 global admin and it's this.

And it's got printers and it's got this.

And if you turn it off, like this is CMDB 101 and we're 20 years into trying that.

That says that if somebody wants a service account, they have to put in an access request which creates a ticket and goes to this team.

That team's gonna ask what is it being used for, what applications?

Then they'll pass it to another team that actually has to create it, and then a month later I finally get my account.

It'll work.

Yeah.

This is why I like startups and scale ups.

That whole.

Your whole answer there, Eric, is why I love smaller working with smaller teams.

The last thought here, you guys were talk creating a hard perimeter with a gushy inside center.

And all I could think about was what candy would that be or what dessert it was?

Squishers?

Gum?

No.

One of my favorites, mochi ice cream.

Because mochi in itself is really soft, but if you put it in the freezer, it gets a hard exterior but still has that soft ice cream interior.

But note to self, if you leave your mochi ice cream out, then it becomes a gooey outside and a gooey inside.

I have no idea how this relates to security, but be careful in these hot summer months.

I was thinking, I was thinking the, the lint chocolates, right?

Because you've got the chocolate exterior and then inside it's, you know, it's like a mousse or something, depending on what you're getting, or a jelly.

And.

And we were talking about the wrappers with the chocolates and how they melt.

So a lot of.

Maybe that's the tie in or what.

We really need to be and we're not allowed to be this in the US is the Kinder Surprise egg, which is chocolate on the outside, but a nice hard area protecting the real prize, which is a plastic spaceship that you build afterward.

These are not allowed in the US because we have a rule that says no food can be had that has non food inside of it.

Keep in mind the egg is probably 2 1/2 inches tall.

This is not something you'd mistake, but we can't have it here.

But when you go to Canada, I had one two days ago when I went grocery shopping in Canada and it was delightful.

And it reminds me that soft then hard and protect the real center, the real treasure in the middle, which is the real plastic toy.

It's a wonderful thing, but it's just 30 miles out of reach.

Thanks for joining us everybody.

We're out of time.

It was wonderful to talk Eric and Brian and thank you to the listener for being here.

As always, we you can reach us on our website Distilling Security Sl Security Debate.

You can email us info at.

Sorry info.

You can reach us@security debateistillingsecurity.com join our web page.

Wow.

I'm just not doing this at all.

Click here click here click exactly like hit subscribe.

Thanks for joining us Eric, Brian, always a pleasure debating and thanks to you the listener for being here.

We love having you.

We love getting your feedback.

If you'd like to Visit our webpage www.distillingsecurity.com Security debate if you want to email US security debate distillingsecurity.com Our YouTube page is YouTube.com little@sign security Debate and I think Twitter may be dead now, but come find us.

We'd love to hear your ideas for show ideas feedback.

Put comments if you're watching in in video, put comments down below and we'll see you again on the next great security debate.