Today's discussion centers on the vulnerabilities associated with AI systems and the increasing threats they face. Our guest, Preston Wood, the Chief Security and Strategy Officer at Databox, highlights the lack of transparency in AI technologies as a significant factor that makes them more susceptible to attacks. We explore how this obfuscation creates challenges in understanding and defending against potential threats. As AI continues to advance, we also consider the evolving nature of phishing attacks and the importance of robust data management strategies to mitigate risks. This episode aims to provide insights for software architects and leaders on navigating the complexities of AI integration while ensuring security and reliability.

The podcast episode features an insightful discussion about the growing vulnerabilities associated with AI systems. The guest, Preston Wood, the Chief Security and Strategy Officer at Databox, addresses the surge in AI-related attacks, emphasizing the need for greater transparency and understanding of AI operations. He explains that the ambiguous nature of AI systems makes them appealing targets for attackers, who can exploit the lack of visibility into how these systems function. Throughout the conversation, Preston highlights the importance of ensuring that AI-generated data is clean and comprehensible to mitigate risks. He compares today's AI landscape to early phishing attacks, which have evolved into sophisticated threats due to advancements in AI technology. This episode serves as a crucial resource for software architects and technology leaders, offering them guidance on how to navigate the complexities of securing AI systems and understanding the implications of AI on data management and security practices.

Takeaways:

1. The podcast discusses the growing vulnerabilities associated with AI-based systems due to their lack of transparency.
2. Preston Wood emphasizes the importance of clean and understandable data for AI performance and security.
3. Organizations are advised to improve their data architecture to ensure AI projects are successful and not hindered by poor data quality.
4. The conversation highlights the evolving nature of phishing attacks, which are now more sophisticated due to AI advancements.
5. Effective security requires a layered approach that combines model training and guardrails for AI systems.
6. Listeners are encouraged to consider how well their organizations are integrating AI into their existing technology frameworks.

Hello and welcome to Software Architecture Insights.

Your go to resource for empowering software architects and aspiring professionals with the knowledge and tools they require to navigate the complex landscape of modern software design.

My guest today is Preston Wood.

Preston is the Chief Security and Strategy Officer at Databox.

DataBonded is an AI powered data pipeline management platform for enterprises to gather, manage and move data reliably.

Preston, welcome to Software Architecture Insights.

Thanks for having me, Lee.

So as AI use has grown over the last several years, obviously so has threats against AI based systems.

What can you tell me is the biggest thing that's made AI based attack vectors so, so vulnerable?

Well, I, you know, I think it's a great question.

I mean we certainly have seen just the explosion of AI over the last several years.

I think we're going to continue to see these type of attacks continue to sort of expand and largely, I mean, there's a number of reasons why, but one of which I think is worth mentioning is really the lack of transparency and visibility around what is actually going on within, you know, your, either your AI infrastructure, your AI decisioning, what are these models actually doing and how they're functioning.

And because of that, you know, you certainly have, you know, products such as ourselves that are driving towards making sure that the data that you are generating and leveraging in AI is clean, it's understandable, it's transparent.

So I think a lot some of the problems kind of rely upon the fact that, you know, AI is new, there's not a lot of transparency into how, you know, some of these agents are actually functioning.

And so, you know, attackers are able to kind of leverage upon that shiny new coin.

So I know sometimes people will think AI is just another technology, right?

And so it's really no different than any other technology when a new technology comes out, we need new ways to protect it, et cetera.

But what you're kind of saying is AI is a little bit more than that simply because it's not at all clear what AI is doing under the hood.

So there's a lot of unknown involved in that that makes it harder to protect.

Is that kind of what you're trying to say?

Yeah, that's One reason, sure, 100%.

So how does that make it harder?

I mean, what is it about the fact that we don't understand how the technology works that makes it easier for bad actors to attack it or more likely that they're makes it a more likely target for them to attack?

Well, you know, obfuscation has always been a, a tactic in a hacker's toolbox.

You know, the less, the less some, less something is transparent, the more obtuse it is.

You know, hackers can usually leverage that lack of understanding to drive, you know, some level of attack or drive a user to do something that they normally wouldn't do.

And you know, given kind of the world of AI we're seeing, it's becoming very, very hard to differentiate between what's being generated, AI versus what is not.

Some of this you could actually kind of point back to the early days of phishing campaigns where phishing emails were sort of crafted and sent into organizations to click on something and become a point of compromise.

And early days those phishing emails were easy to spot or misspellings and bad domains and these type of things, but they still caught people because there was an element of wanting to click on that link.

Today's phishing emails are largely driven by AI.

They're clean, the language is good, URLs are good.

And becomes incredibly difficult for users and organizations to differentiate.

2 the, the obtuseness and abstractness of what's going on with AI and how it's making decisions and actually what it is providing back to you.

And I think we've all kind of heard of the, the concept of hallucinations.

The AI will, you know, send something back that sounds very real and very convincing, but it's a, it's a hallucination.

And so how AI is actually generating what it's generating to, you know, be leveraged within an attack is unknown sometimes.

And so if you don't understand exactly how that attack is occurring, it's difficult to position yourself in a place that you can thwart that attack.

And, and then you have the speed factor.

So, you know, the data and information and intelligence is moving very, very rapidly.

And as attackers can kind of leverage AI to iterate through, you know, developing a world class phishing email or some sort of exploit or adapt on the fly.

I think there was actually just a release by Anthropic where they detailed an attack that they witnessed as part of leveraging AI agents.

Yeah, I think one of the things I worry a lot about is the dramatic improvement of phishing attacks in particular, you know, there's AI is really good at talking to you and having decent level conversations with you.

And the fact that it can do that means it can pretend to be whoever you want it to be.

And whether it is or not is another matter.

I was talking to a customer support agent for my home Internet connection the other day trying to solve a technical Problem that rebooting the modem didn't work and they were going through some setting adjustments and all that sort of stuff to get it working.

And I was talking to this agent for about, oh, 20, 25min, and then I suddenly realized that it was an AI agent and going through some very sophisticated debugging techniques and it was so good at its conversation is.

I didn't know it was AI until about 15, 20 minutes into the conversation.

That level of sophistication at that point I thought to myself, now if this was a phishing attack of some sort, it wasn't.

But if it was, how could I possibly detect that as a mere human to know that this high quality AI with a high quality ability have rather sane conversations with me?

How could I possibly know it was an attack?

Yeah, it's becoming increasingly difficult just pulling on that phishing thread a little bit.

As I mentioned back in the day, the signals around identifying something as a phishing email was much easier.

Misspelling again, bad domains, links that just didn't look right.

Today, those signals are becoming less and less.

And so the ability to be able to leverage technology and in many cases actually leverage AI to tear apart some of those things in real time so you can find some of those signals.

But for end users, users, there's always sort of those telltale sort of signs of, well, do you know who this person is?

Does the email look appropriate?

Does the link you're clicking on look like it's supposed to look?

I think in the age of AI here, it's really driving people to be more attuned and suspect to the information that they're seeing.

Yeah, there's always this race, right, between technology for bad actors and technology to detect what bad actors are doing and which one's further ahead.

So AI also can help with detecting these attacks and be very useful from a white hat standpoint as well too.

But how is that race going?

Are we improving our technology, for instance, to detect that a voice is AI generated versus real voice?

Are we getting better at our ability to detect faster than the ability for it to be undetectable?

We're getting there.

But you're absolutely right, it's a race.

And information security and security in general has always been this race of, you know, bad guy advances and good guy, you know, has to run to catch up.

And you always have this leapfrogging effect that occurs.

And we're in that space right now where the technologies and the abilities to be able to detect and flag, you know, these, these things on the fly is getting better, but still obviously needs some improvement as this is such a fast paced environment.

And I think we're all familiar with Moore's Law with regards to chips.

I wonder what Moore would think when he saw the evolution of AI.

That's true, that's true, exactly.

I'm thinking a lot of different dimensions with that, but absolutely that's a valid point.

Now, you mentioned something that I think you coined the term hands off AI or you didn't coin the term, but you use that term in some of the things that you talk about and that most organizations aren't ready yet for hands off AI.

So first of all, can you tell me what do you mean by hands off AI specifically so our listeners know exactly what we're talking about here?

Well, I don't know as much as hands off.

I think there's a couple of ways to sort of think about AI in the terms of how it's leveraged.

And I think that this ends up being a, a journey for organizations.

And organizations are on different paths of this journey right now of leveraging AI to answer some basic questions.

And this could be your customer service prompts, these type of things where there's a deterministic kind of outcome and then you start sort of moving into non deterministic outcomes where AI is sort of generating more paths that weren't necessarily determined upon input.

And in those cases, I think organizations are putting a human in the loop around approving certain high risk activities or transactions that the AI ran down a non deterministic path so that the organization, you know, gets comfortable that they've got the right guardrails around.

Yeah, I'm getting the right information.

This looks like a good path to take.

Okay, I'm going to approve this to move forward.

So I think you see, organizations are putting humans in the loop around before you kind of move into full sort of agentic or hands off AI where you're comfortable with the cleanliness of the data that you're sending the AI, you're fully aware of what the model is doing.

And even though there may be some deterministic, non deterministic paths it's going to take, you've got the right guardrails in place to prompt humans in the middle or otherwise to truly kind of be in this agentic or sort of hands off AI and organizations are in varying degrees of that journey right now.

And we'll continue to kind of see that comfort grow.

I kind of relate this back to cloud computing.

As cloud computing became on the scene There was a lot of concern around how do we secure the cloud, how do we manage this?

And so organizations sort of tiptoed into that to where we sit today.

There's an understanding of, you know, how the cloud works and how to secure it and how to manage it.

And we're seeing the same sort of journey with AI right now, but it's moving, in my opinion, much, much quicker than we saw cloud.

Yeah, no, I was very heavily involved in that, in that cloud.

How do you secure the cloud Mindset.

And I can almost tell you exactly when the transition occurred from I can't trust the cloud to I can't trust not using the cloud.

You know, it was an amazing transformation occurred.

So you suspect the same sort of thing will happen with AI at some point in time?

Yes, absolutely.

Absolutely.

We're just in that journey right now of opening transparency and understanding and learning and understanding what these guardrails need to be for AI.

So I hear you talking about guardrails a lot.

Is guardrails, is that the solution or is better model training the solution, or is it a combination?

Yeah, I wouldn't say there's a silver bullet here.

Model training, you know, 100%.

How, how is your model reacting?

How is your model being trained?

You know, what alignment is that model having that's equally as important to the guardrails.

And those guardrails are a number of different things around, you know, humans in the loop, you know, managing, you know, prompt engineering, you know, sort of managing, you know, what is going in versus and what is coming out of, you know, sort of sort of your AI, you know, in addition to, you know, managing sort of the speed and performance thereof as well.

So I wouldn't say there's a silver bullet in any of this at this point.

It never is with security.

There's always a layered approach with security.

So, you know, it's these guardrails around transparency and understanding how your.

How your AI, you know, is working to, you know, a number of those other, you know, aspects of, you know, securing the ecosystem.

Yeah, I know one of the things that a lot of people talk about now is using AI as a guardrail against AI and, you know, using, you know, some of those guardrails, as you, as you say, are, you know, inbound data, transparency of data filtering and output, et cetera.

But some of those are actually using AI to analyze what you're doing, what the.

A, what another AI is responding to, how.

How effective is that, and is that an important aspect and how will that evolve over time?

Yeah, no, I think that that will be an important aspect as well.

It's kind of your A B testing, right.

Of AI in that, you know, any of the sort of the AI work that we're working on.

Yeah, we, you know, bounce, bounce that up against various, you know, large language models, the open ones, and then some of the more, you know, the smaller ones out there so that you kind of understand, you know, here's, here's my, here's my expected answer to these sort of questions.

And here's what the AI, here's what these different AIs are providing and kind of understanding why that is the case and making sure that you're kind of aligning to the right models that can reach your outcome.

So I think the concept of quality testing, of ensuring that the LLMs are delivering what you expect and at the same time bouncing that up against other sort of models to understand, you know, if one's hallucinating or one is not, or, you know, one's got an alignment slant that doesn't, you know, align to the questions you're asking.

I think, you know, leveraging and having the understanding and skill set to, you know, test your hypothesis and decisions across multiple LLMs I think will become the norm.

You know, so if I look at the people who are listening to this podcast, they're, they fall into two camps.

They're either senior leadership or senior managers at enterprises, or they're software architects or software development leaders, all very much technology focused positions for those individuals.

What advice can you give them as they go forward with their AI integration strategy into whatever projects they're working on now?

Well, I mean, a couple of things come to mind, you know, so the first is the way I like to think about, you know, how AI has kind of really risen is, you know, for AI to actually function, you need three things.

You need compute, you need models, and you need data commute.

Compute obviously is becoming, you know, much more commodity, but, and there's more being kind of built to leverage.

We understand how compute works.

Models.

A lot of these models are becoming very powerful, some of which, a lot more open models are out there.

The ability to leverage some of these LLMs and models is becoming more accessible to more organizations.

Then lastly is the data.

The AI is only going to be as useful and valuable as the data that you put into the AI.

So I would encourage senior leaders in architectures to take a look at, take a look at what does your data architecture look like today?

Do you have the ability to AI ready your data within your Data architectures or is your data architecture sort of stuck in the, you know, the 90s and hasn't been touched because, you know, nobody knows how it works anymore and they don't want to break anything.

So, you know, taking a close look at these, your data architectures, and if you've got a monolithic sort of data structure and architecture in place, that's one of the areas to take a look at.

Because if you can't sort of leverage some of the modular approaches around managing and cleaning and transforming and normalizing your data so it's AI ready, then you run the risk of any of your AI projects being a failure even before they start, because you're not using good clean data.

I'd like to end with a different sort of question.

App observability is probably one of the hottest areas right now for maintaining the integrity of a running system.

Right?

You use application, whether you're trying to talk about logs or analytic data, whatever that is, observability data is really becoming more and more important to keeping and maintaining a high functioning SaaS application.

But that data you use for that observability falls into two camps.

It's data that you'd inherently use for performance, availability, scaling, those sorts of keeping your application working.

And there's data that you use specifically for security and vulnerability detection and validation and verification and threat detection, etc.

And I guess the first question is how similar are those two types of data?

I think the more important question is what role does AI have in creating and using that data?

Let's say focus more on the using side of that data in order to keep your application safe.

And how does that radically differ from observability data used for detecting failures and maintaining performance?

Yeah, it's a good question.

We're certainly seeing that uptick in observability and customers and others that are reaching out, asking around how do they better manage their observability data with the pipeline?

And so I would echo that same sort of sentiment.

We're seeing the same type of interest there.

But I'll answer your question in the context of data is just data.

And the reason I say it like that is that how you leverage and what analytics and lens you apply to that data is what's going to drive your decisioning process.

And so as an example, a security team might want to understand sort of the peaks and kind of valleys of CPU performance and file system accesses.

And some of these things are sort of deemed as observability.

But in reality, if you apply A model that's looking for abnormal behavior from a security perspective.

You know, sometimes, you know, some of these systems that may have, you know, become compromised don't manifest themselves in being compromised by way of some of the traditional security ways.

You see some of these other performance indicators on your system, and the same is true on sort of the it.

And the observability side is oftentimes some of these teams lack kind of the visibility into understanding what's going on in their firewall environment.

And I've certainly seen my fair share of some sort of outage that happened to be related to some security change that IT teams and observability teams were not aware of.

What I'm really advocating for here is really the convergence of a lot of this data so that it can be modular and repurposed and leveraged for different types of analytics and pulling out of this constrained data and tool centric silo to kind of really within, you know, as a pipeline sort of provides is, you know, that centralized AI ready fabric that blends all of this information and allows your analysts, whether it's your observability side or your cybersecurity side, to leverage that data to make the best decisions.

And so when you can start stitching together, you know, data that is traditionally used in some of these other disciplines in such a way that you're providing more context to these LLMs and AIs to make better inferences and pattern matching for better decisions, leveraging your AI much more broadly in your organizations and moving away from some of the silo approach data architectures.

Yeah, and you bring up good points.

Almost every time there's a security problem, it ends up being a performance issue.

And almost every time there's not almost every time, but many times when there's performance issues, it's also related to security issues that are currently going on.

And so by correlation between them, the different types of data, you get much better results.

My guest today has been Preston Wood.

Preston is the Chief Security and Strategy Officer at databond.

Preston, thank you so much for joining me today and Software Architecture Insights.

Thank you so much, Lee.

Thanks for having me.

Thank you for joining us on Software Architecture Insights.

If you found this episode interesting, please tell your friends and colleagues you can listen to Software Architecture Insights on all of the major podcast platforms.

And if you want more from me, take a look at some of my many articles at Software Architecture Insights.

And while you're there, join the 2,000 people who have subscribed to my newsletter.

So you always get my latest content as soon as it is available.

Thank you for listening to Software Architecture Insights.