In today's episode, we're thrilled to have Niv Braun, co-founder and CEO of Noma Security, join us as we tackle some pressing issues in AI security.

With the rapid adoption of generative AI technologies, the landscape of data security is evolving at breakneck speed. We'll explore the increasing need to secure systems that handle sensitive AI data and pipelines, the rise of AI security careers, and the looming threats of adversarial attacks, model "hallucinations," and more. Niv will share his insights on how companies like Noma Security are working tirelessly to mitigate these risks without hindering innovation.

We'll also dive into real-world incidents, such as compromised open-source models and the infamous PyTorch breach, to illustrate the critical need for improved security measures. From the importance of continuous monitoring to the development of safer formats and the adoption of a zero trust approach, this episode is packed with valuable advice for organizations navigating the complex world of AI security.

So, whether you're a data scientist, AI engineer, or simply an enthusiast eager to learn more about the intersection of AI and security, this episode promises to offer a wealth of information and practical tips to help you stay ahead in this rapidly changing field. Tune in and join the conversation as we uncover the state of AI security and what it means for the future of technology.

Quotable Moments

00:00 Security spotlight shifts to data and AI.

03:36 Protect against misconfigurations, adversarial attacks, new risks.

09:17 Compromised model with undetectable data leaks.

12:07 Manual parsing needed for valid, malicious code detection.

15:44 Concerns over Agiface models may affect jobs.

20:00 Combines self-developed and third-party AI models.

20:55 Ensure models don't use sensitive or unauthorized data.

25:55 Zero Trust: mindset, philosophy, implementation, security framework.

30:51 LLM attacks will have significantly higher impact.

34:23 Need better security awareness, exposed secrets risk.

35:50 Be organized with visibility and governance.

39:51 Red teaming for AI security and safety.

44:33 Gen AI primarily used by consumers, not businesses.

47:57 Providing model guardrails and runtime protection services.

50:53 Ensure flexible, configurable architecture for varied needs.

52:35 AI, security, innovation discussed by Niamh Braun.

Hello, listeners. And welcome back to another thrilling episode

of data driven. In today's episode, we delve deep into the

fascinating and, let's be honest, slightly terrifying world of

generative AI and security risks. Joining us is

Niamh Braun, co founder and CEO of Noma Security,

who's on the front lines of keeping your AI driven project safe from

digital mischief. So grab a cuppa and let's get data

driven. Well, hello, and welcome back to Data Driven, the podcast where we explore

the emergent fields of AI, data science, and, of course, data

engineering. Speaking of data engineering, my favoritest data

engineer in the world can't make it, today. But we

have an exciting, conversation queued up with Niv Braun,

who is the cofounder and CEO of Noma. Noma

is a security firm that focuses on effectively

he'll describe it more eloquently than I can, but effectively thinks about

security in the context of data and AI across the

entire life cycle. Welcome to the show, Niv. Hey,

Frank. Happy to hear you, bro. Yeah. It's good to have

you. And and security is one of those things where I've been thinking about more

lately. Right? So my background was a software engineer and,

you know, software engineers historically have not thought of

security. Then I made the transition into data engineering and data

science, and, traditionally, security is not really at top

of mind, for them either. Now I

kinda look at this, and I kinda look at the landscape that we're in where

enterprises are deploying LLMs,

generative AI solutions, on top of the predictive AI solutions,

fast and furiously, and not thinking about

security ramifications. So what are your what's your take on

that? 100% agree. I think that, it's

even like the the the the current, like, timing is even more fascinating

than the than just, like, a new technology. Because exactly like you said, like,

Frank, like, we all like the data practitioners. We all know that, like, security is

not, like, our top priority. And by the way, like, by, like, like, this is,

like, how it should be. Like, we are focusing on the business and, like, drive,

like, drive, like, the business forward. And this is why we're, like, this is

what we're paid for. The problem is that

because we're not, like, in this kind of, like, mindset, we also, like, like

any technologies in the company, also, like, create some risk. What we see right

now is the LLM drive, which is pretty cool, is that for the

first time, the security teams started to put

the focus and, like, the spotlight on the data and AI teams. Because until

now, let's be honest, they were focusing only on the software developers and

their SDLC and the CICD and all these areas. Like, we were,

like, you know, like, in the shadow. And we were, like, able, like, to act

like exactly like, like, like, completely freely as we wanted.

But now when, like, the security team start, like, to put the spotlight on the

data and AI teams, what they understand is that it's not

only this new kind of LLM threats, but also all

the basic principles of security are not implemented

in the data engineers and the data science teams. Nobody, like, scans all the

code in our notebooks, for example, unlike the software developers that, like, all

their code is being scanned. Nobody helps us to

find configurations in our data pipelines or our

MLOps tools or our AI platforms, like Databricks, for example.

Like, nobody, like, provide us this ability to to find it easily,

unlike, again, the software developers that they receive all this coverage

and everything. Like, on the moment that they have, like, the smallest misconfigurations

in their SCM or their their CICD, they

will immediately, like, receive, like, a notification, like,

helping them exactly, like, how to secure it. And also eventually,

like, in the run time, in the runtime, in software life cycle, in

classic like software application, we also have a lot of API security and web

application firewalls tools that help us to protect the application in the

runtime. But now specifically in LLM, this is, like, very

related also, like, to what you said. Like, there are new kind of adversarial attacks,

all the prompt injection and model jailbreak and stuff like that.

And, again, nobody, like, else would like to protect it, like, in real time. And

I think that this is, like, one of, like, the main shift that we see

today in this area. We understand that the spotlight

moved to the data and AI teams, but we need to make sure that we

do, like, both. Like, we start with, like, a new kind, like,

trendy, like, risk that we want to make sure that we are protected from.

But also that for the first time, after a lot of years, we're

starting also, like, to implement the basic security measurements

needed in our area. But the most important thing, of course,

is to continue and, like, do it without slowing us down. Like, we need to

make sure that, like, everything, like, all the different, like, security measurements that

we take still provide us the ability to move fast, to enable

the data sent the data science and the data engineering teams to

continue and, like, innovate, but in a secure way.

You know, that's a good point because I never thought about scanning a notebook for

errors. Right? Shame on me. Right? Like for code

security I mean, not errors, but, you know, security vulnerabilities. That's not something

that I have seen done in practice. I mean, the the

closest I've seen where security has been an issue for

anyone in this space is,

basically using protected, you know, Python

libraries, right, or or Python library repos, right, where they're those

are scanned by, I forget the name of the 3rd party that'll do it where

you just basically say you point your Python instance to there. Yeah. Because

I also think that Internal Artifactory. Yes, exactly. So

like, what, because I often

wonder, you know, people just like to install.

God only knows what's in there. I can tell that, like, it already, like, happens.

Like, I don't know if you heard, but for example, like, like,

like, pretty recently, PyTorch, for example. Right.

PyTorch that we all know was compromised. We all know and love. We're most people

love. It was compromised. Like, specific version of PyTorch, a

malicious actor succeeded to to put some

code inside that basically,

collected all the the the secrets and token that you have in the

environment and sent it to DNS. Now we all

know, like, how much like like, how many downloads, like, PyTorch have.

And most times, where PyTorch is downloaded to through, like, to

all these different, like, notebooks, wherever they be, JupyterOps,

SageMaker, Databricks, like, we all use them.

And it I can tell that, like, it caused us to a lot of, like,

problem. I can tell, like, like, like, firsthand, like, we saw, like, a lot

of organizations that were compromised because of this attack.

And it happens all the time. And by the way, if you mentioned, for example,

like, if you already, like, touched the point of, of open source,

now you have also Hugging Face, which is completely different area. Now it's

not only Open Source packages. It's all these different Open Source

Hugging Face models and Hugging Face datasets. And there,

all these internal artifact are completely useless because they don't even

scan these models. It's completely different technology, completely different, like,

heuristics in order to find it. And, therefore, you start to

see kind of, like, trends for for the attackers. They started to

upload a lot of backdoored and a lot of malicious models

into Hugging Face. I can tell you, like, we personally, we already,

like, detected, I think, almost, like, 100, back

or the malicious models, on Hugging Face because it's a wild

west. Right. Because how do you because these these models, first off,

they're physically large files. Right? So that there's that's a factor.

Right? I don't know how Hugging Face makes money. I'd be

curious to have someone on the show talk about that. But, you know,

they're doing the service. And, how would you even scan? I

mean, that's a good question. Right? What types of vulnerabilities have you sent have you

found so far? And how does one even scan, like, a safe

tensor or g file? Like, how do you what's what's

that look like? Right? Obviously, I'm pretty sure, you know, McAfee

antivirus doesn't have a thing for that. But, like Exactly.

But, how do you even do that? I'm just curious. Yeah. So this is, like,

exactly, like, the problem. Like, it's even, like, in in in the models, like, it's

even, like, a a more, like, the the risk there, like, is

more, like, clearer because as you know, a lot of time, like,

these models in hanging face are even, like, in pickle. And, like, pickle is, like,

by design, like, insecure, like, file. And so

binary dump, right, of, like, the memory space. Yeah. Like, in the deserialization

process, like, basically, you can, like, put, like, any kind of, like, malicious,

action that you'd like, that, like, the attacker can. So we see,

like, different attacks. Like, most of the attacks come today, like, from pickle files.

Some also, like, not even, like, in the deserialization process, but also, like, in the

model code itself. For example, like, if you ask

for a specific example, like, share something that we

detected, like, recently. We found, like, a very,

let's say, a popular, open source, LLA model that we all

know. But we know that, like, a it has a lot of, like, different

versions. And one of the version was actually a docker

that took the original model, wrapped it up with few

lines of code in the model, which what they did is that every

input to the model and every output from the model

was also sent to the attacker, which basically

just received full visibility and observability to all the

runtime application and production. So, like, all the organizations that,

like, use this model. And performance wise, the

data scientist, of course, they cannot, like, detect it because performance

wise, it worked perfectly because it took the original model. So nothing to be

suspicious about. If we want the data

scientist, every new open source model that they like, like

in Hugging Face, they'll start, like, to open, like, these files and the binaries and,

like, to start, like, to looking, like, in their own hands, they're manually

for, like, a for a for risk. First, like, of course,

like, we understand that this is not their expertise and, like, it it

like, we want to be secured, but, like, like, even, like, worse,

we just spend all their time on security. And I think that

this is, like, the worst stuff. Actually, it's not the worst. I think that, like,

the worst, and this is also, like, something that, like, I saw recently in several

organizations is just, like, to block everything. Organizations

that, like, understand, okay, Hugging Face model, it's, like, true, like, a secure, like,

in secure area. Let's block it. Let's say, like, to

all the data scientists in the organization, you're disallowed to use HAG interface model. I

think this is, like, the worst. That seems like a mistake because

because the people are gonna find a way. Well, 1, where you can't stop the

signal. Right? That was a line from, a movie.

They can't, kudos if people know who that what movie that is.

But, you know, if you block Huggy Face, people are gonna find a way

around that. They're gonna put it on a thumb drive at

home and then bring it in. So percent. This is, by the way, also, like,

what you see, like, with this kind of, like, internal Artifactory. You see that, like,

once you get to you you create for the r and d or create for

the developers or for the data scientists, you create some level of, like,

friction. They will just find a way out to, like, bypass

it and to to lower this, this friction.

Right. So so couple of questions.

One, I've seen, improper naming

Not improper naming, but but basically using, names,

like, that's looks similar to what should be. Yeah. Type will split. Type

type splitting. That's it. I've seen that, which is kind of, I guess,

kind of, you know, dollar store approach. But also,

how does how does it if you wanted to look through these model files, as

far as I know, they're just I just looked at them. I just see binary

stuff. Like, how would you look for malicious code in there? Because I think you're

right. That's not a skill set the average AI engineer or data scientist

would have. Yeah. So, basically, like, you need, like, to manually kind of, like,

parsing it because, like, you have, of course, like, the the binary file, but most

times, it's not only, like, the binary file. You label for, like, the the code

file that run, like, run the model, and you label for, like, the, in

case it's, like, pick a, like, the deserialization process, that you can, like,

parse and then, like, to see, like, the code there. But then you

need also, like, you know, like, you have, like, 2 phase. 1st, you need to

to parse it, you know, like, to see, like, the code, but then you need

also, like, to be able to read code and to understand which

one is valid and which one is malicious, which is also, like, completely, like, you

know, like, you need expertise in this area. If you see bash

commands, is it okay or not? Do you see access to the

Internet? Okay or not? Like, you you need, like, to have, like,

some, like, detectors in there that, that know how to do it, like, build

by by expert or something. So how would you even detect

that if you found it? Like, how was this found? Was this just somebody looking

in network packets? Or, like, what how was it discovered? I'm

just curious. Yeah. This specifically was, like, by our

security research team. Okay. Yeah. That's like, looks a

lot if, a lot like all the time, like, you know, all these different kind

of, like, open source and third party models in order to to help

our users to make sure that, like, everything that they use

is is valid. And again, most importantly, without slowing

them down. They can just, like, download and, like, run, like, with everything that they

that they want. And in case, we see something that is,

that is suspicious, we know how to detect it and to to help them to

to secure it. Interesting. Interesting.

Because I know a lot of people, you know, they they've been downloading

these models from Hugging Face. And just taking it on

faith, and I've heard that these things don't call

out to the Internet. Mhmm. And I fell into that. And then

I kinda had this moment of paranoia where I'm like, how do I know?

I mean, the only way I'm a I'm just a humble data scientist. Right? Like,

so the only way I would think about it would be to have a firewall

rule that would block network traffic going up for that box.

And I'm sure there's probably workarounds to that too. I mean, are these

attacks are these attacks that sophisticated yet?

Yeah. Yeah. And, like, also, like, most times you don't, like, the data

science, like, they don't want, like, to permanently, like, to close, like, the Internet, like,

the outbound because also, like, the application needs it. And also, like, the, you

know, like, the the in order, like, to download, like, the dependencies and the models

you needed. So most times, like, just, like, to block the Internet, it doesn't solve

everything. It was, like, more, like, in the past that everything was, like, network based

only. Today, when you have, like, also, like, the applicative layer here, so

it's, like, a bit more sophisticated.

But yeah. Wow. So

the safe tensor format, as I understand it, what you

know, you basically digitally sign or somebody

digitally signs the contents of it. Is that is

that a correct understanding? Yeah. So it's end up like a

like, in general, first thing, of course, that, like, a safe denture is, like,

much more secure. Okay. I already like by design, and as long

as we as the industry will go, like, more and more, like, towards

this road, because today, like, we still see, like, tons of light pickles.

But as long as we progress, like, all as an industry, we'll already,

like, be, like, in a bit better situation. It's not

perfect, of course. We still see some issues. And, of course, organizations still

need, like, to have some security measurements and processes

to make sure that, like, they're aware of what,

like, Hang in Face are using. But I think that it's already, like,

going to be a bit better. I can tell you something that, actually,

like, recently one of our one of our partners told me,

which was pretty cool, very similar to what you said that you

start, like, to feel a lot of concerns about this area.

VP data science of a very big like,

Fortune Fortune 500, like, very big, like, corporate. And you kind

of, like, the the head of, like, the older data science, like, groups here. And

they told me, you know, Niv, I I already know

what I'm going to be fired about, like, in a in the next, like,

24 months, and it's going to be about that. I know for sure, like, we're

using, like, so much, like, Agiface models. I know for sure that I'm this is,

like, the reason that I'm going to be fired, like, one day. Because today, like,

we're using it, like, freely. We are also, like, very creative. We're not, like,

only using, like, the most popular LAMA model, but, like, we're to,

like, take advantage of this great advantage of the platform, which is, like,

the amount and the diversity of the model that you have there. But I have

no no doubt that we create so many risks that we're just,

like, not exposed yet, that I'm going to to pay with it,

like, with my head. So it it's

it's pretty cool because it's not it's not always that you see,

r and d and business owners that are so concerned

about security even before the security team arrived

to them. But they're already aware of this risk. And it's something that

we start, like, to see more and more because, you know, it's just like it's

it's too obvious. Like, the the the window is open and everybody see it.

Yeah. I I would suppose that's in in a in a very kinda strange way

that's bit progress, right, where people think about security beforehand.

Like, even if they don't know I mean, I think this this this VP,

you know, is pretty spot on. Like, what concerns me about the widespread

adoption of these models and particularly Hugging Face, so there are no knock on Hugging

Face. I think whatever you get your models Mhmm.

I mean, we just don't know. And these things are just complicated.

Right? I mean, they are by design complicated with 1,000,000,000,000 of

parameters. In some cases, I guess, 1,000,000,000,000. But also, you

know, they have this ability to even

even if everything worked out well, even even assuming everything is

fine, right, in terms of the operationalization of these things,

There's still the chance that the model itself and its

training was poisoned. So, like,

I I mean, like, there's just so many because when I my wife works in

IT security, and I was all excited. It was about a year and a half

ago. I I was talking to her about LLMs and stuff like that

and chat GPT and and and those types of things.

And I was like, oh, well, you take all this data and you train a

model and you you distill down this graph and this and this. And then she's

like, that sounds like a big attack surface to me. Yeah.

And I was like like, data poisoning in the classic one and data

poisoning can be, like, in in in 2 levels or, like like, someone

like poisoning your data or exactly what you say,

somebody just, like, this way, like,

create backdoor in, in third party models and open source

models that then, like, everybody downloads. Right.

Right. And we wouldn't know, like, what's

the I mean, the defense against that seems very

intricate. Not impossible, but very delicate and intricate.

So in in in classic application security, there is a

great practice called SBOM. SBOM is a software

billing of material. Basically, it means that, you get, like, in

specific format, kind of like visibility to all the different

software components that build your application. One of the things that

now we're also, like, part of the building is a

official framework of OWASP, the nonprofit organization

around security of AI and machine learning. And

what you have there is for the first time you have like double layer

of visibility. The first one is just like to understand

what models I'm even using in the organization. Everything, like

what models like, include in my application. It can be open

source models. It can be self developed models. Also, by the way, not only not

only LLM, of course, also like vision, NLP, like everything else.

And also third party models that are embedded as part of the application, they

are not open no. They are not open source. For example,

if software engineer add API call as part of the application

to OpenAI, in this way, they embed

LLM as part of the application. This is also like one of, like, the models

that you are using, but you you you want to know this is all my

AI and model inventory that I'm using as Spyro as part of the application.

And in addition to that, you have even the deeper context there, which

is also like what you referred to. It's not only this is

the list of the model that I'm using, but for each one, you want to

understand on what dataset it was trained, what data

maybe also like it has access to in case it's in production, let's say, with

RAG architecture. You want to understand, like, the deep context

of all these, like, models, what I'm using, but also, like, what

happens, like, in this specific, like, model. Sometimes

it's, as you said, for to to understand what data was trained on a

model before, like, I'm starting, like, to use it by 3rd party, a

lot of time is even, like, internally in the organization.

Because once we start to train a lot of models,

we want to make sure that we don't violate

any policy that we have in the organization, either it's for compliance or

security. For example, one of the things that, like, we are like, I keep, like,

hearing a lot of time from, from security and legal and privacy

teams is that, look, we instruct all the

organization not to train any sensitive

data, PII, PCI, PHI, any other sensitive

information on our models. But except instructing

it and speak about it, nobody knows if it

happens. And we don't provide also our data

teams tools that will help them to

detect it in case it, like, it happens, like like, not in purpose. For

example, I can tell you, like, one of the thing that we saw very

recently. Big organization, a huge Fintech company,

that data scientist unintentionally trained all the

transaction of the application on one of the models. Now it's

a, like, crazy big violation there of, like, compliance and

security. The data scientist did this unintentionally. They

truly, like, didn't know it. If they had something that, like, would help them, like,

the basic visibility that you mentioned before, it will truly, like, help them to

start, like, to continue, like, innovate and just, like, in case something like bad happens,

to be alerted in that. And so I see that, like, the the data training

is also, like, very, very important point also internally and not

only the external data train on the external models that we're embedding and

downloading. So you mentioned, OWASP. So just

for the benefit of folks who may not know, because most of our listeners are

either data engineers, data scientists. What is OWASP? And what is the

I think it's with the OWASP 10? Yeah. So

OWASP in general, it's a amazing organization that,

is like a nonprofit one that helps basically,

we combine a lot of people together, gather together in order to make

sure that all our industry is much more secured with a lot of

different security initiatives in a lot of different aspects, mainly of like product

security, but not only. Product security is like application

security. It's building security.

Specifically in OASP, you have several different types of

projects. So for example, one type of project is the OSP10,

top ten, that basically takes different areas

and define the top ten risks in this specific area. So it can be top

ten for API, top ten for

CICD. And now there is also like top ten for LLM.

Addition framework, like, there are a lot of like different tools. Specifically,

if someone wants to understand a bit more about like the wide

landscape and the risk around AI and machine learning,

the framework that I would like recommend on, highly recommend on, is

amazing and very comprehensive called the OWASP AI

Exchange. A group of people, again, gathered together,

that covered not only LLM, but all the basic

principles and risk in data pipelines and MLOps

and start from the building and up to the runtime and start from the

classic machine learning and up to Gen AI, very comprehensive,

very also practical, which is very important and

speaks in both language, on both languages. On one hand,

of course, security, but on the other, also like very oriented

for data and machine learning and AI practitioners.

Interesting. Interesting.

What what do you see

well, here's what I mean, I'll have a lot of questions, but one of them

is, do you think the 0 what do you think the

0 trust approach is a good starting point? I don't think

it's the answer here like it is kinda everywhere else. But do you think that,

that type of philosophy of don't trust anything?

Right? Kind of like, I mean, is that because you you mentioned this

early when I talked about network firewalls, right, where the old approach of thing

is just pull the plug or set up rules. And that used

to work, but there's plenty of other ways around it, Both I think

kind of low skill, mid skill, and certainly high skill

ways around that. What do you you mean then 0

trust is meant to address that. What are your thoughts on like I

mean, is that the pro is that the mindset that either

security folks in this space would have to take on? Like, it's more

if they well, they probably already have. Right? Yeah. I think you're,

like, I think you're actually, like, the the you you you perfectly

defined it because I believe that 0 Trust is exactly like you say, it's kind

of like a, like, kind of like a mindset. It's not like a very

accurate, like, technical approach, but it's kind of

like more like a a philosophy with some level of implementation.

I believe that, like, the right mindset and, like, the right framework to look

on a on a security for AI and, like, all the building

and also, like, the runtime is basically to take all the

different principles that we are all already aware

of. Like we are all, like I'm saying, like the security industry,

we are all already aware of on classic software development,

building and runtime, and to implement it on the

data and AI lifecycle. For example, if we mentioned, like,

code scanning, so code scanning the notebooks, we mentioned open source,

so checking all the all the Ag interface models. But it's not only that.

For example, one of the things that, like, we see, a lot of attacks that

we, like, we had recently in the security area are around the

CICD. A few years ago, there was a big attack called

SolarWinds, that basically, yeah, so you know it

perfectly, just for the audience that, like, are not familiar with the specific details

in, like, very high level attacker that exploited and

misconfigurations in CICD tools. And this is

basically how they succeeded, like, to start, like, this whole huge attack and

breach. Now one of the things that, like, it taught us all as an industry

is that until now we were focusing on, like, securing only

our code. Now we understand that the code is not enough. We need to make

sure that the building tools are also well configured. So

we start, like, to see a lot of, like, tools that help us to make

sure that we don't have misconfigurations in the CICD and the SCMs and all

these different kind of tools. But when we are going to our domain, when

we go to the data and AI teams, as we know, we just use different

stack. We use all these data pipelines and model

registries and MLOps tools and platforms like Databricks and Domino

and Snowflake and stuff like that. The configuration, as we know, is

not like neverwhere. Most time, it's even wider. This is why it's

not managed by DevOps. It's managed by us, by the data teams. It's managed by

MLOps teams, by data infra, by data platform. And we're doing a

lot like, a great job in order to optimize all the configuration for the

product. We're not security experts. We don't want to be

security experts and, like, start, like, to spend a lot of time in that. But

nobody else just like to very easily find all these different kind of misconfigurations.

And this is also a threat and, like, attack vector that we

started, like, to see a lot in the field today. I can tell you that,

like, we see tons of attacks around

different misconfigurations in tools like Airflows and Databricks

and stuff like that. And I think this is also like a very, very important,

like, mindset, like, to be in. And in addition to that, of course, we have

all the all the runtime and all the adversarial attacks there.

There are specifically, if I mentioned in the

OSPI exchange, so OSPI exchange covers everything.

The OSPI 10LLM specifically is more

covering this LLM, like,

specific risk. And then you have, like, all the adversarial attacks, like prompt injection

and model jailbreak and model dn out of service, model dn out of wallet,

etcetera. So basically, the mindset should

be we already know security very well. We already have, like, these

principles. Until now, we just haven't

implemented them on the data and AI teams,

tools, and technology. And this is exactly what we start, like, to

what we, like, need, like, to start to do. And this is what we see

also that, like, you know, like, now we have no reason. Like, we all see,

like, these different kind of attacks. So we start to see that all the organizations

were, like, starting to to already, like, walk the walk.

Wow. Yeah. I I often wonder too, like, what you

mentioned the pipelines being a vulnerability or an

attack surface. Right? Like, or a potential vulnerability.

I often wonder now, like, when, you know, we're looking at agentic

AI, right, where these things aren't just LLMs,

right, producing text or going through these materials.

We're giving them, you know, abilities,

right, to influence pipelines, right, to to or to

whatever. Right? Like, that just seems to me like a giant

security risk. I mean, telling someone you know, there's there's multiple ways to

break an LOM. Right? Like, obviously, there's the the the $1 Chevy

Tahoe. Right? Where the guy did that. Right? Pretty low tech

approach, pretty brute force ish.

But I often wonder, like, well, what

what sorts of things are agentic systems gonna open up?

Like, what does that look like? I think that this is exactly like where we

we will start, like, to see, like, the very big LLM,

breaches, that we'll have. I believe that, by the

way, my belief is that the the how does the

attack start will still be, like, in a lot of cases,

very similar to what we see today. But the impact of the

attack will be much, much, much, much, much higher because now like the

model cannot only like, promise you a

$1 a car, but you can throw, like, I already like

send the order, can send the car to you, can like book your

hotel, can do like everything there, can share with you, like, the data

of maybe, like, other customers in the application because it is,

like, a RAG architecture, and it is also, like, different, like, tools

that provide him the ability to maybe even, like, write different codes

to the application. And then it might also like start like different types

of remote code execution. As long as we are going to

provide to these NLMs more privilege, more access,

more tools, more abilities, the impact of the risk

that they will be able, like, to cause will be much higher. I still

believe again that that pack vectors are going to start from more or less,

like, the same areas, like prompt injection and model jailbreak,

but they they eventually, like, the outcome of these attacks will be much

higher. I could see that. Because we're giving them

actuators, so to speak. Right? Like we're not we're we're

giving them agency. Right? Like where they could actually do real damage as

opposed to because one thing in saying you're gonna give somebody

a $1 Chevy Tahoe. It's quite another to actually place the order,

sign off on the invoice, and then ship it. Right? Yep. And what

if you'll do, like I don't know. Like, you'll you'll start, like, to see it

also, like, in banks and in investments. They will start, like, to transfer

your money. They will start, like, to invest, like, to buy stock. They will like,

the the the the amount of, like, potential impact here is, like, a

crazy high. I believe, by the way, that eventually, this is going to be one

of the things that, like, we'll see also, like, slow down the adoption, not

less than the than the technology or, like, finding, like, the

right use case. Yeah. No. I could see

that. I I just think that we're just setting, as an industry.

We're setting ourselves up for a huge exploit that we

haven't figured out is already there yet.

And so so what what

can AI engineers, data scientists,

data engineers do today to make things

better? I know we can't fix it because we don't know what's we really don't

know what's broken. I think that's one of the frustrating and kind of fun things

about security work is, like, it's not that there's no vulnerabilities.

You haven't discovered any vulnerabilities yet. Right? There are no unknown there are

always un there are always unknown unknowns.

But if you have an unknown unknown or a known thing,

you can you can say that you pretty much figured that out. But there's this

whole aspect, which I don't think data scientists

fully appreciate. I think they can understand the concept of the unknown

unknowns. But in terms of the consequences of it, I don't

think I think it's gonna take 1 major solar wind style

issue or CrowdStrike style issue to make people conscious

of of that. But how do

we how do we prepare ourselves? Right? You can't

stop the hurricane, but you can board up your windows. Right? Like, you

know, how do you Yeah. I and I totally

agree that, like, what's going through, like, to to shake every everybody

will be, like, the the first SolarWinds or, like, the 4 log 4

j attack that we see, like, in these areas. I think that,

like, I think that you broke it very well

and that we need to relate to both categories.

1st is, like, the known,

which already, like, exist. Like, we know that, like, you know, like,

we see that as scientists. Like, we are not a scientist.

And we see that one of the the things that, like, we see

in in in our code in compared to software developers

is that we don't give a

tip on, like, everything, around security.

Like, you'll see, like, tons of exposed secrets in plain

text. You'll see tons of, like, test and, like, the sensitive data

just like playing. And, like, it's state, like, exposed, like, in the notebooks.

You'll see that we download, like, any dependencies without, like, like,

even, like, think about it. Even so that, like, yeah, it looks like maybe, like,

a bit suspicious and stuff like that. So it's it's far

from from the basic. Let's make sure that, like, what we know that is not

best practice, just, like, start, like, to implement it. And

then regarding the unknown unknown, so, of

course, like, you don't know how to handle it. I think that, like, as you

as you said, you can start to prepare yourself. How do how do you

prepare yourself in security? It's basically to be very

organized and to to make sure that you have, like, the right visibility and

governance. As long as you have, for example, like, you know how to build,

like, your your AI or the machine learning bomb. You know all the

different, like, models that are built or embedded as part of the application,

and you have, like, the right lineage, which one

was trained on which dataset, etcetera.

Once, for example, that now let's say we'll continue with the

examples of of Hugging Face. Like, a new Hugging Face

model is is is now, like, published as a like, someone,

like, found that it's, like, malicious. You because you prepared

yourself and you have, like, the right visibility, you are able to go

and very easily search exactly, like, if you use it and

where you use it in all your organization. And this is also

because you prepare yourself. This is exactly what happened, like, in Log 4

j. In Log 4 j, it was like a dependency that

found as a critical vulnerable. And a lot of

organization, what they spent, like, most of the time is to try to understand

where they even use this Log4j. And they seem that, like, if you prepare

yourself, you are like, if you are organizing everything, you'll already

be very, very, like, ready for the for the

attack of, like, the unknown unknown. And, of course, everything

in addition to to, you know, like, learning and, like, educating

yourself. If you start, like, to understand, you'll go

to, I don't know, Databricks, for example. A lot of people use Databricks. You'll

go and, like, start, like, to see what are, like, the best practices of how

to, like, configure your Databricks environments and what are, like, the best practices

there. It's something that you can, like, find very easily, like, in the Internet. You

don't need, like, to to do it, like, from scratch.

But I'll say that, like, you you know, like, it's still, like, when we are

aware of that, it's not still, like, the the top of our mind as the

data practitioner to start looking, like, in our free time for this

kind of concept. Right. I mean, that's a good point.

Right? The fundamentals are still fundamental. Right?

You know, making sure, you know, you track what

your dependencies are. Right? So that way, if there's a breach in a hugging face

model, like you said, you'll know right away whether or not it

impacts you or not. Also too, I think you're

right. This isn't top of mind for AI practitioners. Right?

Even when I code, like, an app, my met

my thought process are very different than when I'm in a notebook.

Mhmm. It's just different wiring.

Yep. And by the way, it's kind of like, it's kind of

like a paradox because most times on the notebooks,

we are connected to much more sensitive information than on our

ID. Right. No. Exactly. So

it's kind of it's like the worst, one of the worst case

scenarios. Right? And and you're right. Like, people wanna work with real

data, and they they just assume that if they're on a system that's

secured and internal, they

they, they don't have to worry about such things,

which I think you're right. Like, with these systems that have access to

sensitive data, these pipelines, I mean, it's one of those

things where we need to start thinking about this. And what would you do

you think that there's a, like, a career path for, like, an AI security engineer?

Right? So it's not just a security engineer, like, in a traditional

sense. Right? But also a someone who specializes

in AI related issues. You think that's a growth industries? I

have, like, no doubt that we are going to like to see more. Like, we

already see these kind of practitioners in the field. I have no

doubt that it's going, to be more and more frequent. And in

addition to that, I believe that, like, even in the future, it's it's going to

be even, like, several different, like, roles. For example, one of the

things that, like, a lot of people that we work also, like, very closely with

are AI red teaming. Right. It's not even,

like, just like a AI security engineer, like, general one. Specifically around,

like, credit teaming because all these kinds of adversarial

attacks on models are very different, requires

different techniques, different tactics. And the red teamers are the

ones that, like, to, like, learning all these different

types of adversarial attacks and how to, like, check your model,

in your organization. And by the way, specifically in this

area, I do feel that it's kind of, like, top priority and

like top of mind also for the data science

team. Like you do see that on LLMs,

once they are deployed into production, the data

scientists, they are kind of like understand that there are a lot of risk there

and they are starting, like, to take also, like, responsibility even completely, like, regard

regardless of the security team to make sure that, like, we we

we reduce some of the risk there. Now the risk is not only

security. The first thing is security, like, to try and, like, make sure

that you are secured from all these different adversarial attacks or that you know how

to detect sensitive data leakage, for example, as part of the response and stuff

like that. In addition to that, it's also a lot of time

like safety risks. You want to make sure that once you deploy LLM into

production, your model doesn't give any financial advice to your

customers, doesn't give any health advice in case it's not your business.

So you then have, like, these kinds of responsibility, or example, like in the

Chevy example that you gave, that you just, like, you don't just, release

free cars or flights or books or a tail off, like, anything

like that. So I think that because the

the the the amount of potential risks are

so high on the run time. In this area, I

believe that, like, the data scientists already understood that this is, like,

under their responsibility. They see it also as part of,

like, being a professional data scientist. If I

deploy this model, it has, like, a lot of, like, accuracy, but,

like, it creates all these different kinds of risk.

I would define myself as not a super professional data

scientist, unlike on the supply chain, unlike in the

notebooks that if I code a code that is not secure, I wouldn't say that,

like, it's not professional. I would say that, like, it's okay. You're just, like, focusing

on the business. So I do believe that we start, like, to seeing this shift

also, like, in the mindset of the data scientist because of the risk of

the Gen AI, but now it's also, like, like, a move

to to all the the development and the building practices

that we have. Yeah. And I think data

scientists are acutely aware that LLMs

are just taking they mean, we talk we we call it hallucinating when

they get things wrong. But realistically, they're

always hallucinating to a very real degree. Right? It's just they

happen to be correct. And what these things are doing

under the hood is they are looking for patterns of words.

Sometimes those patterns of words are wrong, obviously wrong.

And sometimes they may give out sensitive information

inadvertently. So I can talk at least at least there's some common sense out there

when they when they do realize these things are higher risk than I think

we've been led to believe. Yeah. Actually, I love this this finish. They are,

like, hallucinating, like, all this time. Sometimes they really find it

as wrong. Like, they do the same thing as always. Right.

Right. The they don't know they're hallucinating because they're just operating normally.

And so when they go in a different direction and I've noticed

that, you know, kinda like a little bit of, you you know, off by a

little bit, and then then then it generates an off by a little bit, off

by a little bit. I ran an experiment with a hallucination, and

I read it through I ran it through a bunch of models and each one

of them didn't do any fact checking, which I mean, realistically,

I wouldn't expect that. Right? In the future, I think that'll be kind of table

stakes. But, you know, it would just go through. So

I took a hallucination, fed it through notebook l m, which then

create even more hallucinations. Right? So it took this little

genesis of something that was wrong and then made it even crazier wrong,

which I think is an interesting kinda statement and and and

also is a risk. Right? Like hallucination on top, compounding

other hallucinations. And I don't think we've really seen that yet because we've

only really seen for the most part, I've only seen one kind

of model in production. But if you have these models that will kinda work together

as agents or, you know, whether they're agents

that do things or agents that it's different LLM discrete LLMs that talk

to one another. They can get things wrong and make things worse. I mean, I

haven't I think it's too soon to tell either way, honestly. Yeah. But, like, the

like, theoretically, like, it makes a lot of sense. I think in general, like, we

don't see, like, a lot like, we hear a lot about Gen AI.

I think that, like, the level of adoption and the amount

of business use cases that, like, businesses

found are not that high yet. I think that, like, the

most of the usage today is done by, like, consumers, like,

like, directly, like, from, from the foundation model providers, like OpenAI and stuff

like that for day to day, like, jobs, like, you know,

like, reviewing mails and stuff like that.

The the big businesses are still trying to find these

different, like, use cases. I do believe that the that the

agents are going, like, to open a lot of different use cases

around it. Right. Right. I could I could see that. And

I think I think it's just too soon to make a statement

either way. But I think grounding yourself in the fundamentals

is probably always a good idea. Mhmm.

And probably a good a good

approach. So so tell me about NOMA. What is is it NOMA? I

I don't wanna make sure I pronounce it. NOMA. Okay. NOMA. Security. What does

NOMA do? Is it security firms that focus on this space? You

mentioned red teaming. Is that is that a sir service you offer?

Yeah. So NOMA basically is an like, our name is Nomo

Security. The domain is Nomo dot security. So it's Oh, okay. Sorry about

that. No. No. We're good. So, so, yeah,

what we do is, like, secure the entire data in the AI life cycle.

Basically means that we truly, like, cover it end to end. Like, we enable, like,

the data teams and the machine learning and the AI teams, to continue and

innovate while we are securing them without

slowing down. And this is like the the like, we are built from, like,

data practitioner, like, the company. So this is, like, our main focus,

meaning that we start, like, from the building phase. So if we

said, like, notebooks and hugging face models and all these different stuff and the

misconfigurations are on all the different stack and all the envelopes

tools and AI platforms and data pipelines and stuff like that. So we are

connected seamlessly on the background, and,

basically assist the the data teams to to work securely,

without changing changing anything in the workflows.

And then also, like, we provide, as you said, the red teaming.

Before you're deploying the model into production, you want to

understand what is the level of, of

robustness and security that the like, that your model has. And

what we do is we had, like, a big research team that,

like, builds, simulated, thousands of different

attacks. And then we dynamically start to run all these attacks against

your models, showing you exactly, like, what kind of, like, tactics

and techniques your model is vulnerable to, and exactly

also how to mitigate and improve it to be more

robust. And then the 3rd part is also the runtime.

We are mapping, we're scanning all the prompts and all the

responses in real time, making sure that you don't

have any risk on both sides. The security, we are detecting all these

different kind of, like, a host and a little, like, adversarial tax prompt

injection, model jailbreak, etcetera. We check also the responses for

sensitive data leakage and stuff like that. But in addition, also the

safety. We see a lot of organizations that the data scientists, as we

said, they understand the risk of deploying

models into production. And this is why not even, like, the security, but more like

the the Chevy example and, like, the the health advice and stuff like that.

So they built for their own, model

guardrails in order to make sure that they are, like, controlling what

are, like, the topics that the model is be able like, is allowed or

disallowed to communicate about. And what we do is basically to save

them also like this time. We also provide them, like, all this

runtime protection already, like, as a service. You can define exactly what kind

of, like, detectors and in native language, what kind of, like, policies you want

to make sure that are enforced. And then we also, like, protect it in the

run time. So, basically, we just, like, cover you, like, end to end, start from

the building and up to the run time. It starts from the classic data engineering

pipelines and machine learning and up to gen AI. Interesting. Interesting.

It sounds like something I think is totally, I think, a

needed needed service and and skill

set. Because you're right. Like, I mean, there's just so many risks

here, and the hype around Gen

AI is so over the top.

It is gonna be revolutionary, but

maybe not in the way you think. Right? And I always call back to the

early days of the dotcom. Right? Where it was pets.com. There was,

you know, this.com, that, you know, like all these crazy things. But the

real quote unquote winner of, you know,

.com was some guy in Seattle selling books.

Mhmm. Right? No one no one I mean, selling books. Like, really?

Like, not, you know, and it's

it's interesting to see how I think I

think that the the obvious use case for chat for for

LLMs thus far has been chatbots. Right? Customer

service type things. I think that's really only the

the the the the the surface of it. I think for me, what

I've seen is most impactful is the ability for natural language

understanding and their ability to understand what's happening in a in

a block of text. And I think

that that has enormous potential. I

agree. A lot of risks too. Right? Because what if, you know, what if

I I mean, to your point. Right? You wanna make sure these things stay on

topic. Right? Like, I don't if I'm talking to a

financial services chatbot and I say, hey, I have

my my leg kinda hurts. Right?

It's, you know, the risk of moving into health care, like, it's just kind

of, I don't how mature are those guardrails? Because I've

not really seen a good implementation of

it yet. Yeah. So, you know, like, I

don't want to to give ourself, like, a compliment, but,

we Oh, you guys are pretty good at it? Yeah. Like, we're pretty good. Like,

we were, like, you know, like, with fortune 5 100, with fortune 1 100.

Not in vain. But, yeah, I believe that in general, specifically, like, when we speak

more, like, on the guardrail side, I see that the most important thing is

to make sure that it's, it's building the

right architecture to be very flexible and easily

configure for the organization because eventually, like, you know, like, each

organization is completely different needs, completely different

context to the calls, like, in their customers, internally to their employees.

So everything should should be, like, very easily configured, but very flexible.

Interesting. Interesting. I wanna I I could talk for

another hour or 2 with you because this is this is a fascinating space.

Where can folks find out more about Noma and you? I you think it's Noma

dot security? Yeah. Noma dot security. Can't believe that's now

a top load pain, but,

and, any any, NOMA dot

security, you're on LinkedIn, and, anything

else you you'd like the folks to find out more?

No. I had, like, a great time speaking with you, Frank. Great.

Likewise. And for the listeners out there, if you're a little bit

scared and a little bit paranoid about generative AI and LLMs,

then I think we had a good conversation. Because I think we need a little

bit of that fear in the back of our heads to guide us and

maybe think about security issues. A

little bit of thought ahead of time will probably save you a lot of problems

later. And want to lose some. That's

that's all I got, and we'll let the nice British AI,

Bailey finish the show. Well, that wraps up another

eye opening episode of data driven. A big thank you to Niamh

Braun for sharing his expertise on the critical intersection of AI,

security, and innovation. If today's conversation didn't make

you double check your data pipelines or rethink your Hugging Face

downloads, well, you're braver than I am. As always,

I'm Bailey, your semi sentient MC, reminding you that while

AI might be clever, it's never too clever for a security breach.

Until next time, stay curious, stay secure, and

stay data driven. Cheerio.