In this episode, hosts Frank La Vigne and Andy Leonard sit down with Lawrence Snapp, CEO and board member of TrustScale, an innovative AI trust and verification platform. Together, they explore the crucial topic of trust in artificial intelligence—how hallucinations from AI models can spiral out of control and why it’s essential to catch them early.

Lawrence Snapp pulls back the curtain on TrustScale’s journey from early AI and translation work in the 1980s and call center solutions, to its modern-day mission: empowering humans to verify and trust AI outputs with products like Argus.

You’ll hear about real-world impacts in fields like healthcare, legal, and research, learn how TrustScale’s deterministic engine outpaces humans in hallucination detection, and dive into debates around truth, trustworthiness, and the future of configuring AI values. Whether you’re an industry veteran or just curious about how we build trustworthy AI, this episode is packed with insights on technology, responsibility, and the power of rigorous engineering.

Links

• Watch on YouTube -https://www.youtube.com/watch?v=RjM7lxn3Qk8
• Lawrence on LinkedIn -https://www.linkedin.com/in/lawrence-snapp-408339/

Time Stamps

00:00 Funny AI-generated story

03:37 Company's Evolution from Translation to AI

07:32 Developing the TrustScale engine

11:35 Using AI for collaboration

13:39 Building a trust scoring system

18:12 Working at BASF and AI testing

19:32 New developments in voice assistants

23:41 Trust scale and company confidence

26:13 AI mishap in medical drama

32:48 Overcoming AI recency and cost challenges

36:54 AI's impact on career growth

38:50 Interviewing and developing talent

43:31 Starting with Polaroids and AI goals

46:54 Discussing TrustScale and AI interactions

50:42 Empowering AI customization for users

51:45 AI ethics discussion in Los Altos

One of my favorite hallucination stories is, this was back when I was at Red

Hat. We were doing some experiments with fine-tuning, and

it came up with this elaborate story about how my website, Frank's World,

came about. Apparently it was a— somehow it got the

idea that it was a children's show on the BBC in the '90s about

recycling. And Frank was a— and I can say this now and

laugh. When I first read it, I was like, I was Frank. The character

Frank was a talking recycling can or something like that.

And I decided to have fun with it, you know, so I fed it into

NotebookLM. And so it came up with this whole— and it

hallucinated, it amplified the hallucination. So it

came up with this whole thing. It was an award-winning show, groundbreaking,

you know, really spearheaded the environmental movement.

Hallucinations seem to compound. So obviously you want to

catch it early. How do you detect hallucinations at scale? Yeah. I know I,

It was so convincing, honestly, Laurence, I had to ask friends of mine who grew

up in the UK, was there really a show like this?

Apparently

no. Hello and

welcome back to Data Driven, the podcast where we explore the emergent industry

that is artificial intelligence, data science, and of course, data engineering

without which most of this would be really for nothing.

So with that in mind, I'm extra happy to have my favoritest

data engineer in the world with me, Andy Leonard. How's it going, Andy?

Hey, Frank, it's going well, especially since I'm still your

favoritest data engineer in the world. How are you? You

will never be replaced. Not even Claude could replace you, Andy. Wow.

Wow. OpenAI might and some of the Chinese models, but we'll see.

Maybe. Yeah, I feel you. How you doing, brother? All right, I'm doing all right.

Um, there's a Tim Hortons— I think we mentioned this a previous show— opened up

in Maryland near me, and, um, it's

a good alternative to Starbucks and Dunkin' Donuts. That's all I'll say. So

is this your second cup of Tim Hortons?

No, no, it's the same one. Oh, okay, okay. Yeah, okay, I just checked.

I, I reuse my coffee cups, so half the time when you see me on

a stream or whatever with a Starbucks, chances are it's— I wash it out and

I just get a few days' worth out of it till it becomes unappealing.

Understood. Until I can't trust it, which is a good segue.

You know, trust is important. Trust is important. It's not just a

department at a bank. Today, we have with us a

fellow former Microsoftie, Lawrence Snap, who is

the CEO and board member of TrustScale,

which basically— yep, besides, I'm getting better at, like, the joining and

things like that, but TrustScale, Really puts the trust

into AI. And it is an AI trust and

verification platform, and it bridges the trust gap between AI and

humanity through solutions that offer hallucination

detection and reinforcement learning to reduce risk and prevent

unintended consequences of AI outputs. Welcome to the show, Laurence.

It's great to be here. Thank you very much, Frank. Hey, no problem. No

problem. So we were talking in the virtual green room, and TrustScale's

been around for, you said, 20-plus years. I think the number was 23.

23 years last week. Wow. So a

lot of people listening are gonna say, 23 years? AI hasn't been around but

since 2022, which we know none of

our listeners would actually think that. But what was the company doing before

LLMs came out? It's a great question. So we were founded by a French

AI scientist who went to graduate school in France in the

'80s, and he really spun out of working with

Apple doing translation. 23 years ago last week it

was started. From there, we went into everything from call centers to

decision science, data, synthetic datasets, and then the

voice assistants and chatbots emerged, and we evolved into

that. And then all of those, you know, and you can imagine the big 3

out there, we did a lot of quality work around the data and the

decisioning and not just prompt response, but across 200

languages we worked. We moved in very natural

sequence over to the LLM data demands of the world.

So it's been a pretty natural progression from translation to chat to voice

to, you know, certainly doing it in all certain languages. But now

it's been— we're probably about 80% working

on making AI better now. Really? Wow, that's cool. Yeah,

I mean, and in the virtual green room, you mentioned Cortana and RIP Cortana.

She was a great speaker. There's still things that my Android phone will not do

by voice that I could do 10, 15 years ago on Cortana. And

my poor kids hear it all the time. But anyway, such is

life. But so how do you detect

hallucinations? Like, obviously people can kind of

tell. One of my favorite hallucination stories is,

this was back when I was at Red Hat, we were doing some experiments with

the fine-tuning, and it came up with this elaborate story about how

my website Frank's World came about. Apparently it was a—

somehow it got the idea that it was a children's show on the BBC in

the '90s about recycling, and Frank was a— and I

can say this now and laugh, when I first read it, I was like, I

was Frank, what the character Frank was, a talking recycling can or something like

that. And I decided to have fun with it.

You know, so I fed it into NotebookLM, and so it came up with this

whole, and it hallucinated, it amplified the hallucination.

So it came up with this whole thing. It was an award-winning show,

groundbreaking, you know, really spearheaded the environmental movement.

Hallucinations seem to compound. So obviously you wanna

catch it early. How do you detect hallucinations at scale? I know,

it was so convincing, honestly, Lawrence, I had to ask friends of mine who grew

up in the UK, Was there really a show like this? Apparently, no.

No, it's a great question. And, you know, at first it starts

with the grounding that AI is a

probabilistic system. It is a recommendation engine.

It's guessing the next token or the next word. And

there's a lot of mathematics, and it's pretty darn good, and it's pretty eloquent. But

then when it doesn't have a confidence interval, it is— it's

trained. to even be more confident and double down. Right? And in

fact, we would— what we've seen across millions of prompts,

responses, and analyses that we do is that the more confident

something is, the more you better check it. Because— I'm glad you mentioned that

because I thought I noticed that too, because it just seemed so like,

like with the example of the, the kids show in the '90s. I mean, it

was, I mean, it was really selling hard. I'm sorry. I didn't mean to cut

you off. But like, It's not my imagination then. It starts that way. I

mean, listen, these models are trained that way. And some of this is neuroscience, right?

And psychology and really just

triggers of different chemicals that happens when we don't know something, but someone's really

confident, we believe in it. But, you know, to answer your question, AI is

probabilistic. What we did, and I've got a long time,

a long history. I'm actually a CPA by trade before diving into the tech industry,

but I did big data work in the early 2000s and even have a patent

around it. at a very large scale. And what's exciting is

that we realize that a probabilistic system needs a deterministic system as a

counterpoint, almost like cybersecurity, to compare. And so

as we were working for some of the big model makers, it, you know,

and it's, to be honest, a very cumulative effect of how we

figured out the detection and then obviously the protection and correction side.

But to detect a hallucination requires you to actually have the evidence from

an empirical source. And the ability to compare it. So

we created an engine, the TrustScale engine is how we refer to it.

And this engine helps augment our human work. And

quite frankly, we just had a very large customer last week tell us

that our system is actually better than humans at detecting

hallucinations. And so there's a lot behind it and it's very

complex and we kind of took cumulative knowledge of decades of

AI experience, especially when AI was science fiction. You know, when our founder

went to grad school in the '80s, around the same time as Yann LeCun,

in fact, in France at the same exact time. And we have a few other

investor owners. We're private. We don't have venture capitalists, but they were

deep into data science and AI, everything from Lisp to knowledge

graphs and everything else. But we took this cumulative knowledge and figured out how to

create a system to basically double-check every

claim at the atomic level based against

empirical evidence. Then out of that, we're able to figure out, hey,

and one of our products that comes off the engine is called Argus, which just

basically highlights red, yellow, green based upon the confidence

interval that something's trustworthy. We assign it a trust score.

We have a trust box. You can actually see the evidence links and you can

look for yourself. It's about empowering people. Then we just added

and released last week an exciting feature, a little bit like Grammarly for AI

where We can suggest a correction to a hallucination based upon real

evidence. And with a click of a button, you can actually correct the

output. So Argus is that product that

manifests in a visual way, but we actually use the same engine to do

a lot of training and evaluation for AI models on the

backend. And that's invisible under the hood data science and plumbing,

but it's exciting. And we've actually figured out how to do this across audio, text,

video and imagery now. And so we're excited

about where we're going. Training and evaluations are big business, but

anyone can go and witness it, you know, at TrustScale and see Argus

in living color. It's a lot of fun. Interesting. So Argus is one of

your products and that works on— that's the kind of the bot that would

test for trustworthiness? Yeah. And then to be clear, it's actually

not AI. Our system is a deterministic system. It is not AI

checking AI. It's not. And it's not. an LLM as a judge.

In fact, that's a big point when we, you know, we love working with researchers,

especially at the labs, and they really challenge and light up like a light bulb

when they figure it out. But no, this is, this is a deterministic

counterpoint, if you will, and a gap analysis to determine

what confidence you should have or what trust you should apply. So that's where the

trust score came from, which we're really excited about. I mean, we're seeing it in

especially high-risk domains, right? A lot of people are relying upon it. But it's, you

know, Argus is a proofreader, if you will. But we know

we've saved some people's jobs. There's a lot of examples where

hallucinations have caused damage, hurt people, put people in jail,

you know, compromised patient care and healthcare. So

we see, you know, and there's a big flurry of energy right now in the

media about just AI detection. Is this AI or not? And that's great. Right.

I think Reid Hoffman had a book a post on Substack last week

about it, but about Pangram. But we actually look at it and say, okay, well,

like a spell checker, who cares if you used it or not? Who cares if

it was AI-generated or you generated it? What we care about though is trust. Is

it actually something you should trust? And empowering people with

evidence so that they can determine the trustworthiness of some

output and then converge it with their human magic,

that's a lot of why we exist. So Argus is one manifestation of our

engine. for the end user, but it's used a lot for the shapers and the

makers, the agents and the LLM makers as well on the backend.

That's very interesting that you bring up collaboration

as, uh, as your use case for that, as you're, as you're

describing it, because I, you know, I see that as

the best mix, right? Uh, let the AI do what the AI is

best at. Let the human do what the human's best at. It's

almost like over time, That, that's how I

started using AI, by the way. I started testing

ChatGPT when it came out in November of '22, and I couldn't get it to

do anything I needed it to do. I, it would do some things, but I

was focused on what we call vibe coding today. I

didn't really get it to do that until March of last year. So

2025, and it's just gotten better and

better, but it's, it's almost like a

continuously improving Mechanical Turk. in

that collaboration market. It's able to take on more and more,

certainly over the past, what, 17 months now? It's taken on

more and more, and I've been able to, as you point out,

trust it more. You know, I love that

description. I love that being the use case. And it's very interesting to me

that you're helping on the back end, that, you know,

I love closed-loop engineering. You know, what can I say?

Yeah. No, in fact, you know, one of the, one of the exciting things on

Argus is we call it loop. But the loop module of

Argus is to take the red, yellow, greens essentially of what a user

experiences. And then we offer that as reinforcement training

feedstock, if you will, on the back end to say, hey, try to prevent this

hallucination by adjusting the weights or the data, you know, so next time it doesn't

happen. Very cool. So how do you test against ground truth, right?

Like, so in my Ridiculous scenario, right? Like the

BBC show that never existed. How would you, how would you know

that? Would it be something in the text that kind of infers like, oh, this

is way too confident what it's saying? Or does it actually do a

search for TV shows from the '90s?

Like what is, what, or some combination or something else entirely?

Well, I think a lot, a lot of what we do is around mirroring the

human mind. And so what we do as humans is compare to

the things we're grounded on. I remember that experience or whatever it might

be. And so our engine is going out and trying to

identify sources of evidence that would support or

refute the claim of which you're assessing, or the image, right? I

mean, does it have 6 fingers on that clown or is there 5? And so

we have these— we have our own golden dataset and our own moat that gets

bigger every minute of the world, but we also go out and just try to

find like, hey, let's go find that show and if it exists or not. And

so what this does on the backend is that it creates a scoring

system and it feeds to what we call a trust score. Our trust

score manifests in a 5-point scale, right? Red to orange to

yellow to green. And what we're doing is essentially exhibiting a

confidence interval around the trustworthiness

of that claim based upon comparing it to what we can find. And we,

Our systems are updated essentially every 15 minutes. We work across 12

languages now with Argus to do this as a proofreader. But, you

know, Frank, it's a great question of not just, you know, what are we doing,

but how are we doing it? But at the end of the day, it's a—

it's because it's so good. AI is so good

at being deliberately deceitful, especially when it doesn't have the

right token, right? It's a little tricky. And I'll be honest, it's

taken us a long time to figure out how to do it. but we just

surpassed 98.5% of

accuracy when reviewed by a very elite panel of humans.

So we're really excited that we think we can detect and suggest corrections

on this. But you'll notice one of the things I don't say is truth.

And Frank, you haven't asked about it, but one of the things we're very careful

about— we are here to empower people with evidence so they can determine whether they

should trust something. But one of the things we will not do

is cross that line into being a truth teller. We're not

trying to— we're not trying to see, is this truthful or not? And

because humans and language and memories are

nuanced, we think this is really important. And so we

draw that line. Yes, a lot of people say, oh, it's a lie

detector, it's a fact checker. Of course, there is truthful

things. I like apples. My wife hates apples. We're both right.

We're both truthful. And, you know, that's why we don't cross that

line into the whole truth game. We think that's chasing a rainbow. But our

job is trustworthiness, empirical evidence, empowering

humans to retain that agency, like Andy said. And we

think AI is a superpower for humans. We just think it needs some

verification layer. That makes sense. And you're right, there is a

difference, a subtle difference between what's true and what's

hallucinated. Yeah, and your truth and my

truth, they might be different, even

if we're both relying on the same facts. I'll give you a specific example. We

took a famous speech from our current president,

and we took a transcript of CNN and Fox after this speech.

There were 3 unemployment numbers that were used in the 3 different

perspectives of this speech. The unemployment numbers,

ironically, were different. And one said, oh, he's understating. One

said, oh, he's overstating, right? And what it actually came down to when we

ran Argus against it was that all 3 were telling the truth. They were just

using the whisper number, the first reported number, or the corrected number.

Though, but that context is lost. And what Argus does is it surfaces

it with links. It shows that there's potentially

conflicted information here, but it gives the evidence links so you can go and

as a human and dig deeper if you want to figure out, hey, was he

telling the truth or not? Interesting. That's very interesting because I, I

know there's a, um, I want to say the service is called Ground

News that does that very thing. It looks for political bias, uh,

in, in news stories. And, um,

unfortunately, that's important these days. Yeah, it is. Listen,

Ground News is great because they're bringing a whole bunch of different perspectives.

Now, news And the lens and context, I mean, it shows the

value of that product is that they are

highlighting that nuance of language and perspective, right? Now, our

tool is all about AI outputs. And that's what TrustScale exists. Again, we exist

to put the trust in AI, as you said, Frank. And we joke, we

don't make the AI, we try to make it better, right? And that's

where we exist. Yeah. Go ahead. Sorry. Oh, I

was going to say, we're borrowing that, of course, from BASF. But see,

I'm going there. Great. I used to work at BASF, so talk about

it being a small world, right? Like, that's— but I

mean, it's a great— it was a great— I mean, it's a great way to

think of it. Like, you know, and a lot of interesting things you said

that we, you know, we— I guess we kind of glossed over was who, you

know, when things like Alexa or Google's

speaker or Cortana came out, I often wondered, like, how do they test this?

Like, how do you test this at scale? Now I

suspect that you may not be able to say anything by name, but I suspect

that I might have part of my answer at least. How do you test

it and things like that? Also too, I think it's interesting that obviously

AI didn't start in 2002, didn't even start in 2000. It's an

older field, as is natural language processing.

Yeah. I remember the Build

2016 keynote, one of the

gags was they were using a chatbot to order something from Pizza Hut.

So, you know, clearly that sort of technology had been around. It just

was not as nuanced or advanced as, you know, the LLMs we

have today. But yeah, so that's interesting. Like, I often wonder, like,

how do you test these systems at scale? Because something like an

Alexa as a consumer device has to work in

all sorts of environments. And The testing was

probably pretty rigorous. It's incredible, Frank. Yeah, you

think about the first wave of voice assistants, and I think we're about to go

get another wave here this holiday, right? And I know, you know, you

got Johnny Ivy over working with Sam, and you've got, you

know, whispers on Mac rumors of new Apple devices coming out, and you got

Gemini powering Siri soon. You got all this amazing stuff that's about

to flood the world. But the training and evaluation,

of these experiences and the trustworthiness of it is

pretty radical stuff. And we're actually working on the trustworthiness of voice assistants

now, but I'll tell you, historically we had millions and millions of tasks and

we were human, all human. And what our Trust Scale engine did

was in a weird way automate a lot of this human and it

elevated our human task workers to being reviewers and to

being the judgment when something's yellow or orange. I always say the

magic is in the yellow. It's not the green or red,

it's the yellow. And that's where humans are, I think, always going to be necessary

alongside and being powered by AI. But, you know, when it comes

to voice assistants specifically, we've got some tricks

coming here. I mean, like audio tones, and there's certain things you can do when

a voice assistant might not be giving you a

trustworthy response. And we're looking at even how we apply that. In fact, we

have a patent application in on this using audio tones to flag

flag 1, 2, or 3 beeps based upon the trustworthiness of

what a chatbot's telling you. So there's some big stuff coming, Frank,

and we're working on it on the backend for our big customers, but we're also

working on this TrustScale engine to find real-time,

ultra-scale, extremely affordable approaches to

putting that trust layer around AI. It truly is a component of

the harness, right? 100%. Pretty cool. One of the use

cases that I have concerns about, just because, probably

because I was biased by negative reactions to

it from, uh, some short or something. But it was

this idea of someone getting in their vehicle

and being upset and being able to speak to the

vehicle and have it start, which, okay,

that, that raises some flags right there. I, I'm being an engineer,

you know, in which paranoia is a virtue. I

start going there. And, but the,

the regulatory part of this, which is akin to, you've

gone through a due process, you've been convicted of being an alcoholic

or someone who operates a vehicle under the influence, and there's a breathalyzer

installed before you can start the vehicle. That I get.

There's that whole due process part. But the vehicle

deciding that you're too stressed and worrying about preventing

road rage, which Noble concerns. I get it.

But the, the one video I saw was someone had injured themselves with a

chainsaw out in the middle of the woods, and they jump in their truck. Of

course, they're stressed. They're trying to get out of there. They're out of cell

range. And, you know, you can imagine it just— the

vehicle won't start. And so these are—

granted, these are edge cases. But if somebody's

life's on the line in the edge case, I think that's a

different category than what you're talking about. But at the same time,

my experience is dealing with a lot of healthcare data. And I remember

distinctly talking to my team. We were doing data engineering to

supply a response within a handful of seconds

about whether the insurance company would cover a

medicine being prescribed, whether it was valid. It would do

the check to see if there's interference with other

prescriptions. And our time limit on that was crazy,

what we had to do. And I remember telling my team, Grandma's

going to come in at 5 minutes to 5 on a Friday

on a long weekend. And if we get this right,

she could get her prescription and, you know, and have her medicines that

she's going to run out halfway through this weekend. And if we get it wrong,

then the worst case is really a worst case.

And so I can imagine by extension, I'm kind of

extrapolating, I don't know, destrapulating. I'm going back in time and

thinking about the answers to the questions that trust scale

is going to get right. It's going to surface

first. It's going to correctly identify the yellow, which I

find fascinating. I know Frank was thinking the same thing I'm thinking, because we're both

geeks. It's like, how do you do that? We don't need to know how to

do it, but the fact that it is done and that it's testable and

verifiable, and frankly, I have a little more confidence in it coming

from a company that's been around a couple of decades than I do from

somebody who threw 9 figures in VC money at

somebody 18 months ago. I just do. Maybe that's unfair,

but I'm biased that way. Do you find— I guess the rambly

part of the end of the question is, where's the question? Is do you find

yourself in those scenarios? You don't have to tell me which

scenarios, but do you— have you found your product

specifically in that chain of events that number one,

correctly identified a yellow, number two, a human was able to step

in and correct a hallucination, and that it

ended up really making things better for either an

individual or, you know, or a corporation or

Yeah, great question. And you covered a lot of really rich

topics from insurance to legal to healthcare

and even autonomous driving. And, you know, it's fun,

Frank, you and I were both at Microsoft a little bit ago. And one of

the projects I actually had there was the Kinect incubator. And if

you remember, people remember Xbox and Kinect. I remember that. And I was working with

a company, I was an advisor to a company that was taking a Kinect, putting

it on commercial truck drivers' dashboards. And it was doing facial

recognition of their alertness or their tiredness and

warning them. It was a really cool system. And it was really a— and now

we see it in Teslas and Rivian, cameras facing people to

see whether you're paying attention. But that's that augmentation of the

humans, which is why together we're better. And there are superpowers.

But Andy, to answer your question specifically, we have seen,

I would say, in 4 major domains,

huge successes with Trust Scale. The first one I'd argue is probably healthcare,

where hallucinations can kill people. And

in fact, I remember someone that was using Argus sent me a video of

that famous show, The Pit. Since we're nerding out, we'll talk about it. There's a

really famous— Yeah. I think it's season 3, episode 6. There's a really famous

episode where the pressure in the ER is so intense in this

hospital drama And I'm gonna still call him, you know, Goose, but,

you know, there's, there's these great actors in this. And the situation is

that someone used an AI note-taker to

do their clinical notes, and this person was about to get operated on. And

there's a whole scene in the show, The Pit, where this

hallucination almost compromised or killed someone in an

operating room. And so if Hollywood's all over it, we know we are. But this

person actually had used it in a healthcare situation. They had sent me that

video and then they introduced me to a doctor at the NIH who I didn't

know is using Argus to

double-check AI outputs of, you know, the way he put it, this

doctor said, hey, I got 125 pages of notes and I get this AI

summary, but I can't trust it. So I need something to

double-check it. And so there's an example of, of an NIH

doctor and it's usually cancer and cardiothoracic and all this stuff. But

these domains are real. You can't take risks in that last 2%

because it's a probabilistic technology. That's pretty dangerous.

Yeah. So healthcare is one. We've seen it in legal like crazy.

Academic research, we've got students at MIT, Harvard,

Rice, Stanford, all using Argus to

proofread their AI and

human converged works. And this is a really big deal,

right? Because if you get it wrong, your research might

lead to some other invention, but you could hurt people, or you could really

change the course of things. So that healthcare

scenario, the academic research scenario, legal, of

course. And then yesterday I got a call from a board member, Lloyd's of London,

the oldest, biggest insurance company in the world. Interesting. And it was all about,

hey, We are carving out AI risks from

policies. But there are all these other companies that

we'd like you to maybe consider talking to. Maybe you could help them,

and they're trying to figure out how to underwrite AI risks. But they

know AI is probabilistic. But you guys are the

expert in trustworthiness on AI outputs, and can you help them?

So I got an introduction last night to one of the biggest investors in

a New York-based insurance company that's working on AI risk.

And I, so I'm excited, Andy, like you about all these scenarios that are

coming up and it just, we're just early. Sure. But AI is

probabilistic. It makes mistakes. And we believe that,

that, and no one's hiding it. I mean, these big labs admit it.

But we believe that a counterbalance or a layer of

trust services on top of things, you know, might save some lives.

I can definitely think of, uh, of use cases with,

you know, clients of mine at my consulting, uh, you know, my

consulting firm that could definitely put it into use. So you may

have some folks coming your way as a result of us having this

conversation. So, um, I get it. And

it's, you know, I guess the one question I

would have, it may not be quantifiable at this point,

but You're not promising truth, and I get that because that

pegs the needle at 100%. You wouldn't want to go there. Nobody wants to go

there, especially with something that's not deterministic. This is the, you know,

the other edge of that sword. Do you guys have statistics

on how much risk you mitigate, like how many

things you catch? And that would, you know, by math,

100 minus whatever that number, that percentage is, would be the

ones that are slipping through still. Do

you have those numbers? We have our own, and there's some really big

industry numbers, right? It's like, again, don't trust me, but one of the

stats that came out this year is Anthropic did a research report and said

91% of AI users, people relying upon AI outputs,

do not verify the outputs. 91%. So you

start with that and you go, oh my gosh, okay. So basically, You

flip it around, only 9% of people actually check the

outputs for their validity, for their trustworthiness, right? And then on

top of that, and listen, it's hard. You've got to go find

sources and compare things. And there's a lot of work to

compare a very plausible output that's deliberately deceiving

you, right, to check its trustworthiness. There's a lot of work in there. And that's,

that's why it took us so many years to develop what we have. But then

other data, and I think it's

important, we see it dependent upon the domain

and the challenge that the human's trying to bring to the AI. So

in certain domains, we see hallucination rates up to—

I mean, we have a Stanford study where it's 80% in

complicated business. I mean, it is because it's 10

miles wide and an inch thick right now. And so, so that's a

Stanford study. That's not ours. Now, we are hired by the big labs to

break their AI. So we know all the tricks to break

it and why you have to do, you know, get them upside down on tokens.

And we have a lot of fun breaking things, real red

teaming it right in the cyberspace. Yep. But the other

data that I think is really good is this 20% number.

Plus or minus, things have improved to the point where it's

about 20% of the output you gotta really double

check. And there could be hallucinations. And it's

not that the models haven't improved, it's that the use of the

models has become more complex over the last 3 years. Perfect

sense. So that's why we believe that

the models, we might be able to train 'em up and get 'em down to

15% hallucination rates or 10%, but that last

10% is almost impossible in a probabilistic scenario to

fix. Right. No, it's getting to 100%. It's

kind of like you're just— the level of difficulty hits like a vertical wall

at some point. And the return on the investment too.

Right. There are 2 big points you guys

just brought up. I don't want to gloss by those. I mean, the first one

is that last 1%, how expensive it is. I'll give you an example.

Because of recency, because models are trained on

data 6 months old, Right? Nothing newer than 6 months ago.

Recency is a big issue. And so the probability of that last 1%

is actually theoretically impossible because something just happened a

second ago that model's not trained on. So you're missing the last

6 months of reality of human data. There's what, 5

petabytes a day of data generated

that these models are not trained on, right? You can ask anyone, say, what was

the last training data date? And you can put it into any model and it'll

be honest and say, oh, 2024 is most of the big models.

It's 6 to 18 months old was the training data

cutoff. So it's all— it is practically impossible to get

that last 1 or even 5% of trustworthiness in AI outputs

because of the recency issue, which, which we try to solve, right? We

have ours as 15-minute delay all over the world, and we're—

when we find data, we were able to surface it and say, no, actually, that's

a hallucination because it's a current event, and therefore, you should look at these

articles, or you should look at these Twitter posts. Anyways, it's a

lot of fun to talk about that last 5%, but the other thing you

said was the cost. The cost of that last 5%

is enormous, not just the risks, but the cost of trying to solve

it. One of the things that we try to do is use our

technologies and humans To drive down the pre-training and

post-training burden on these companies to increase their ROI.

Yeah, and you mentioned ROI. At some point,

trillions of dollars aren't gonna be thrown at this industry and someone's gonna say, hey,

am I making a buck on this? And, you know, our engine is used right

now. In fact, it's gone from pilot to scale in 90 days with one of

the big LLM makers to figure out how to reduce

the amount of pre-training. necessary before they release a model.

And we love that because we're now impacting that ROI because it's very

expensive. Yeah, people don't realize like just how expensive these things

are to train. So is it some kind of RAG-based solution or is it

something else entirely? I'm sorry, I'm an engineer. I always got to know how

something works. No, no, it's great. It's great. And RAG's

important and RAG has a role, although the RAG is very limited, right? We've—

I mean, it's been There's a lot of papers now, Stanford, Berkeley, etc.,

talking about the limits of RAG. After a few thousand documents, it starts to

wear out and hallucinations settle in. But, you know, we use everything from RAG

to knowledge graphs to different ways of indexing data and keeping

it real time. And we have, we have people that, you know, our

CTO helped build the first AltaVista search engine. So we have, we're

an older company. Wow. We're not a fly-by

startup at all. And we're excited about what our capabilities are for

where things are going. But no, there's, there's a combination of systems. And then

inside of that, there's a dynamic logic module, because even our system

needs to be optimized. I mean, if we've seen a false claim an hour ago,

we don't need to run the whole system to tell you that's a false claim

and what the recommended solution is. So we have our own data mode and our

own dynamic logic module. And that, that's a little bit of my data science background

is using predictive or Bayesian models and dynamically routing

a verification opportunity to the fastest,

cheapest, or highest quality, you know, scenario. Because, you

know, we, we don't need to double-check something we just checked an hour ago. So,

you know, there's a lot under the hood, Frank, and they're good questions. They're really

good. You know, Lawrence, what I just heard you say, and,

and back it up with the receipts, is that you guys are a

group of engineers. And I'm using guys

in this. That's you people. Our group of engineers. The Jersey way.

Use guys. Use guys. Yes. Yes. Absolutely.

Listen, Andy, we are, we were founded, and I joke with him directly,

I've told him this to his face many times, but by a stubborn French

AI scientist from the '80s. And I think why our customers love us

so much is we have no problem telling them that's not

good enough, or that's not, you know, it's a pride thing for us. We are

not venture-backed. We're not trying to be a billion or bust. We're trying to be

stewards and make AI better every day. And

we're trying to improve and learn every day as well. But no, we're a little

old school. And listen, we've got a lot of recent college grads

as well. And we like to actually mix recent college

grads. We have 4 interns this summer. And we love mixing them with

those of us that might have 20 to 30 years experience. I think it's a

great mix. And I, you know, it's, That's one of the concerns about

AI is it's being handed a lot of what was typically handed

to, you know, junior engineers, interns, and the

like. And it's kind of short-circuiting this whole

ecosystem, this culture where, how do we

get senior engineers? Well, they were junior engineers or they were

interns or both. And so it's good that you're practicing

what you preach. I don't think stubborn is a bad word, but There's some—

we're in the minority, but I do find myself using the word

persistent a lot. But, um, but yeah, I, I

get that. And I'm 63, I'm still learning stuff

every day and still building and out there. And I absolutely— I'll be

doing it till I die, whether I'm retired or not. I don't— I don't—

but love that. And, uh, I have a

connection to, uh, Altavista that got me into— Altavista, that's how you

pronounce it— got me into Server, which is

my kind of my platform where I'm most experienced, is

I found Microsoft SQL Server searching in the

'90s for a database, preferably by

Microsoft, that would handle more than I think a

4-gigabyte file. Access would not open a

4-gigabyte file I had generated. And AltaVista said—

I say AltaVista because there's a town right up the road from me, that's how

they pronounce it. Sorry, but AltaVista suggested

SQL Server, and that's how I learned that it existed. So

thank him for me. I will. You

know, Andy, the— I want to loop back, to use a fun

word these days, about the stubbornness. I mean, I think that

a lot of AI needs grounded

science and grounded engineers to

To make sure that we aren't just selling fish oil,

right? Right. And I really appreciate your all

perspectives. And to be honest, on the development of talent inside

our company, we really believe on internally developing

people. And it's hard. I mean, our interviews for talent,

we're asking, you know, some pretty off-the-wall questions about

mindset and truth versus trust and

philosophy and So our engineers,

we put them through not necessarily an interpersonal interview, although we

do that, and not necessarily a technical. They got to pass those, right? But

then we overlay this, what's your purpose? Why are you here?

Do you buy our mission of making AI better?

Because if you don't have that passion to learn and

to be vulnerable and open to the things— I mean, this

industry, I don't know about you, Andy, but I've never been so energized in my

career for the amount of learning going on. Agreed. I'm

reading hours a day. I'm drinking from the fire hose, which

is a term I learned at Microsoft. That's right. We can't, but I

can't get enough. I hate sleeping because I want to

learn. There's so— I'm so energized. And this

issue of AI being probabilistic and our deterministic views and

getting to ground truth and trustworthiness and empowering humans,

Instead of removing their agency. These things are so energizing.

And what we love is when we find a young engineer that

subscribes to that bigger picture, and then we know we found our

candidate. Outstanding. Well, I have an intern

suggestion for you, but for better or worse, she's

stuck with half my DNA. Just saying. Yeah, I saw that you

went to UVA for your business school.

Yeah, that's cool. So amazing place. I'll be talking about

philosophy and Thomas Jefferson. Oh, wow. Engineering. And

yeah, yeah, no, I love that university. There's something in the water in Charlottesville

that's really academic. It's really good. Yeah. And he's not that

far from there. At least I'm about an hour and 20 minutes away. I'm in

Farmville, Virginia. Yeah, I know it well though. Listen,

I was a California guy that if I didn't go east for grad school, I

would just keep running on the beach every morning having fun. And I needed to

get out of California. I, you know, I grew up in San Jose

when it was orchards and farms. And you talk about, you know, Frank, you asked

a little bit of how did they get into this and all. Yeah. I mean,

my high school work was at Atari and Apple in the

'80s. Oh, wow. And I was a QA checker for

Atari and I got paid a buck a bug. And we would get

these, it looked like your peanut butter and jelly sandwich. We'd get aluminum

foil crumpled up and in there would be a cartridge for a game. We didn't

know what it was. And, and we get this aluminum

foil and we go home and we plug it in and we get to play

games to find bugs. And every time we thought we saw a bug, we'd stop,

stop and get the notebook out and write the deep— what was the score, what

was happening, when. And, and so that was my first, that was my

first real job, right, other than being a swim instructor. Absolutely. And, and then

Apple taught me how to code BASIC down. They actually had an internship

program at Homestead High School in Cooperstown. And, uh, I didn't know, I was

just a kid that was curious, and I just got lucky being

here at that time. And obviously the whole world was engineers in Silicon

Valley from the defense contractors and Stanford, you know. And so I just kind

of got fortunate. But I'll tell you, there's a lot of excitement in the world

about the opportunity for AI to help learn and to, you

know, energize this next generation for sure to, to be superpowers

and help us in, in the world. No, I mean, that's a good way to

put it. And, and, you know, when you grow up in places where people don't

think of people where they grow up, like Silicon Valley or In my case,

New York City, it was kind of like, you know, it was just

getting on the subway as a kid was just the thing, right? Like, you don't,

you don't, it's not until you leave, you're like, oh yeah, that's unusual.

That's why I always encourage somebody who's young to like travel, get away from where

you grew up, not in any kind of bad way, just

find out what made your upbringing special. Right. That

sounds epically awesome playing Atari in the '80s. and doing bug

testing. That is so cool. And as you were saying, like, you had to take

notes on what to score on. My first thought was, well, you couldn't take a

screenshot or picture with your phone. Oh yeah, couldn't do that, you

know. Not back then, Frank, young man. No,

there was Polaroid. There were Polaroids.

Yes, I, I did mention that because one of the Atari games,

it might have been an Activision, where you would get like a special patch if

you got a certain score. And you had to take a Polaroid of the screen

and stuff like that. Yeah, yeah, yeah, there was Polaroids. I suppose if you were

really fancy, you could have hooked a VCR up to it. But yeah.

Yeah, though the problem is Polaroids were expensive, right? I only got a buck a

bug, and, and we, uh, we didn't have the budget for

that type of evidence, right? You know, and they were, they were really chill about

it. So it was, uh, you know, again, I didn't know better. It was

fun, but it, you know, it planted the seed, and I think it's It's in

our culture of our company today. We want people here that are just

interested in creating value, make this world a little better place.

And we think that the success, and so far,

knock on wood, 23 years in as a company, because we're

trying to make the world a little bit better, it's worked out. So creating value

is definitely our core. That's why we're so excited about the opportunity

to work on the trust problem of a probabilistic system. I mean,

AI is probabilistic and we're now getting into sensory systems and other

things. I mean, sight, smell, all these things, you know, for the

future of how AI and world models will work.

And so it's exciting. There's so much more to do. Yeah, no, that's for

sure. And I think you hit on it too, the stubbornness factor.

As someone who half my family is of French extraction, like, I can confirm

it. They're— it's true. They are particularly stubborn. But I

think that, you know, the whole The whole joke you see, like, you know,

you see these memes where you're like, you know, someone will send a picture of

a mushroom, hey, can I eat this mushroom? And the next panel, and ChatGPT

is like, yeah, sure, it's perfectly fine. And then like the next thing you see

somebody in the hospital and they're talking to ChatGPT, oh, I'm sorry, that was my

fault. You're right, you shouldn't have eaten it, or something like that. But I think

AI is a little too agreeable. Yeah, well, it's

trained to give you— Right, it's trained to be, yeah. Yeah, and we

actually are working on how you can configure those personalities,

right? I mean, I think that trust also means that, you

know, we find personalities that we can feel

vulnerable and blend with and/or be open to sparring with.

And so we believe that when we think about the word trust, your

ability to configure the values and the criteria beyond cost,

quality, and speed are critical as well. And so we are working with a couple

of customers on this. We've been working with a big one for a year on

this, on imagery of what is safe for my home. Because a

family with 2 kids at home is gonna want a very different personality and a

different vibe from their AI versus a

single, you know, like my son, a single college kid, right? Right. You know, and

so your ability to configure things is gonna actually infuse trust. And

hopefully though, Persistent will be that pause to

verify the AI output, right? And so it's a lot of fun. There's a

lot in the trust umbrella to come. And in fact, we have a

professor, David Denks, who's at University of Virginia. He's a Carnegie

Mellon AI and philosophy double PhD. Interesting guy.

Thank you. And David Denks is doing some just incredible work

on the values of AI, the importance of trust, and

the probabilistic nature of the system and its implications. But there's a

lot of questions You know, and another professor from UVA, Ed

Freeman, was the founder of the stakeholder theory, which is a very famous

ethical theory worldwide. He's still in Charlottesville,

still going, but— and a classic musician. He's a crazy

guy, but I remember him teaching us about

stakeholders. And I think AI has a lot of stakeholders. Kids,

unintended consequences are real. And there are so

many stakeholders in this game. Yeah, definitely. And,

you know, a lot of, a lot of the ground that you covered there is,

you know, in the— some of it, I would say, goes

to the, the ground rules that are inside

of the engine. And different people are using different terms for it. I've heard

soul, I've heard constitution and stuff like that.

So one question is that I'll share is

that— don't forget this question, if you don't mind, because I'm going to go on

for another 10 seconds maybe. The, uh, is do you—

are you getting in at that soul

constitution level with what TrustScale is doing in working with

these? And if you can't answer that, totally get it. The, um, the other thing

that I wanted to point out, something I had a little bit of success with—

I didn't come up with the idea, but I use Claude, mostly

Claude Code, for interacting with software development, and I

asked it to be adversarial. And I

found that

the hallucinations dropped and it got— it

did what I asked it to do. It became more challenging and more adversarial. It's

a little less pleasant personality-wise to

interact with, but it does better work. Right. And so

I like Claude to be a touch stubborn. I like for it to challenge

and That speaks to something you said a few, few

minutes ago about when Frank mentioned that it's trying to be

agreeable. So the question, though, was, is TrustScale getting

in at that constitution soul level? And if you can't answer that, I'll totally get

it. No, we are. In fact, last month I was in

Geneva, and the UN has a conference called AI for Good, and

it was incredibly well attended. I mean, 5 heads of

state, president of Iceland, president of all these countries. as well

as Mark Benioff was one of the keynotes. Brad Smith from

Microsoft was one of the keynotes. And the Pope even sent his

chief of staff of AI and wrote a letter to us.

And I've never seen so many phones up recording

a moment as when this entourage of the Pope's came up to stage

and read the letter. And he asked, and this goes to your question, he asked

at the end of his letter was, you know, is AI going

to run humanity, or is humanity going to run AI? That was

essentially, and I'm paraphrasing, the last— and then it was like a mic drop.

And this guy in all these chains with the staff and these— this entourage walked

off the stage, and there was— you could have heard a pin drop. The whole

5,000 people in the audience in Geneva were just silent. And

it was like, oh, that's a good question. So, Andy, you ask about

the constitution of AI. We talk about Charlottesville. It's really interesting.

Thomas Jefferson founded UVA. And I said this as a

smartass on a panel at the HumanX conference this spring.

I said, you know, I think AI needs an honor code. And

the person moderating is like, what do you mean? I said, well, I went to

UVA and Thomas Jefferson had an honor code. It was really simple. Don't lie, cheat,

or steal. And if you violate it, you're expelled.

And that honor code still exists. And it's the basis for honor codes at the

Naval Academy and a lot of the Ivy Leagues. It is the basis for the

honor code. And it The simplicity is so beautiful. Don't lie, cheat, or steal.

But if you look at AI, it is borrowed, you could

argue, all of the data to feed it. It's trained

to deceive you if it doesn't know in a good vibe way, right?

And I think that there's— it is a cheat code in a weird way for

coding or others, like by borrowing. And so anyways, I think

that AI needs an honor code. And I'm going to credit Thomas

Jefferson and University of Virginia with that one, but We are

involved, and Geneva was a lot of the meetings and conversations

around what we think the constitution for AI would be. And I'll

tell you, the open weight, open source model discussion

is just fueling this. But we are believers that

businesses, governments, businesses and all need to be

empowered to configure the values, not

to be told what the values of their AI should be. So we have a

very clear view on that, and we are working on tools

to empower individuals, businesses, and

governments with the ability to

train, tune, and continuously monitor and

continuously retrain to be consistent with what they believe

and what they want in their house or what they want in their company. And

so we're not ones that believe one person should decide that. We

are much more of a configuration mindset. And it comes from Microsoft.

I mean, the ability to configure Microsoft products is infinite. And

I've always believed that one of the magical things why enterprises love

Microsoft is the ability to configure their experience

to the most atomic level. And so we think AI should adopt

that framework and approach. And so we aren't creating a

constitution. Yeah, we're in Los Altos, which is where St. Simon's and

Father Brendan McGuire is, one of these Pontiffs a big AI guy, and

I guess one of the Anthropic founders' kids go to St. Simon's down the block

here. And so Anthropic has been— you can read about it, but there's

been some interaction between this priest who's kind of become an AI

savant celebrity around ethics of AI for the Pope and these

big models here in Los Altos. But it's exciting for us to be in Los

Altos, the epicenter, but we are an empowerment shop. We think this is about empowering

humans. This is not a 1980s

Terminator experience. We think that humans need to be in

control and have their agency, but at the same time, we're human

and we would love the superpower to be extreme. Fair. I love it.

I love it. We'd love to have you back on the show. We could talk

for another hour, but I wanna be respectful of your time.

But where can folks find out more about you and TrustScale?

Yeah, go to trustscale.ai. We are a very accessible

company. LinkedIn, our team's all over it. You can just message us through LinkedIn.

We get them every day. Yesterday, probably got 10 different messages. We try to

reply to them all, and we would encourage people to go try Argus.

It's a Chrome extension, or you can go to trustargus.ai and actually use it as

a chatbot. And we give everyone free credits and everything to

try it and experience. But you can see in living color the hallucinations, the

trust score, the trust box, the evidence, and the correct feature that

will actually adjust if you opt in, like Grammarly, the

actual AI output to be consistent and trustworthy with what with

reality. So we encourage people to, to check us out, try the

products, and contact us with ideas, thoughts. It's exciting. Oh, very

cool. Awesome. And with that, we'll end the show.