In this episode, hosts Candace Gillhoolley and Frank La Vigne are joined by Neil Katz, Chief Product Officer at Valantor AI—a four-time Emmy winner whose unconventional journey spans from technology startups to award-winning journalism, and now to the forefront of enterprise AI innovation.

Neil shares his unique perspective on the evolution of AI, from the early days of digital design and machine learning, to building large-scale AI and document intelligence platforms for major organizations. The conversation explores the critical challenges of knowledge extraction, document comprehension, and securing sensitive data in today's era of sovereign AI. Together, they uncover the hidden complexities behind Retrieval-Augmented Generation (RAG), discuss the importance of hybrid search strategies, and reflect on where the field is heading as enterprise needs push the boundaries of what AI can do.

Whether you're a data professional, an AI enthusiast, or just curious about how language models are transforming how we understand information, you won’t want to miss this candid and thought-provoking deep dive into the future of AI and data integrity.

Links

• Neil on LinkedIn -https://www.linkedin.com/in/neilkatz/
• Watch on YouTube -https://www.youtube.com/watch?v=qf8qifz_RL8

Time Stamps

00:00 Early career in tech and journalism

04:01 Early consumer AI experiences

06:49 Early AI and machine learning developments

11:43 Anthropic's new findings on AI models

16:15 Data sovereignty in AI systems

19:27 Implementing open source AI models

22:49 Breaking down documents for models

25:45 Understanding the RAG system process

28:47 Challenges in AI data processing

31:22 Challenges in RAG with Insurance Data

36:35 Understanding and managing data security

39:35 Early days with OpenAI GPT

40:48 Explaining vector and similarity search

46:57 The evolution of computing models

47:29 Computing evolution to cloud and edge

Hello and welcome back to Data Driven, the podcast where we explore the emerging field

of data science, artificial intelligence, and of course, all the hype

around AI. All of it is impossible without data

engineering. However, my favoritest data engineer in the world can't make it

today, but I did bring the most curious person I know,

and that sounds really bad, quantum curious, but she's also data curious

too, Candice Cooley. How's it going, Candice? It's great. I'm very

excited about today. Our guest has a really exciting

background. Yeah, just looking at his, his

LinkedIn profile makes me ask a lot of questions. Our guest today is

Neil Katz, who is Chief Product Officer at Valantor AI

based in New York. In the virtual green room, we geeked out on some New

York stuff, but he's also a 4-time Emmy winner

and he knows that it's a non sequitur. So welcome to the show, Neil.

Thank you very much, guys. Great to be here. Much appreciated. Good to have you.

I have to ask first, the Emmys. Did you work in media? Did you work—

what, how did you get an Emmy? I, yeah, I've, I probably have— 4

times. 4 times, true. I probably have one of the, one of the stranger

backgrounds to be in leadership at an AI company these days, but I actually

started my career in technology. When I came outta college, I built one of the

first digital design companies outta New York City. This is Web

1.0, so kind of dating myself here, but After a couple years of doing

that, we sold that company and I had a soul-searching moment. I just realized I

had spent my early 20s just spending all my time in

dark rooms with computers till 4 in the morning. And I didn't

want to do that anymore, or at least not for my whole life. So I

retooled, became a journalist, and spent 20 years in

journalism, having the great fortune to be able to report in places

like Iran, Vietnam, India for quite a time,

southern Mexico, all over the United States. for places like the New York

Times and CBS News and, and then Weather Channel, where I was running the

digital news division at the Weather Channel. We actually built a digital news

operation from the ground up, which was really a cool opportunity. And

strangely enough, even though I mentioned these nice brands like the New York Times and

CBS News, the 4 Emmys actually come from our time at the Weather

Channel, where we built a documentary

unit and went all around the world, really. Wow.

Reporting on the relationship between climate change and extreme weather

and social issues in a way that you wouldn't expect.

So we reported everywhere from Iraq to

Ethiopia to Sudan to Central and South America,

and of course all across America. And then we're lucky enough to win

some Emmys for documentaries we did. The first one we did

was actually— that we won for— was about the southern border of the United States.

and how many immigrants were dying actually at that

border, actually inside Texas when they crossed over. Things had become

hotter, things had become drier, and the passage had become more dangerous.

It's not how it works today because right now everyone goes to the southern border

and basically says, hi, I'm here and I want asylum. But back then you snuck

in, and in Texas it had become very dangerous, and they would

find a couple bodies, corpses a week, out in kind of the badlands of

southeast Texas. So anyway, starting off on a weird note for an AI

podcast, I realize, but I spent a long time in the journalistic world

and happy to go into that if you want. And then something interesting happened.

IBM came along and purchased the Weather Channel in one of probably the

strangest acquisitions in the era. Yep. We

can say a lot about that if we want to, but then I got thrusted

right back into technology. I'd run from it. So I'm going to do this content

thing and then tell people's stories and got thrust right back

into tech. And IBM really I think to their credit, made us much

more of a technology company than a media company. That was actually probably pretty good

for us. And then I was running big product and AI teams

under Watson, IBM Watson Group, and met my,

my current partner, Ben Fletcher, who's our CTO. And we formed

a company called EyeLevel, which just got acquired by Valontour. We can tell that

story in a minute. Ah, very cool. But even back in,

I actually looked up the date right before this podcast, back in 2016, so about

6 years before the launch of ChatGPT. We were using

the Watson model to build some really popular AI experiences for

consumers. We built something called Watson Weather, and about 2

million people a day were chatting, if you will, a chatbot that was on

Facebook Messenger, of all things, which was the popular place to chat,

that gab back then. And 2 million people a day were chatting with

our AI bot and trying to get their weather or hurricane warnings.

And these models were not nearly as good as the GPT-class models.

But what we saw was really interesting. We were, we understood what you

wanted maybe about 35% of the time.

So pretty high failure rate. I was thinking of the early Siri experiences. You're like,

if this is artificial— At the time it was magical, right? At the time

it was magical. 'Cause I also worked lots in the, the

before thing. I think you're right. I think people are gonna remember the, at least

the early part of the 21st century as there was before

ChatGPT launched and then after ChatGPT launched. You're probably right. The before time. But, but

a lot of people don't realize there was natural

language processing. It's old. I remember being a kid on the Commodore

64 playing Zork, right? And again, that seemed

magical at the time. Really couldn't chat with it per se. Yeah. But

just the idea that you could type something in. And before that there was ELIZA,

which you've probably heard of. But, you know, but

I remember building out chatbots. I remember using chatbots on Facebook Messenger

where it would basically do a lot of linguistic processing. was

not nearly as coherent or, or as

good of responses as you get from ChatGPT. But

yeah, no, people forget. Hopefully none of our listeners, because a lot of ours

are very savvy. But a lot of people, a lot of the normies out there,

just assume there were no chatbots prior to— There was a lot of

hard work that probably won't get any glory because it

just wasn't as cool. But we saw something I think really

we saw a piece of the future when we built this system, which was that

when we got it right, when people were able to communicate effectively with

this chatbot, the engagement stick looks

basically the same as the ChatGPT engagement stick. It just was vertical. Right. So

if we got you, you were hooked. You'd come every single day, you'd talk with

us for many minutes a day trying to get your weather. If we didn't, of

course you were gone because we didn't get it right. But you could see already

what was possible if you could make this work in a more generalized way.

The stickiness of chat cannot be overstated.

You know, yeah, the way you've been talking about it, you, you know, you were

involved before ChatGPT. So my question is, how

has your definition of AI changed over the last 5

years? Oh, that's a good— that's

a really good way of putting it. Look, I think in the before times,

You would build— like, most of what was built was machine learning, if we're honest

with ourselves. Most of the things that you would practically create was more about

basically statistical algorithms to try to find efficiencies or try to find

things that were popular. So much of it was something that people

wouldn't even recognize today, actually, when they say something like AI. These were like internal

tools that you would use to improve the various performances in a company, maybe improve

your supply chain by 5%. But already you saw

the kernels, as I'm saying, of what you could possibly do. I think now obviously

we're in the era of much more generalized models, which to me feels like one

of the big breakthroughs of GPT that was very different from what was happening with

Watson. Watson in its day was pretty badass. I don't know if I can say

that word on your pod. I don't know what your PG rating is, but— We'll

figure it out. All right. You gotta bleep me, let me know. But

you remember this thing won Jeopardy, which was very hard to do a long time

ago, but things were much more specialized. You'd be building models that were

around one very specific business problem. You try to train it

on the specific business problem. And that kind of made sense,

especially from IBM's perspective, because they have business customers that all it's about— it's only

about private data. But it was very limiting. And I think that's what actually

the breakthrough with GPT was, like, wait a second, actually the right way to get

to something that's more useful even for specific tasks is to make the

models more generalized. And obviously now today, people, you

can almost ask it anything And it can do almost anything.

For me, the exciting part though is not even where we are at the moment.

Language models— this is something a lot of people in the public don't understand, your

audience will understand— language models are trained on language.

A lot of people don't understand that they're not math geniuses. We're trying to make

them better at math, but they're actually word geniuses. And

that unlocks a huge amount of human potential that computers never touched before.

Our knowledge really has been— it's been unable to

communicate with computers in that way. But even though that's been

really amazing, I think the next step, which we're not there yet, is getting the

models to do things that are not language, right? So people don't quite understand that

you can't build a rocket ship to go to Mars with a language model. We

don't— that's what the models do today. So we're still like a step away, I

think, from some very different innovations that probably don't look like the language models

today. So that was a long answer on that question, but— All

the math that these models are good at all go—

they're not good at math per se, like you said, but they are good

at math. But all the math goes into linguistics and

text, right? You can ask it— what's interesting is now if you ask it to

say how many Rs are in strawberry or what's 2 2, it'll

actually write code. It's good at writing the code that'll give you the answer,

right? But no, you're right. And it's an interesting—

it's very curious to see how

people misinterpret what AI is doing, right? People say, oh, it's

hallucinating. I was like, from a certain point of view, it's always hallucinating, right? Because

it's just guessing the next logical step. And I know there's

attention mechanisms and all that that do keep it honest. But I

was once at an event, and I hate to name-drop, but

this was Esther Dyson. Oh, yeah. Right? Yeah. Somehow I ended up in the same

room with her. I'll never figure out how, but, but she was, I

was on the same panel and she was saying like, this was 2023, so early

days of what we're, what we have today. Yeah. And she was saying, well, there's

no real difference between autocomplete on your phone or

guesses the next number and, and ChatGPT. And I was

like, far be it for me to tell her she's wrong. Sure. Certainly in a

public forum. But also too, she's right in a sense that the stealth

bomber is the same tech, technology as a paper airplane,

right? They're both doing a lot of the same things. Obviously one's way more

advanced than the other. What fascinates me— I won't explain on that one.

Yeah, I think both are radar invisible. But no, but I'm

actually— I often find myself surprised at how far we've taken

the transformer and LLM architecture. I didn't think— I thought

we would run out of steam after a couple years, right? But

things like things that have no business working, fine-tuning,

That was— for fine-tuning, that's harsh. But for things like

distillation, it shouldn't work as well as it does.

Reasoning on these things also works way better than I would have

bet on. I'm sorry, I cut you off. I'm agreeing with you. I think reasoning

and logic is something you would think maybe would be beyond guessing the next word

or the next token. But you can see

emerging properties in language models that seem to go far beyond what you think guessing

the next word would offer way

more. And I can only imagine that's probably the attention

mechanism. There's some kind of wisdom in the attention mechanism. But that—

I think it's going to sound weird. I almost think we don't

necessarily know. No, that's crazy. Yeah. There's the

new paper that came out from Anthropic, I think last week, that was talking about

that. It wasn't even totally new because this idea of a latent space inside

models has been around for a long time. This idea that there's some kind of

its own thinking space where it's doing things in its own language, in its own

quote-unquote mind, whatever that means for a model. But Anthropic reconfirmed that

and is finding that their models are now— they've been able to figure out where

inside the model this activity is happening. That's interesting because

obviously no model has been built to do that specifically. And so what is so

surprising is just by training them on the world and then

refining them, they have emergent properties that aren't specifically developed.

That's a whole new kind of thing in the history of human invention. There's no—

like when someone invented the fork, it didn't have— there was no way it would

have some emergent properties and suddenly become a spoon and do something else. So

something, something special is going on as these models get bigger,

smarter, better trained, and more refined. Oh, that's a great way to put it. And

it makes you wonder too, what— maybe language is the secret

sauce to— I don't want to say consciousness, but intelligence. Maybe language

is in itself the magic. Emerg— no,

the notion of emergence, emergent properties is not unique to AI,

right? You'll see birds will flock a certain way, right? And

all they're doing is following the bird in front of them, right? And they inadvertently

makes these patterns. And it seems

language is the gateway to some kind of emergent properties of

intelligence and reasoning. It could be. And I'm actually also interested in

languages that aren't our own. If I think about like DNA, a chemical

language. It's one of those areas where I think language models will eventually be pretty

good, as opposed to the pure physics of building a spaceship, which they

will be good at as well, but this is a different problem. But there's lots

of languages that humans are not naturally conversant in. We can't think in

DNA, but a language model perhaps can, and that can— You could find—

that's a great point, right? What sorts of things could this possibly unlock?

And mathematics is also, to your point about building a spaceship, mathematics is also a

is a language in itself. Most humans don't

think it, can't think of it natively, but there are some, right? It might be

the gateway to, to other ways of thinking about this.

Very much so. Yeah, it's— Candace, you kicked off an awesome question, by the way.

I think that you answered that question maybe 10 minutes ago and it kicked off

a great conversation. So thank you. No, it was great. I really, I'd like to

dig a little deeper into the idea of DNA as a chemical language.

So what do you mean by that? And how has thinking

about biology influenced the way you think about AI?

Boy, that's a good deep question. Look, in our practical work and the things we

do for customers every day, I wouldn't say that we think in a biological sense

or that DNA is part of that story, at least the work we deliver today.

But it excites me because when I try to think about what are the unique

things that— where are the places in the universe where language can create something new?

DNA literally is the language of life. English or

Spanish, human languages, are like the language of consciousness.

DNA is the language of life, the fundamental building blocks of creation,

at least on this planet. We don't know what it'll be like on some others.

That's powerful witchcraft. That's powerful stuff to be playing with. And so I—

and things that the human brain just isn't really built to speak that, to

think conceptually in DNA. So to me, that's going to be a very

powerful space when we can really get language models just to be native,

natively speaking the language of creation of life on Earth. That's

super interesting. It's nothing we do. I wish I could say I'm delivering that to

Air France and EDP, some of our customers today. We're not, sorry. But that's some

cool sci-fi where I think we're not that— we're not 20 years from that. Maybe

we're 10 or 5. Baby steps. Yeah. So one of the things that I

see on the Valintor website is

really piqued my interest for a number of reasons. It was, you mentioned

enterprise visual intelligence, which one, that's

very interesting. 2, for sovereign AI

environments. So tell me about what sovereign AI means

to you, because everyone has a slightly different take on it. I've written a book

on it. I have my own take on it. I'll send you a free copy

of my book and you can, you can, you can tell me if I'm full

of it or— Same, same. What? I'll sign it for you. Yeah, I'll

sign you, I'll sign you the PDF. But the, the sovereign AI

went from being this really weird niche topic to

now is the topic of the G7, right? And

obviously there's fallout for that. So what does sovereign AI mean

to you? For me, it means, I think for us as a

company, it means that we believe, look,

fundamentally, I think the world's most important

The world's most important knowledge, in a lot of sense, is not on the

public internet and never will be. So much of

the vital data in the world sits

behind firewalls, right? Sits behind corporate or government or military walls.

And if we want to liberate that with AI, with the

newest technologies, we think that's not going to come from companies or

organizations that have built 20, 30 years of digital security around the golden

egg of knowledge in their company. That's probably not going to be them opening up

the kimono and firing that off across the internet and sending it to ChatGPT

or Anthropic. There's a big— the next wave, I

think, is letting companies, governments,

countries even have sovereignty over their

entire AI stack, from the data that they've been protecting for

decades to getting the models and the

harnesses and the stacks to run right next to the data. So

you have a boundary around your world, not like a

militaristic border, but like a boundary around your space so you can

confidently and safely build AI that sits right next to the

data that you've been working on protecting for the last few decades. I know in

the political space, as you brought that in, obviously the

EU is thinking about this, the Middle Eastern countries are thinking about this. We

often get calls from, from the Middle East on this stuff. I think

they think it more from like a geopolitical perspective of Can we trust

America? Can we— increasingly, companies, countries feel like they can't.

Our relationships are fraught. So if we can't trust America, maybe

we can't trust the products coming out of America, and we need to have control

from the global perspective. We're not really thinking about it from

geopolitics, not what we do. But when we go to a company like

ADP, for example, obviously they have a tremendous amount of

enormously privileged data from their customers, the largest payroll provider in the

country, hundreds of millions of people's financial information is stored in that

company, a lot of it actually down in mainframes in the basement, old

IBM machines that we used to work for, old iron.

That's not going out to ChatGPT. They wanna build AI around that

stuff too, but they're not gonna use some public endpoint to throw that at

Fable, whoever it is. So they need solutions to bring really powerful AI

that can sit next to their best data. And that's really how we think about

it. Because as

organizations— I'm sorry, go ahead. No, go ahead, Candace. I've been hogging the mic because—

As organizations continue to adopt AI, how do they— how do you balance

innovation with the need to keep sensitive data

secure, compliant, and under your own control?

I think when people— when people say— I'm trying to guess what the word innovation

means here for you. I think what you might mean is the latest, greatest model

from the biggest companies. And, but the open source universe

of models is not far behind. It's generally, maybe it's 6 months

behind. There's a lot of debate about how robust the models are. But it's

getting shorter, right? The time span's getting shorter. And I think that's an interesting metric,

right? 'Cause it used to be, ah, they're about a year behind or, or early

in 2022, right? Ah, no one can ever touch OpenAI and where they

are. Look, the, look, the, those companies may also

make business decisions, the OpenAIs and the, the frontier

model companies, as we're calling them, they may eventually move to a model that's not

just SaaS, right? Today they've decided to be pure cloud. That doesn't

mean they have to be pure cloud in the future. But from a, like, how

do you help a company implement today standpoint, what we're really talking about

is helping them run open source models

inside their security perimeter. And our core technology, Ground X,

is an enterprise-grade RAG and document understanding system.

And the way that we've built it allows you to bring in all the documents

of your company, all the knowledge of your company. And we've done

some really special things on the first principles so that you don't even need a

frontier model to get the best performance out of the information

inside your company. We've actually built our own, trained our

own vision models to take apart your documents, break

them down into small pieces. Here's a table, here's a diagram, here's a text

block. You break things down into atoms, then you could feed those pieces,

small pieces of data one at a time into models and get the same kind

of performance you would get as if you were trying to work to a frontier

model. So we make it possible for companies to use smaller, faster, cheaper

models on-prem or in managed cloud

so they get all the benefits of the frontier on their private data.

Yeah, that's how we're approaching it today. Interesting. So do you— are you talking about

a chunking strategy? For RAG, or are you talking about

fine-tuning or all the above? It starts with a chunking

strategy. It's funny, our product really comes out of— we're talking now in a

philosophical way about AI, but our product really comes from really practical problems.

When we were at IBM, we'd see this problem. The same thing

would happen over and over again on big projects where you were

trying to get a company's information to work with a model. At that time,

Watson was the model. Now it's other models. But the problem is always the same.

What is a company's knowledge? Essentially, it's typically stored in documents,

and documents are visually complex. And fundamentally, these

documents confuse language models because they're

either really long, or they're very visually confusing,

or they're very dense in certain parts. The frontier models are getting

better at understanding these things, but when we started, it was really bad.

So our first problem was like, Company

X hands you 100,000 pages of

stuff. They say, this is what I know. This is what my department— this is

what my legal department knows. This is what my HR department knows. Okay, how do

you make that work with AI? At the time, you really

couldn't. And what we did was we

trained a vision model. We actually trained a video model because the document

models weren't good enough. The Detectron

2 is where we eventually landed. It's our most recent one we're working on from

Meta. It's a really great video model. So the purpose of that video models designed

to follow a soccer ball around the— around a video screen or

security footage and things like that, but they're really good at detecting

objects. We trained that vision model on a million pages of

corporate data, all kinds of stuff— legal, medical, financial, what have you.

And the purpose of that was because we realized the

problem of a document or a million documents was too big for

models to deal with. And this is still true because no matter what they

say the context window is, it's not really true. And so you— it

became obvious to us that to make models work with your documents and your

data, you had to break them into really small pieces. And so we did that

with a vision model that we trained to basically identify on every

single page where's the table, where's the

text block, and where's the graphic. And then we have an

agentic pipeline that essentially feeds those small

objects into a visual language model. Again, one that can be

smaller, cheaper to run because we're giving it small bits to work on,

and have it explain in great detail what those objects are

into text that's friendly to language models, or text that's

friendly to essentially search is the other aspect of this. Because when

you build these systems— I'm in the weeds now, and you tell me if you

want me to come back out to 30,000 feet, the engineering weeds a

bit— these systems are really— RAG

systems are document understanding is the first problem and

search is the second problem. Yes. The document

understanding problem, everyone thinks it's solved. It's actually not. It's still pretty hard.

And if you get it wrong,

the downstream consequences of it are extremely severe.

It's not so much that you have incorrect

information in your database. That's a problem. You ingested 100,000 documents or a

million documents. And let's imagine you're using a system that's not ours and you've got

15, 20% of that's incorrect when it gets turned into LLM-ready

data. Okay? Okay. That's a problem.

But the real problem is it's a silent failure.

So you'd never know which chunks are right and which chunks

are wrong. So now you've got this burning problem sitting in

your database, in the, in your AI representation of your data, which you've now

vectorized, you've turned it into numbers, you've done these things with it. But underlying, it's

not the same as the actual information that came out of your company. And you

don't know where it is. Kind of smolders just below the surface,

and you never know when it's going to cause trouble. I'm sorry, I cut you

off, but— No, that's right. And it— how does that represent to folks? It represents

in a funny way, because at first blush, it looks like we don't have a

problem at all. You produce an MVP, you're like, oh, it looks pretty

good. Most of these answers are right. Looks pretty good. Then the SMEs get

in there and really play, and they're like, wait a second, this is in— this

is In an intermittent way, which every engineer knows is the worst problem you can

have. You don't want an intermittent bug. In an intermittent way

that's nearly impossible to figure out why or track, this thing is wrong.

Now everyone thinks these are model hallucinations. We have

found that 90 to 95% of the errors in RAG-based systems are

not model hallucinations. They're incorrect—

it's like an information gap. It's a comprehension gap.

It's that the things that you have bad information— it's garbage in, garbage out. The

information that sits in your RAG database is not an

accurate representation of the world, of the documents you put in the first place.

Right. Now someone asks a question or an agent asks a question of your RAG

system, you've done a search. We can get into how you would do search and

why that's complicated. You basically search your RAG database for the 20 to

100 blocks of text most likely to contain the answer to this question.

And then the model synthesizes those together and produces a final answer.

This is RAG. Most of the errors are not because the

model just made some stuff up, because the whole point of RAG is that you

can only answer based on the content I've just given you. It's typically the

answer— the content I've given you is wrong, and it's hard to

figure out why or where. So I went down a rabbit hole here, but this

is the core of our technology, the problem we've been trying to solve for a

number of years, this is the problem we started solving at IBM. And

frankly, IBM wouldn't let us solve it, and so we left. From

IBM's perspective, it was a research

problem that was too middle

space. I couldn't commercialize it 6 years ago at IBM,

so IBM said, I can't make money today. And it wasn't a

$100 billion bet like quantum computing that they hope will make them into the next,

you know, bigger than SpaceX. It was this weird middle research problem.

And so we left it high level to build that. And that's the core of

the Ground X tool, that and really powerful search, which I can get into if

you want me to get into that. But that's the ultimate beginning data layer

of how you build knowledge, an AI empowered knowledge system inside a company,

whether you're using that for customer support or agent orchestration

or finding fraud in insurance claims, which a lot of our customers do.

You got to start with that, the engineering right to turn a company's knowledge

So you don't have a comprehension gap into something AI can understand.

No. And it's funny you mentioned IBM. Sorry, Candace, I cut you off.

But one of— I used to work at Red Hat up until a few days

ago. So a few days ago, literally. And so I'm quite familiar with that.

So, and then one of— I'm sorry, what? We're built on actually the first

Kubernetes, the first on-prem implementation we did was built on OpenShift. Oh, very cool. I

worked on OpenShift AIs. It was But

what's fascinating about this is that I, my last project was

a RAG, kind of like how to teach the field sales, like what RAG is,

right? Because there was this product called InstructLab. InstructLab said, hey, just

fine-tune your models. You'll be fine. Don't worry about RAG.

Didn't really work out that way. But so it was like, no, let's reintroduce

RAG into the conversation. So ultimately, kind of like the lab I

worked on, so I, I had the opportunity to, to deep dive into the space

for a couple months starting in January. Yeah. So it's a spot near and dear

to my heart because RAG RAG solutions are really going to be

the shortest path to enterprise ROI. I still

think so, obviously partially because we sell a product that does it, but

we had to— we could choose to pivot into something else, but we— and every

few months someone comes out and says RAG is dead. Why do they say that?

They say that because the models are bigger and better, but we've

always had this point of view that if you really think macro,

Okay. The world's— it would be—

it's very difficult to move all of the world's information,

which is currently stored on the cheapest medium possible, hard drives and flash

drives, move that into GPU, the world's most expensive

silicon. So unless there's a huge architecture

change on the hardware side, and I think eventually there probably will be,

but for where we are today, you're not going to move everything

from a hard drive onto a GPU. And that means by

definition, you need something in the middle that can figure out what to

feed the model when, depending on what kind of— what an agent wants

or what a human wants to do next. And it's like RAG, I think, is

that middle layer that's gonna be with us for quite, quite some time. Yeah, I

think at most it won't die. It might evolve. I know there's RaFT and then

there's something else. PEFT, I think, is the other one. No, RaFT

is the other one. RAG-assisted Retrieval augmented fine-tuning, right? But

I think the RAG pattern, whether or not it'll still be called RAG, is— you're

right. I think it's fundamental. I think it's a fundamental force of the AI

universe personally. But— Yeah, we do too. And you

mentioned fine-tuning models. That was when this whole craze started. I think it would go

into big enterprises and all of them were like,

yeah, RAG sounds interesting, but also boring to me. I want to go fine— I

want to fine-tune my own model. I want to do it because that looks like

a lot of fun, frankly. Then you find out, and it's

cool, at the other side of the engineering project inside a company, you're like, hey,

we have our own model inside the company, and you're a hero. And that's awesome.

Resume-driven development is probably what's behind it. Maybe.

But when you get in there and you're like, wait a second, how do you

actually fine-tune a model? You have to fundamentally— you need

question-answer pairs. You have to distill knowledge out of, again,

those millions of pages of documents we started with. You still have the same problem

in the beginning. Now you're going to try to distill that into

basically question-answer pairs that you can then go fine-tune a model with.

And I think everyone found out pretty quickly that is dirty, messy, difficult

business. And companies' knowledge is really

chaotic. And trying to distill out the perfect

tens of thousands of QA pairs out of that was actually pretty difficult and

time-consuming and out of date. As soon as new information happens, your

fine-tunes are out of date. So that being said, now

we're doing something that's a little bit in the middle. One of the things our

technology is really good at is extracting information from documents. And the

pattern that most of our customers did in the beginning was pure RAG. Okay.

And GroundX is an end-to-end RAG platform that

does the ingest, the processing, the parsing, the storage,

the search, the re-ranking, all of that. And you can set it up in 3

lines of code. Wow. So that's cool, you know, problem solved.

However, increasingly we're seeing is a different kind of pattern

where a lot of the information, a lot of the things people

ask are not actually good for RAG.

People would— and customers don't often understand this, right? So

people ask questions that typically merge structured

information and unstructured information and sometimes graphed

information, relationship between things. Okay, so let me give you

a— let me give you a perfect example from some of our customers in the

insurance space of a question that RAG is natively bad at. Okay,

we do a lot of work, we have a lot of customers, we have another

product called FraudX, and FraudX helps insurance carriers

find the potential evidence of fraud in their claim files

about 40 times faster than human review. Interesting. When you have

an accident, let's say an accident in a building or on a construction site, How

do you actually— what does fraud mean? Often someone's faked an accident,

or they've had a real accident and they've inflated their injuries so they can have

a much bigger settlement. Okay, the evidence for

that is sitting inside the documents. It's sitting inside 5,000 to

10,000 pages of documents. A human would take about 50

hours to read these 5,000 to 10,000 pages. Yeah, a well-trained

human will take about 50 hours to go try to find the inconsistencies in the

stories. I'm going to get to the point. A lot of that is

contained in the medical history, a medical chronology. Okay,

so this person had an accident, 2 years later they had back surgery.

Does the accident relate to the back surgery 2 years later? You could discover that

in the medical chronology of every single thing that happened to them medically over 2

years. An investigator might want to say something like,

how many times did this person go to physical therapy Before they had back

surgery. Because if you rush to back surgery,

if you haven't gotten all the physical therapy, maybe you're trying to get a settlement,

or maybe the doctors you're working with are trying to push you too fast.

Okay. RAG cannot answer that question. Why?

Because RAG can't count. So it's a judgment call, right?

I mean— Not that. Not that.

They're actually asking a very concrete question. They're asking a structured data question.

How many physical therapy appointments did this person have before they had

surgery? If you ask that question in SQL,

you could answer it instantly for a millionth of a penny.

Easiest thing in the world to do in an Excel spreadsheet. Or select

star from wherever. Yeah, easiest thing to do.

Weirdly, a RAG system cannot answer this question well.

Why? Because RAG is using unstructured data. It's

unstructured search is what RAG really is. Fundamentally what RAG is, is you've

taken all this doc, all these unstructured documents, you've stored them.

Someone asks a question, you then search that data

to try to find the 20 to 100 blocks of text most likely to answer

the question. Semantically most likely, right? That would— that's sure.

Okay. We use a mixture of techniques, but let's imagine it's just purely semantic for

the moment. What if you've had— let's

imagine you get that, you nail that search, but someone has had 200

physical therapy appointments. By definition, you're going to miss them.

The other problem is RAG doesn't actually know how many things, how many events

exist, because it doesn't— by its nature, it's searching, it's

filtering out information, searching and giving you a piece of it. So let me

get to the— sorry, let me get to the point. Okay, we're going to— Candace,

I see you're floating. I'm going to, I'm going to take you home. We'll get

it when we get there. Okay. We're gonna get right back to DNA in a

second, I promise you.

What we see now is that

customers don't understand the difference between— even you guys were a little like, wait a

second, there's a difference between a structured and unstructured question? You guys do this all

day long. They don't understand very upfront that

you might be asking a structured question— show me how many times a thing happened—

You might be asking an unstructured question like a judgment call,

Candace. Explain what happened in the ER visit.

That's a good unstructured question. You might be

asking a relationship question. How often did this doctor and lawyer work

together on a case? And so what we really need are

systems that when you ingest your documents, you can

extract the information and interpret it in multiple ways so that you could

store it in structured databases, unstructured

databases, which mostly is vector, and even graph

databases. And then when someone asks a question, they don't have to care

that whether it's structured, unstructured, or relationship-based like a graph.

So that's where we're going on the tactical level. We're seeing companies really need—

it's going beyond RAG, but it starts with that

document comprehension. If you don't get the documents right, if you can't

extract it, all is lost downstream. All right, let's talk about DNA. Let's get off

this topic, right? Let's get out of here. Let's talk about using language models to

talk to whales or something like that. Let me ask you though, so then

is knowledge extraction becoming the new competitive advantage?

We think it's 2 things, and this is where the conversations start. It's knowledge extraction.

It's, it's, can you accurately represent, we call it the comprehension gap. Can you

accurately represent the information inside this company in some way that AI can do

something with? And can you keep the information

safe, which gets into this sovereign AI? And can

you keep it inside the bound— whatever boundary a company is comfortable with? Are

they comfortable with our SOC 2 cloud? Smaller

companies are. Are they comfortable with a boundary that lives inside their

managed cloud on AWS? Some information is

okay there. Do they need the boundary to be in the basement? Do they need

to be in a data center they control machines? Do they need to be air-gapped?

The military needs things to be air-gapped. So we're built to run in all of

these environments, wherever the boundary is. You can natively install us right

next to the mainframes if you want to and still get that data comprehension

to work. So would better

documentation, better document

understanding be more valuable than simply deploying another

AI agent? The AI agents haven't—

what's an agent? An agent is just a, it's a bunch of prompts. To me,

an agent is like the future of the if-then statement. Okay. A very

sophisticated if-then statement, really. An agent is a bunch

of prompts that, you know, use a model for judgment to do a thing,

whatever that function is. But underneath— but you have to have the data underneath for

it to know what it's talking about. Ultimately, a company's knowledge is probably going to

be stored in documents and conversations, and that has to be turned into something that

agents can understand, or else

you've got a bunch of nothing at the end of it. You've got a bunch

of agents running around in circles talking to themselves.

That makes sense. That's an interesting way to put it. Think of it.

What, what do you think is next for

kind of the rag space, right? I think you and I are both in agreement

that it's here to stay. It's not going to die. There'll be many of clickbait,

ragebait articles saying that. But,

and the other thing too, actually, is you said there's multiple approaches,

right? Obviously semantic is the most obvious, but What are the other ones

that you find most effective? Yeah,

mechanically under the hood, search is a hard

problem, actually. And it's really core to making RAG work. It's obviously

fundamentally 2 things. Can I get the document— the documents right?

And when someone asks a question, can I find the right pieces inside the RAG

database? We come from a funny place on this, actually.

Because we started doing this before GPT launched. Once GPT

launched, let me get a little tinfoil hat for a second. That's

going to make it juicy for you. Okay, maybe a little tinfoil, make it juicy.

I'm not a conspiracy guy, I'm just going to give you a little bit. Okay,

juice it up for you. All right, so when—

a lot of people don't know this— OpenAI originally had a RAG

product. Okay, and

the RAG product actually was— No, that's right. I'm sorry, I cut you off.

I had a moment of realization. We were actually one of the first, first 20

testers of OpenAI. We were like one of the early guys who were given access

to the GPT models back in the day. And they had— they weren't really

commercializing anything just yet, but they had a RAG service. And

what a lot of people don't realize is it was not a vectorized service. They

were not vectorizing the data yet. They were actually— their first pass

at this, which is a bit more similar to what we do, was just storing

it as text. And then doing various kinds of text search

against the text to try to find the chunks that would eventually go

into the language model. Now, here's the tinfoil hat part. Okay,

when they launched ChatGPT publicly, they

removed that RAG service. They killed all the

web pages, so you could never even see that they had one unless you use

the Wayback Machine. We've got some emails that we're talking about it with them way

back, and then they told everybody, you know what? The better way to do

search actually is for you to vectorize all your data and do

vector search, not text search. And what do they wind up selling

the next day? An embedding service to vectorize your data.

So embedding models, that's the— and I'm not saying they're a tinfoil hat. They're an

awesome company. They have great products. We use them all the time. They have plenty

of detractors without us joining the— I'm not in the hater group at all.

I'm making it fun and funny, but We always

thought actually vectorization and vector

search, which you call— most people call similarity search,

is one of several techniques you need to blend together to

get to the answer when you do search. Why

vector search? When you say similarity, it's basically trying to

find things that are related to each other, that's more similar to each other, to

the question, right? Semantically similar to the question. Here's

the problem. When you have a lot of data,

you have too many things that are similar to each other. It's— if

you've got a jar of red marbles and you've got 10

red marbles, all different shades, and you say, find me the medium

red marble in a jar of 10 marbles, you can do it.

If you have 100 million marbles, all different shades of

red, find me medium red marble. It's really hard.

And so what happens in systems that are heavily vector-focused or

100% vector-focused, which is a lot of systems out there today,

the more information you add, the worse your RAG database performs.

We've done a lot of testing on this. Most vector systems will

start to fail or decline in performance in as few as 10,000 pages of

information. Wow. Remember, our world, we

do a lot of scale. But we have customers that have— an insurance customer

might have 20, 30 million pages of information

across their thousands of insurance claims.

So I can't go to a customer and say, after 10,000 pages, look out,

things are not gonna be so good. A lot of projects don't realize that's a

problem because a lot of things haven't scaled yet. A lot of projects are still

working on small knowledge bases. Yeah, they're all POCs. Yeah, there's

a lot of that. We're getting past that now. We're going beyond that now, but

still a lot of things are living in the small database world. When you scale,

you start to get problems. So our approach has always been really fundamentally

different and maybe a bit more similar to OpenAI's original idea.

We actually begin with the first search is a

bigram text search, taking advantage of 20 years

of good text search that's been like solid technology around a long time. We

use that to downsample to 1,000 potential retrievals out of the

system. Then We've, we have a

heavily tuned model that we designed that real-time

vectorizes those 1,000 results, then does the similarity,

and then does a re-rank at the same time. And then you get down to

the 20 to 100, let's say, much more likely

answers to your question. At the same time, we're

also doing something different with the data. When we

ingest your data we don't just represent it precisely as it

exists on the page. We're creating a lot of

metadata around every chunk so that

the chunks themselves are more differentiated than

any other system, so that when you do similarity, you wind up

with things that are more different than each other. So you don't run into that

marble problem we just discussed. So it's a more powerful way to do vector

search, more differentiated on the data. It's merged with other

techniques, and it's also merged with what we call like a

micrograph inside of it. So we're— every time we ingest documents, we're

creating keywords around all the chunks in there, and then we graph those.

That allows us to graph the chunks together so we can see relationships between things.

So basically, it's a 3— the marketing version would be like

3-way hybrid search, but now I just told you how you actually do it.

And well, yeah, that makes sense, right? Each one of these Each one of these

is gonna have their own weaknesses, right? So hopefully it's like the power system where

it's a 3-phase power, right? You know, when they all kind of, if you

do enough different approaches, they, where one is strong, one's gonna be weak

and vice versa. Yeah, I think that's part of it for sure. Yeah.

So we've always come at it from just like a first principles perspective, which is

sometimes good and sometimes bad cuz it's, you're, you're going against the grain of

maybe what everyone thinks is the right way to do it. But to your point,

what people think is the right thing— way to do something is

there's— everyone has an agenda, right? That's interesting. For sure.

Even OpenAI itself was a controversial bet. Now that the bet paid off, people don't

realize it was a bet, right? There was a lot of controversy a decade

ago of would this transformer architecture work

well enough? If I just add more data and more money and more training, does

this become something really interesting or not? Or do I need a different technique?

And OpenAI made this enormous financial bet that, nah, just keep working on it, keep

putting more cash, more training, and more data into this particular

model architecture. It will be something quite extraordinary.

So did that bet ever pay off? We don't know if it'll

pay off yet, right? They haven't— Oh, pay is the operative word. But I think

you're right. And I think, like we said earlier, like the transformer

architecture, we've gotten more mileage out of it than I originally thought.

And out of— than a lot of the original creators of it thought. I mean,

there's a reason why— I'm like, I'm not inside Google, but I assume there's a

reason why Google created— literally created it and was like, interesting,

but am I gonna throw $10 million at— $10 billion at that? Maybe not that

interesting just yet. Someone else proved it.

Sorry, Candace, I was hogging the mic because I was geeking out pretty hard.

No, I'm totally geeking out. I'm totally fascinated. And I'm wondering

if you think RAG is a long-term architectural

pattern, or do you think it'll eventually

evolve into something more sophisticated as enterprise AI

matures? In a way, my answer is—

you're gonna hate this answer— I was a journalist for 20 years, and I hate

when someone says both. I was like, pick a lane, man.

And here's why I say both. It would be silly to say that what we

have today is what we'll have tomorrow. Obviously, that's just not true. There'll be more

interesting patterns and techniques to do these things over time. And

we might be 10 years out, but the chipset will evolve. So maybe it can

take on— maybe it can hold more data inside of the

model than we can do today. But at the same time, if I think about

the history of computing, so much never changes. We're still—

take a really simple example, edge versus cloud. This

conversation has been happening since the beginning of computing, right? There was the

mainframe that was cloud, if you will. There was the

compute was in the center, and then the mini computer moved it to a

little bit of a rim around that, but it still was in the center. Then

PC revolution, massive. Everyone's forget it, compute's on the edge.

We have all this power on our desk. And then that happened

for 20 years. Then I was like, wait a second, maybe we should go back

to the cloud. And then the cloud came along, we start to put the compute

in the center. Then mobile came along, and now we've got these powerful supercomputers in

our phone. It's like, now it's at the edge. Now it's going to— same thing.

So just a lot of metaphors in computing just seem to be forever. And to

me, I think there's something in RAG that's pretty fundamental, which is

that the big fence, fancy, expensive thing in the middle

will probably never have all the information it needs to. And it's always going

to need something that has a larger database of knowledge than it

and filter it in real-time ways so that it can answer a question or perform

a function. And if you just think about what RAG is really fundamentally, it's that.

It's that the universe for information is really vast. It's out here.

It's hard for me to store all that and use it in a functional way

in real time inside whatever thing I'm building. And so you need something in the

middle that helps you search and sort and filter and deliver it in the right—

in real-time way. Sorry, that was not a straight-line

answer. That was like both sides of the mouth on that one, but I think

that's the truth. That makes a lot of sense.

We are— I would love to talk to you another hour or 2, but I

think we're at the end of time and I'm going to be respectful of your

time. So where can folks find out more about you and your

company? Yeah, so the company is Valantor

and you can get us at valantor.com.

V-A-L-A-N-T-O-R. Luckily it's spelled like it

sounds. I just spent a week and 2 weeks in Croatia and

I don't know what they're doing with the alphabet over there, but I'm trying to

make sense of it. It's an awesome country. I got nothing bad to say, but

they need some vowels. Okay. But Valontor is easy to

pronounce. So valontor.com, check us out. And for

me, look, if you want to see the documentaries, you could search— would you search

weather films or Weather Channel documentaries on YouTube and see some of the cool stuff

from my past? But check us out at valontor.com. And if

you're an enterprise that has the kind of problems we're talking about, large

amounts of data, scale, security, accuracy is what you need, say hi.

Cool. Excellent. That sounds great. That sounds great. Thank you so much for your time

today. It was fabulous. Thanks a lot, guys. I really appreciate it. And

next time we'll get back on the DNA, Candace. I'm sorry, it felt like we,

we ran outta time before we can get back there. It's okay. We'll have you,

we'll have you back and then we'll— Have you back. Yeah. That's

it. Thanks a lot. I'll let the outro music play.