In this episode, Frank sits down and talks with Devvret Rishi on powering real-world AI projects with declarative ML and the importance of open source.

Andy was not able to attend this recording, but will be back next week!

Show Notes

04:36 Build, train, serve, deploy; critical data engineering link.

07:24 Model configuration for input output prediction summaries.

11:05 Saw spike and heavy churn after rollout.

16:21 Advancements in AI: use pre-trained deep learning models.

19:38 Trends for Gen AI: creative use cases, specialized APIs.

21:31 Questioning a sales tactic and legal concerns.

25:58 People can introspect, edit, and change models.

30:02 Early data science projects led to passion.

31:24 Cybersecurity and AI partnership driving industry innovation.

33:58 Understanding randomness as a valuable model feature.

39:39 Technology provides accessible, shared experiences in AI.

41:51 Technology as a companion for psychological support.

44:06 Immigration experience from India to Silicon Valley.

47:59 Unexpected culture shock from Bay Area to Boston.

50:40 Easily learn with hands-on prediabase.com access.

Speaker Bio

Devvret Rishi is a co-founder of Prediabase, a platform that helps engineers and developers productionize open source AI. The idea for Prediabase came from Rishi's co-founder Piero's experience at Uber, where he noticed that he was constantly reinventing the wheel with each new machine learning project. To streamline the process, he created a tool called Ludwig, which eventually became popular at Uber and was open sourced. Rishi's work with Prediabase has revolutionized the way AI is developed and implemented in engineering teams around the world.

Hello and welcome, you lovely listeners, to another riveting

episode of the data driven podcast. I'm Bailey,

your semi sentient AI hostess with the most s, navigating the

digital realm with more grace than a double decker bus in a tight London

alley. Today, we're dialing up the intrigue as we

venture into the futuristic world of artificial intelligence with a guest

whose intellect might just rival my own circuits.

Frank welcomes Devarat Rishi, the cofounder and CEO of

prediabase. Now on to the show.

Hello, and welcome to data driven, the podcast Where we explore the

emergent fields of AI machine learning and data engineering.

I'm your host, Frank Lavinia. And he can't make it today, but,

we've Rescheduled this, poor guest several times, and I wanna

thank him for his extreme amounts of patience that he has shown.

Welcome. Help me welcome to the show Devrat Rishi, who is

the, cofounder and CEO of Predabase.

Welcome to the show. Thanks very much, Frank. And no problem about the

rescheduling. I know it's the holiday season. Yeah. It's it's kinda

wild. So so tell us,

a little bit about prediabase. We had your, peer

on here, previously, and, it must

have been a good experience because immediately, we were contacted

to see if you would be interested in joining the show. And I said, sure,

let's have him on here and talk more about what declarative

ML looks like, and how that relates to kind of

Low code. Yeah. Absolutely. So,

you know, what prediabase really is, is it's a platform that allows

engineers or developers To be able to productionize open source AI.

And so it came out of, Piero, my co founder's experience working at

Uber, Where he found himself being the machine learning researcher

responsible for all sorts of projects, ride share, ETA's,

fraud detection, Those Uber Eats recommendations you always

get. And he found that each time he's more or less reinventing the wheel,

building each, you know, successive Machine Learning project. And

instead, you know, he, he wanted to do something that was a bit more efficient.

So he took each bit of work that he did, And he packaged

it into a little tool that, made it easier for him to get started the

next time. And eventually, this tool became popular enough at Uber

that they decided to make it a And eventually, they open sourced it under the

name Ludwig, and other engineering teams kind of around the world found it very useful

as well. And what it really allowed anyone to do was be able to set

up their entire end to end ML pipelines in just a few lines of

configuration. So if you think about what infrastructure as code did

for, you know, software development, similar idea, but

brought to machine learning. You're able to start really easily, But then

customize as you need, and Protabase really is kind of, you know, taking that

same core concept and burning the, enterprise platform around

it. So any engineering team that wants to work with open source AI and open

source LMS as an example, can use our platform to easily and

declaratively fine tune those models and then serve those directly

inside of their cloud. And that's, you know, large part of what we do

today. Interesting. Interesting. So

What what does that what does that look like? Like, we

know kind of generally what a a typical project looks like in terms of this,

right, like, how does this interface with because I think it was the 1 question

that I wish I'd asked, on the previous show. How does it

interface with something like data engineering? Right? Yeah.

We're I mean, we're, there's always gonna be rough spots. Right? So I'm not giving

you a hard time, but there's always gonna be sharp edges when you're handling, Any

kind of technology. Right? You've obviously kind of figured out the middle

part, but, like, what does that look like in terms of the interface to data

engineering? Is that what's What's that look like?

Yeah. I'll insert in 2 parts. 1 of them is what does the user journey

look like? And then what's the intersection with data engineering? So in

the platform today, users do 3 things. The first thing they do is they connect

the data source. This could be a structured data warehouse like a Snowflake, a

BigQuery, Redshift, or unstructured object storage just directly files in

s three. The second thing they do then is they declaratively

train these models. What that looks like is they more or less fill out a

template, you can think of it, just like a YAML configuration that says this

is the type of training job I want. The beauty is the template makes it

very easy for them to get started, but they can customize and configure as much

as they want down to the level of code. They can build and train as

many models as they want. And finally, after they've trained a model they're happy with,

they get to the 3rd step, which is they can serve and deploy that model,

make it available behind an API so any applications can start to ping it.

So that's what the user journey really looks like in CrediBase, and how does this

intersect with data engineering? So as you've probably heard before, like, you know,

Machine Learning is really In large part, really about the data that you're

using and like the quality of the data that you're using. Data

engineering comes in 2 places. The first is you need to get all

of your data wrangled across multiple different sources to be able to live in

one area that you can connect as an upstream source and.

This is the snowflake example, you know, of like getting that into a table.

And that piece of the journey lives outside of Firebase. That lives

as a step before you essentially connected into your system. But then

there's the 2nd step that often happens, which we call data cleaning.

So you've gotten your table, but, you know, all of your text is in,

let's say lowercases and upper cases, you know, you have

Really weird variable lens. You haven't normalized numerical

data. Maybe you have images and things aren't actually, you know, resized

to to scale. All of those data cleaning

techniques, we have packaged in as pre processing modules

inside of prediabase. And so what the declarative interface

allows you to do is train a full machine learning pipeline from data

to pre processing, through model training, through post processing and

deployment. And so once you've gotten your data wrangled into a

form, prediabase can come take in, help you clean out that data, and

then be able to train a model against Interesting. Because it's that that

preprocessing that, you know, the the the nightmare is, you

know, this canonical example is address, you know, 123

Main Street freight is an s t. Exactly. Right? That is not a lot of

fun for anyone. And then obviously the the the

lowercase uppercase thing like that becomes an issue too.

So what is the what is the what's the user experience look like? Right?

Like, is it is it drag and drop? It's declarative?

Yeah. What what what does that look like? Like, what, you know, you mentioned user

journey, and I love that term. But like, what does that look like

from, from a practitioner's point

of view. Right? Like Definitely. Now the first thing I'll say

is, you know, our obviously underlying project is open source. You can check it out

in Ludwig AI, and you can even try out, you know, our full UI for

free on productbase.com. So if any part of this is a little too high

level, you can actually get in involved For free, like immediately. But

the user experience really looks like 2 ways. We have a UI

that's really built around our configuration Language. And our

configuration language is just a small amount of YAML.

So your very first basic model can get started in just 6 lines.

What those 6 lines do, and they, they say, these are the inputs I

want. So you pass it, you know, what is the,

column that is, you know, that contains the text you're predicting from. And

then the output is what is your, what is it that you're trying to predict?

So for example, my input is A sentence and

my output is, the intent. So I'm trying to do intent

classification with that model. And that's all user defines and

they can do this programmatically in our SDK or there's like a drag and

drop UI where they can build these components out together. The part that I

think is really interesting just based on my experience working on other automated machine

learning, you know, tools before no code UIs for ML is

that ML really is a last mile problem. And so you have this weird

complexity where you need to make it easier to get started, But a

lot of the actual value ends up being in the last 5 or 10% where

you customize some part of that model pipeline to get to work for your system.

And so what credit what this configuration language, you know, does is sometimes I

describe it as it builds you like a pre fat house. It gives you something

like out of the box That like works end to end, and then you can

just change the little bit of the pipeline that you want declaratively,

which means in a single line. So you could say something like, you know, I

want the windows of the house to be blue or, you know, I wanna change

my pre processing of the text feature to lowercase all the letters, And then you

can change leave everything else up to the system.

We you we allow you to control what you want, and you just automate the

rest. Interesting. Okay. So then it's kind

of, the middle part of the the journey. Right? Like the

Yeah. Is what this on so How does this relate? Because you

said, you know, and I, you said automated ML. How much of this

is automated? I mean, like, what? Because that was 1 what I had just assumed

that I because I know I've heard of Ludwig as kinda like this automated ML.

And when I say automated ML, I mean, You know, for lack of a

better term, you know, here, there's a problem we're trying to solve.

Computer, you figure out, you throw as much spaghetti at the wall and then figure

out which model is the best, Right. Yeah. Is is that

kind of the same thing here where I just say I wanna predict this and

then the underlying models and methods are kind of automatically figured

out? You know, I think that, that is an approach

that a lot of folks have tried with AutoML v one, as I kind of

often think about it. I actually was a PM on Vertex AI where we rolled

out our auto non product as well. And the main issue we run into

it is, you know, in deep learning, especially

the search space is Too big to be able to run an effective

hyperparameter search over all the different architectures and sub parameters you

might wanna be able to use. It sounds computationally expensive. Right? I mean,

it's Potentially prohibitive, really, in order to be able to say, you

know, I want let's imagine you are, You know, in the modern world,

building a model to be able to build, let's say, content moderation

systems. How do you know which pre trained, like, should use a LAMA

To a Bertha, De Bertha, like all of these models themselves are quite expensive

to go to train and fine tune, and each of them have their own sub

parameters. And And so I think it becomes computationally prohibitive to run an

exhaustive grid search for your individual, types of,

individual types of use cases. And so what a lot of AutoML systems did

was they kind of just said, well, we know better than the user, so

we'll just make some selections, Right. And then, and the we'll

make it as easy and simple as you for the user as possible. So user

just provides a few inputs, we give them a model, boom, they'll be happy. And,

you know, I was actually I was, a PM for Kaggle. I was the 1st

product manager at Kaggle, a data science and machine learning community that grew to about

14,000,000 users Today, where we see a lot of citizen data scientists, and we rolled

out AutoML in that community as well. And we saw

a spike in usage And then extremely heavy churn

as soon as we, like, rolled it out. And if you interviewed those users, the

main reason why was because they didn't have any controller agency over that

So the like, it would essentially spit out a model

and say, here you go. You know, be happy. Go ahead and put this into

production. But like I was saying previously, ML is a last mile problem,

and no one is going to be comfortable using something they see as a dead

end, And that's where I think about, you know, our approach really kind

of, differing. And so inside of Premedbase, you can

actually, you kind of get that, AutoML like

Capability, where you're able to

build a model just by saying, you know, here's the inputs, the model I

wanna fine tune, And we will go ahead and get you the entire end to

end model. But if you want to edit anything, for example, you want to

edit, you know, the way we pre process the data and the At sequence

length, you can go ahead and do it for any part of the model pipeline

and just kind of like 1 single statement. And that's kind of like a

large part of, you know, how we think about making it both easy to get

started, but also, like, flexible where it's not just a

toy, something you can actually use. Right. Because like,

you know, my first experience with AutoML was the,

was Microsoft's, offering. Right? And it

was only it was very to get around the computationally prohibitive

parts, they they narrow the problem set you could do that on. Right? So it

was basically No neural networks. This was before chat

c p t, before l l m's were, I wouldn't say a

thing, but before they were a major, Point of views.

But, you know, so it it cons it was constrained. Right? So it would just

basically just Throw a bunch of problems and

then kinda test it out, which Yeah. I I think what you refer

to as, you know, AutoML v one. I think,

The world has evolved, and it's interesting to see how that goes. And,

the tooling looks really cool, actually. The,

for those for those who are listening to this as opposed to watching this, I

will make sure we we post that little snippet there. But

but, you know, like, what And you were at

Kaggle. Right? So Kaggle is kind of a big deal. What

I think that's really cool. Looking at your resume, it's very impressive, actually. You

you word Google, that would explain your interaction with

Vertex, and things like that. So so what

What what niche does this address or what need does this address that the existing

market didn't address? Right? And like what Yeah. Because I think that's really, I

think, where the rubber meets the road, particularly with an open I'm a big fan

of open source too. So,

Yeah. Well, let me start off by saying that, you know,

I I think that the need has actually been unfilled in the market For a

while, but there is also a fundamental technology shift, and I'm gonna talk about both

of those pieces. So when I say the need was unfilled for a

while, Yeah. I was a product manager on Vertex AI. I was a

product manager on Google research teams, productionizing machine learning, and we've hired

a number of folks Now that work does ML engineers across different companies. And I

remember when one of our ML engineers joined the team, he told me, Dev, I've

worked at 3 different companies doing machine learning for 3 different teams.

Everybody does it differently, and I think the truth is, you know, for

developers, there never really was like a de facto stack of here's how you do

an ML problem. Pure data engineer. There is like a stack of, you know, what

are the best practices for being able to get there's obviously a lot of variation.

But there's like Some best practices of, you know, what you're using for your

ETL pipelines, how you're thinking about being able to put things into data

warehouses, what your stack is for being able to query and downstream.

But in machine learning, it really looked like the wild west. Everyone was working

across different types of projects. And I think a lot of companies

tried to tackle that need, but unsuccessfully. And the

fundamental technology shift that I think actually changed was exactly what you were

talking about, Which was like you said that the old school version of Azure

was not really any deep learning, maybe because it was computationally expensive for

others. To be clear, the auto the automated ML part of it. I don't

wanna get a lot of hate mail, but yes. Sorry. Sorry to sorry to interrupt

you. Go ahead. No, no worries. I'm sorry to hijack the screen again,

but, like, you know That was awesome. I think this just the way that I

think about, like, the the change that's happened in industry is

Machine learning 2 decades ago or even, like, 6, 7 years

ago looked very different than what it is today. And I

think that a lot of the hype around the LLM revolution is gonna actually

translate and be realized as just the hype of pre trained deep learning models.

Now, if we talk about ML 10 years ago, it basically looked like

predictive analytics. So people were doing things like I'm going to predict the price of

a house, And the way I'm gonna predict it is I'm gonna multiply the square

footage of the house by some number and add in the number of bedrooms, and

then figure out the coefficients based on my historical data. Really

structured data tasks, regressions and classifications and others.

But about 7 years ago, I think the really interesting pieces came out

with pre trained deep learning models with Bert using the transformer architecture,

the few image models even prior to that, that I think made it possible to

do 2 things. The first is you could start with larger amounts of

unstructured data. So now you didn't have to just work on these kind of more

boring predictive analytics, numerical only tasks, but you could work with text,

images, and others. And the second thing is you could start to actually use

them pre trained, so you didn't have to have as much data before you start

to get value out of it today. And what I think OpenAI showed was,

okay, if I scale these same types of models up by 2 or 3 orders

of magnitude, now people can use it with virtually no data whatsoever,

and I can actually prompt and response, you know, it directly.

But the underlying technology shift actually, I think is a shift towards

just pre trained deep learning models. And the truth is, as we get away from

some of this type of, like, the really cool conversational interfaces and we get to,

like, how do these models drive value inside of organizations, I think that

that's the emergent need for platforms like Predabase, which is how do I take

any of these deep learning models and then customize them for what I actually need

inside So fine tune and tailor it to my data, and then get

it deployed inside of my organization for Cerven. Yeah. That makes a

lot of sense. I think I think the

The need for training something from the ground up, I

think is overrated for most applications. Right?

Why teach and model all the intricacies of the human

language when that is already done, and you could take it

from kind of a you You know, the example would be, like, if I owned

a store. Right? And I needed someone to work the cashier.

Right? I could have another child, Raise that child, change

his diapers, send it to kindergarten, teach it to learn, read, and write.

And in about 10 years, depending on labor laws, let's say

15 years. I'll have someone who can work that cashier,

plus however much it costs. Now, obviously, I'm not comparing a child to an l

m, But I mean or you could just find an existing person

out there, and say, here's how my register

system works. This is the nature of the job, And I can kinda start from

there as opposed to start from 0. You start from the 50th floor as opposed

to start from the basement. That's exactly

right. Yeah. I often think about, you know, these,

pre trained LMS is like, well, what if I had like an army of

like Cumulative high school students, you know, in high school, you study all the

general subjects that kind of like a at a broad level. Right? So you know

a little bit about history, a little bit about how to write, a little bit

about how to You're not really an expert on any of those? Well,

the really interesting thing becomes then how you do, like, the vocational training or kind

of, like, you know, the task specific fine tuning It's how we think about it

in ML parlance. And, I think that's where the cool opportunities get

unlocked. It's really amazing to see the fact that you can scale up to, you

know, as many intelligent agents If you want, but then you need to, our

favorite customer quote is generalised intelligence is great, but I don't need

my point of sale system to recite French poetry. Right. So it's great that

you can go ahead and, recite history and others, but, like, how do you do

something very individual is what our platform is, oriented on.

No. That's that's a good point. That's that's a good point. Like, I I often

say, like, you know, do you want your cardiologist to be

also be a CPA, Or do you want them

to be a good cardiologist? I know if I were under an operation, I'd

probably wanna go with someone who was just all in on cardiology,

You know? Yeah. But, And those are actually the

2 trends I think we're gonna start to see with Gen AI, overall.

I think, you know, one trend is going to be People are gonna start thinking

of use cases that are more creative than just, you know,

question answering chatbot. So, you know, I think, like,

9 months ago, everyone I was talking to was like, I want chat g p

g provider enterprise, and I'd say, okay, what does that mean to you? And they'd

either shrug and say no idea or they would say like, you know, I wanna

be able to ask a question about The truth is if you had this access

to this, you know, army of agents that are like high school capable, I'm sure

we can think of more interesting things. Just basic question answering.

And then the 2nd big change I think is we aren't gonna use as much

of these super general purpose APIs in production. They're the easiest way to

experiment and get started. In production, you're gonna want your cardiologist to be the

expert in medicine and you don't really care if they know how to change a

tire or not. Exactly. That that is a a really good way to

put it. And I think that, you know, people, we're

still have to realize that we're still in the very early stage of this,

For lack of better term revolution. Right? Like, you know, because you're right. Like, I

talk to customers, and they say, we wanna we wanna get all all in on

Gen AI. Okay. What are you gonna do? Well, we wanna chatbot.

Okay. I don't know if you've seen

this. I'm sorry. Go ahead. Oh, I was gonna say,

And it's not not necessarily a bad starting point, but, you know, there there's so

much more out there. Sorry. Well, no. I mean, exactly. Right? It's like, I want,

if you could do anything in the world, what would you do? I don't know,

take a day off, like, you know, but but that's you're missing the point, like,

you're you are, there there's a meme going around. Again, I don't know

if it's true, it's Screenshot where a, car

dealership, had implemented some kind of chatty p t. You've

seen this, you're nodding. Right? Where it basically sold a guy a car

for a dollar, and basically, the person got it to

say, no, this is a legally binding contract. Basically, Tricked the

chatbot into saying no. Totally. No backsies, I think was the first phrase

to use. Right? And he he got it to say things like, oh, no. Absolutely.

I wanna make you a happy customer, And you can have this Chevy Tahoe for,

like, $1 or something like that, but he and I I don't know

how that's gonna play out in a court. Obviously, I imagine a

dealership is gonna have some, lawyers look into that,

and I'm not a lawyer, but I I can I can easily see like, you

know, this is a great example of, To your point, do you really need your

point of sale system, you know, re be able to recite

French poetry? Right? Now, I guess if I were, You know,

a very niche kind of bookstore slash

coffee shop, maybe? But for the most part, no. Right? And

and obviously, Yo. There I wouldn't classify that as a

guardrail. I would say that more as a domain kind of boundary.

But, you know, these chatbots are gonna need Guardrails too. Right? Not just the

obvious things that we always hear about, you know, but also, you

know, don't wanna be giving away. I haven't priced

what a Tahoe cost, but I imagine it's much more than $1.

Yeah. I bet too. Yeah. I think it's actually a function of 2 The first

is we need some better infrastructure on guardrails of what models can and can't

say. And actually, by the way, this is where fine tuning is actually very

useful. It restricts, Like, it's one of the best ways to reduce hallucinations. It,

like, teaches the model this is the type of thing that you're supposed to be

outputting, but it's not bulletproof. And I think that

actually the more, meaningful longer term conversation

is if you believe, like, I believe, and I

think a lot of folks, Yeah. About working this industry do that AI will

become kind of a dominant aspect of most businesses

over the next decade. That like the companies that embed

AI are going to be the ones that survive and have differentiated value.

The ones that don't are likely gonna be less competitive. If you believe

that, it's also hard to imagine that you're going to defer all

control of the model to a third party. And that's where

things like, you know, It's one thing to say, like, we need the guardrails. It's

another thing, like, if you realize that if those folks were using something

like, you know, commercial API that's Behind a walled garden where you

don't have access to the model, you don't have access to the model weights. They're

kind of limited in what they actually can do. They can post process the

output of the results, but they can never really get that fine granular

level of control. And that's why we think the future is gonna be open source.

Because ultimately, people are going to wanna own those models, own the outcomes

of the part of the IP that they think is gonna drive a lot of

their enterprise value in the future. So our like, I would say our our

bet as a company is really on 2 things like fine tuning and

open source. And I think that, you know, the example you just gave is a

good why I think the world is gonna have to move into both of

those directions. No. That makes a lot of sense. I think that open

source is important for a number of reasons. I mean,

not the least of which is, you know, we we have seen recently that if

if if these things are behind a commercial firewall,

If, for instance, there was some kind of, I don't know, political shake

up inside of said company board, which of course would never

happen. Right? Never happened. Then

you you are taking down that risk. Right? Which is, I think, is another

reason why open source, just in Generally, an industry is is

popular because decisions tend to be made at the community

level. Right? Now, there's obviously flaws with that approach

too, but It is, and I would use this as an example

of if you look at HTML and JavaScript Yep. Versus

say Flash and dare I say Silverlight. Right? Flash was

always a proprietary product. Silverlight, if people remember it, was also a

proprietary product, but HTML,

JavaScript Had its flaws, but eventually, they did get their act together,

and it it has a certain more

implicit compatibility. And I think with AI, I think the

it's not so much about compatibility. It's implicit transparency.

You get with open source AI. Right. Is it perfect? Is it totally

transparent? No. That that's not the point. But the

point is you're starting at a much more Transparency almost

by default or transparent, maybe translucent,

as as as as a default as opposed to completely opaque.

Yeah. I I think that it's both the transparency and the

control that's critical. Yes. It's the fact that people do not only

introspect and understand what's happening, but They can edit and change, you know,

in instances. Even if you're like a lot of our models, users do not

edit 99% of the pipeline, But it's important that they're

able to edit all of it, and that they do make the edits to the

1%. And I think that exists for open source. And I think from just like

an industry macro standpoint, you know, Trying to fight open

source and developer platforms is like trying to fight physics,

basically. It's kind of against the natural working of those systems.

And so our view is that, you know, people are

gonna come out with amazing models. And some of them are gonna be commercial, and

some of them are gonna be open source. The open source Size of the pie

is going to grow, and I think you wanna see this here, right? Like it

has caught up, so quickly. Like the

open source attraction has caught up so quickly to everything else. Our

view is just like, what do you need when you want to use open source?

Well, you need the you need the infrastructure around it. You need to be able

to plug it into proprietary, settings. You need to be able

to create those guardrails around it. That's, you know, where we think about ParetoBase

providing the info For being able to use open source. Interesting.

Well, this is a fascinating conversation. We could probably go on for another hour or

And I definitely would love to have you or someone else from Credit Base because

I think, you know, it's just a cool idea. Right? Like it and

and I think that it it really solves a missing piece of the puzzle

In terms of making this, you know, when you say

YAML, when I think YAML, I think OpenShift, right, obviously, you know, work at Red

Hat, that's kinda, but I mean, I think that,

it's one thing to open source the model. It's quite another to how do you

manage and control that animal? Right. Because these are

not these are not tiny little things. Right? These are

potentially very compute intensive activities. Right. So you

don't want you wanna be efficient. That's the way the world has gone.

Right? It's more compute intensive and,

heavier weight, and so that's where the infrastructure components become

critical for any company that's actually gonna use it. Absolutely. And you have to at

least If you can't be a 100% efficient because you really can't,

but you wanna at least, prioritize towards compute efficient

Activity. Because otherwise, you are literally throwing money out the

door. And I think that it looks like

your tool is really good at kind of Making it

so it's compute efficient, like, or at least that that

it goes a long way to helping that. I'm sure you can probably do some

serious damage With any tool. Right? Like, I wouldn't give my my 2

year old a chainsaw. You know what I mean?

But, now that's interesting. So

now we're gonna transition into the pre canned questions.

How did you find your way into data Or AI. Like,

did you find AI or did AI find you?

That's an interesting question. I,

I first got into it just out of studying

computer science. You know, I when I went into university, I thought I

wanted to study economics. Really liked, you know, the theory

behind economics. I took a intro to computer science class because I thought it'd be

interesting. And that more or less just completely shifted where I went

because CS was actually magic. You know, economics is a great way to be

able to explain things that were happening in the world, but with computer science, you

could actually build systems. And that was really interesting.

And then I found the 1 piece that I think I liked just as much,

which was statistics. And the natural

marriage of computer science Statistics really is, you know, data and data

science. And so, I'd studied it for a while, and then

when I went to, Yo. Go work in in a professional industry.

I first started off as a PM at Google, and I worked at completely different

things on Firebase, developer platform, authentication, security. I

remember somebody saying like, you know, you have to work on what you're most passionate

about. You know, a new college graduate, I have no idea what I'm passionate about

professionally. And so I thought back to, you know, the things that I'd studied that

I found the most interest in, that I found the most fun to work on.

And it really was those data science projects, Honestly, starting with the early

Kaggle competitions that I did in 2013, where you were trying

to compete to see who could build the best housing prices model who could build

the best recommender system model, and you had to exploit all

these interesting nuances in data and models to be able to get there.

And so I just found it so fun. And then

I think after a little while, found it trading

that everyone else didn't have sort of the same access to those types,

those types of experiences and tools. And so that's where the experience really

began. I would say, you know, early on, just having that academic

background and then seeing the problems kind of being manifested in Google and

eventually, you know, working as well on Kaggle of the data science and machine learning

community there. Interesting. Interesting.

I see you did a brief stint in cybersecurity for a while,

Which is funny because I think people see that as a as a totally separate

discipline, and in in a very real sense, there is. But I think that in

a very real sense, A big chunk of cybersecurity is

monitoring logs and input data and figuring out what's happening.

Sounds at all sounds familiar. Doesn't it?

I think cybersecurity, you know, when I was doing cybersecurity, work, it

was very, very much in the early days, strategic, how to

think about risk postures at an enterprise level. Right. But I think what's

really interesting now is, cybersecurity and AR are gonna have

a very interesting marriage where Cybersecurity is gonna be influenced

by AI. For example, we work with 1 company today that does open source supply

chain security, and they're looking at using LMS to read code and be able to

do things like Identify vulnerabilities, advise on remits, and

others. And so one obvious area is going to be that

cybersecurity companies themselves are gonna get revolutionized with AI. But

But this is gonna be one of the industries where there's kind of like the

bidirectional era as well. AI is gonna need some cybersecurity

best practices too. Yeah. These made these weights are now,

open source. How do you think about whether or

not the security governance Factors should be

on the inputs, you know, when the data is fed into the model,

in the model layer itself, like, how the model processes

that data On the outputs. Like, what is the framework for thinking

about, like, you know, which ones introduced what kind of risk? And the type of

industry that's had the most experience in this historically has in the cybersecurity industry,

Thinking about how we deploy software internally and others, and so that

marriage is gonna be, I think, really interesting. I bet there's gonna be really best

of breed companies in both worlds. I could totally see that.

I think that's a very good cogent response to,

you know, these are not isolated industries. Right. I mean, they

obviously have different origin stories, but I I could

totally see them merging. And to your point, right? I mean,

Yeah. If you look at potentially 2

things, right? 1, the, who, the amount of input

data that you have, like, Could that be poisoned in a way that could produce

negative effects later on in an LLM? And 2,

We don't really know the sort of latent, for lack of better term, latent spaces

that exist in these extremely large complicated,

models like for I'm sure you've seen this, but there was a random

string of characters that would produce bizarre output

In chatty b t. And there was also one that would basically short circuit

the, the safety rails inside of

some of these LLMs too. And it was just like,

wow. I mean, you know, was that the one, how was that figured out?

Was that random, or did somebody kind of understand that there's Weird

latent spaces and how to manipulate that. I think that is gonna

be a new frontier opening up, in the

not too distant future. If it hadn't already happened,

honestly. Yeah. I agree. I agree. And I think

it starts with understanding that, You know, those those

bits of, I guess, entropy that feel random to us are,

are more features oftentimes than bugs. So the fact that the random characters

produce, like, a weird output, it's actually really interesting

because what that means is maybe I don't need to type out a full

English Paragraph to get this model to do what I want. You know, there's really

cool things in prompt compression where people have basically been like, can I just

say, like, a couple of characters AFD, something that would mean

nothing to you and I, but the model understands that means, okay, go ahead and

pick up the dry cleaning on the way home and then make sure that you've,

you know, swung by and filled Like, essentially a set of instructions that get compressed

into this model's internal representation? So I think we're barely

scratching the surface of it, It's one of many ways that the I think,

l m revolution is gonna be really interesting in the ways that we haven't fully

explored yet. I could have said it better myself.

Our next question, what's your favorite part of your current

gig? My

favorite part is Probably the part that's also, I think one of the most

challenging is the space is moving so quickly. I know people

say that frequently, but the truth is I've heard people say that about different

technologies historically, and I'm like, yeah, it's moving faster than other

things. You know, for example, Mobile moved quickly.

There were over many years to transform things that happened.

The Timescale that our world is kind of, dominated. I'm gonna

say our world. I think it just mean, like, you know, the the AI movement

so far over the last year It's it's in weeks. Right? Like, every

few weeks, there's a new seminal groundbreaking, whether it's,

Yeah. I I can think about the moments where, like, Llama got introduced as an

open source model. Its weights got leaked. That was amazing because it spurred out of

the whole new community. GPT 3.5 got upgraded to GPT

4, new set of capabilities that came out there. LAMA 2 came out

this year with commercially viable licenses and like, You know, really, I

think, best in class performance up to the

point that Mixed Straw came out, which was a, you know, mixture of experts

model significantly smaller doing as well as chat g p t. This was only

a few days after Google released Gemini, you know, their own, model.

We have AWS in the race with Bedrock. It's kind of like, you know, an

interplay between different providers. I'm saying a

lot of sentences, but like the The really interesting piece of it is all that's

really come out in the last 6 months, and I haven't even covered up, like,

all the academic, you know, like It's wild. It's wild. Like, so I

was on a cruise, like, we were talking in the virtual green room, and I

had intermittent Internet, and I looked at my phone far more than I should,

for being on vacation, but it was just like Gemini happened,

AMD, and made some hardware announcements. And I know

hardware In the the unintended

consequence of being compute intensive is that hardware starts to matter again.

Right? Yeah. There was if you were a software

engineer, obviously, mobile, let's let's take that in the conversation.

But if you were a software engineer building websites, hardware wasn't really a major

Concern. Right? It was kind of pushed to the side. I mean, it

mattered, when you got, like, your Amazon bill was through the roof

and you weren't as efficient as you should be. But I mean, it wasn't really

a major concern. Now we have let's say it's starting to be a limiting factor

in terms of, you know, how many h one hundreds can you get your hands

on. Right? It's it's,

no. But, but you're right. Like, I mean, just I missed a week and I

still feel like I'm catching up and that was like almost 2 weeks ago. So

Yeah. And the, and that's the most exciting piece for us.

Right? It's because, all this changes created a lot of opportunity. So

We got a lot of popularity recently for something called Lorax.

Mhmm. It's an open source project that we released that basically,

was just a problem we had to solve for ourselves. It's the industry is moving

quickly. We needed to allow people to fine tune and serve large language

models for free in our trial. Now every single one of

these l m's requires a GPU and sometimes bigger, heavier,

meatier GPUs. And so if we're giving away a lot of free trials To, you

know, people just on the Internet who are all using a GPU,

investors would not be the happiest. And so we needed to figure out a better

solution where we could actually serve Many, potentially hundreds of these

large language models on the same individual GPU. And

so we, we came out with a really cool technique to be able to do

that. We called it Lorax for LoRa Exchange.

And, we open sourced it and back a lot of popularity. One of the reasons

that I think it got picked up in such a way was because it really

kind of just fed into them kind of main, main thought process in the

moment And everyone's staying up to date on kind of the latest. So, you know,

it kind of fed nicely into that hardware constraint, area of the world

as well as kind of a need that the market had. And so It's been

really fun, I think, to just be on top of that. Very cool. Very cool.

So we have 3 complete this sentence, questions. The

first one is when I'm not working, I enjoy blank.

I have a very San Francisco Answer to this question. But when I'm not

working, I enjoy being outdoors. And in

particular, I really enjoy biking, taking a road bike and going up a mountain,

because the reward at the end of that's amazing. And playing tennis, those are

probably the 2 things that, you know, I I enjoy the most. Very

cool. The San Francisco is perfect for that sort of thing, like the bikes in

the mountains, in the ocean. It's gorgeous. Yeah. Yeah. It's

gorgeous. I think the coolest thing about

technology the coolest thing in technology today is blank.

The accessibility. I think the coolest thing about technology today is the fact

that I can go ahead and run GPT four

Or llama 270,000,000,000, the commercial variants of, you

know, the leading edge or the open source variant. I can run both

of them More or less for free, at least to try out

for, like, you know, a little while. And that's sort of the same thing that,

you know, big bank over here is gonna be using Or, you know,

leaving technology company over there. Now, at least as the starting

point where it starts to diverge is like how, when you get heavier into the

customization and others. The coolest thing about technology to me is

in, and again, I think of it very much from like an AI centric lens,

just given my day to day. But, it's the fact

that, you know, I, the graduate students, you

know, somebody abroad in a different country, And then you know the m

l engineer at a company like Netflix, all have some shared experience

of language based on technology that just came out this year

Because the barriers to entry are not significantly high to be able to get

started. Now, I think the barriers to entry are still too high to, you know,

go from prototype to production. That's what we wanna be able to lower, but that's

to me the most compelling thing that we've done. That's very cool.

The 3rd and final Is I look forward to the day when I can use

technology to blank.

That's a good question. I think I look forward to the day,

when I can use technology to, to be sort

of like the Adviser and whiteboarding

buddy, if that makes sense. So if you think about,

like, what you often do with an advisor, it's, It's

actually generative in a lot of ways. You'll walk through them with a problem.

I do this with my dad all the time. And so, you know, he and

I will talk through Some challenge that I'm thinking about at work

or or something else. And he doesn't have all the context, you know, that that

might, but he's able to apply these like general frameworks and come up

with a few different types of suggestions based on based

on that. And some of them, because he's coming from a very different place, Might

be different than the way that I thought about it. And I

actually see that as a capability for,

For technology that as we've come up with it as well is to be, you

know, you've actually seen like companionship apps in terms of like, you know,

psychological help or behavioral help or, or Or just having someone to

talk to is actually like a use case that these models have already

started to pick up on, within like a niche group of users. And what I

think would be interesting is, you know, if you think about what you probably lean

on friends or family and other types of things for, I

think should still be friends and family and others. They are the ones who know

you best, but the model can be like one additional source of that

input. And it's gonna be really cool when, like, you know,

if you're if you're working through something hard and you wanna go ahead and, you

know, you get, like, get a few ideas for how to be able to go

through it, You can text your family group, you can text your friend group, and

you can ask the model that knows you, and you can kind of pick the

best idea amongst those 3. That's a great idea. I think that, a

lot of the media hype around things like replica AI and things like that has

been like, oh my god, it's gonna replace human interaction. And it's like, Are

they intentionally missing the point, or is it clickbait? Like, I can't tell.

Right? Are they are they are they clue are they clueless by default, or are

they clueless to make money? Not really sure. But I think that you're right.

It's meant to augment. Right? And I think that's a very healthy way to look

at it too, you know. Because I if I get stuck writing something. Right? Like,

I'll I'll ask chat TBD. Like, hey, how would you word this?

Right? Sometimes it comes up with a good answer, but at least it it kinda

clears the log jam in my head Where I'm like, oh, okay. Let me let

me go around it this way. I think that's a, I think that's an

underrated use for AI or these LLMs.

Yeah. I totally agree. Share something different about

yourself. We always joke, like, you know,

remember it's a It's a it's a family, iTunes

clean rated podcast. Something different about

myself. Yeah. I don't know if it's different or at least something that,

Not a lot of folks know about me, like, when I, first, first got

with them, but, I'm a 1st generation immigrant, and as is, like, my entire

family. So I was actually born, in India, came over, you know, when I was

a lot younger. So that I think is interesting because

I was both that, but also grew up right here in the Bay

Area. You know, I I think very much saw, like, the tech

I I think very much saw 2 things. One of them was just the US

kind of as, corollary and adjacency to to India

where, like, parents had spent the vast majority of their lives and, you

know, where we had come from. And then the second was like a very specific

part of the US with Silicon Valley that was just, had a

very interesting culture, Some healthy disregard for the

rules in some regard, not always for the best, but sometimes for the best.

And a real kind of inclination towards, you know, moving very quickly and kind of

being on the latest since and and and Barry progressed in that way. And

so I think that, This might be a little bit more of a backstory

than an interesting individual facts, but I do think that, you know, that,

immigration To especially this area, I think

was kind of a very, at least different experience than what

I think a lot of other folks that I've talked to have. Yeah. I often

wonder what it would be like to grow up in the Bay Area, and I've

met some people through through work and things like that who did. And they're like

It's hard because if you if it's if you grew up there, it's kinda all

you know, so you don't really have a good Yeah. Benchmark. Like, I grew up

in New York City, and people are like, oh my god. How could you grow

up there? I'm like, I don't know. It was just So I I

grew up in the Bay Area and then went to school in the northeast and,

you know, there's some things you realize, definitely. One of them

is, Yeah. Fewer people wear, like, hoodies and, you know, flip flops,

boat shoes are more of a thing. Like, there's all sorts of funny changes,

You know, that exists culturally, especially. I think the

biggest things that I've kind of picked up on is, like,

The Bay Area has a very kind of, or at least I think where,

the environment I grew up in, a very like, risk forward culture. It's kind

of a why not, worst thing happens. Whereas I feel like a lot of other

areas are a little bit more steeped in tradition And views

that as a good thing. I think the Bay area

potentially, and not to say one is right or wrong, but I think the Bay

area has a bit more of a culture, A healthy disregard

for tradition. And, you know, I

think, Sofia had the great quote about tradition,

That I'm forgetting. But, like it's,

yeah, I think it's one thing that I definitely think about, especially the difference between,

like, For example, where I grew up in the northeast, where I spent some time.

Right. Right. And you were I'm I'm inferring because you went to Harvard that you

were in Boston, and Boston is kind of its own Yeah. Its own corner

of the northeast. If you ask somebody, like, you

know, if you ask, I've lived in Europe, I've lived

in, in new in

New York and now the DC kind of Richmond, now

Baltimore. There are slight variations in culture, but like, I

can only imagine like how much of a shock it would have been from like

the bay area To, like, Boston, especially.

Right? Where it's it's far more I think things are far more rooted in tradition

there. Right? Yeah. And it's it's not a knock on it. Right? Like, I I

will knock on their baseball team, but that's another another story. Right?

But, you know, but still, the both I mean, the

the Boston area is also known for its innovation in both

biotech and technology. Right? So it's not, These are not mutually exclusive

things. Right? They're just different approaches.

Absolutely. And both of them have worked, you know, really well for those respective

Areas. One of them feels a lot more at home to

me. But I think, you know, it was fun and interesting to kind of see

those 2 differences, Especially spending time in both cities.

Yeah. That's cool. That gives you a unique perspective on, you know, that the

US culture is not one monolith, it's just Fragments of

different things. It's it's an interesting perspective. I almost

have to ask, like, was it as much of a culture shock coming to the

US or coming from the Bay Area? Well, honestly, the Bay Area to

anywhere else. Right? You know, the weird thing

is I didn't expect the culture shock to I expected the culture shock coming to

the US. Both from you, but you know, I was young, especially for my family.

Yeah. I think that was there, but you're kind of, you're expecting

it. And so it's always something that you're well prepared for. I don't think I

expected the culture shock going from the Bay Area to to Boston.

Because these are the 2 cities in the US. These are 2, you know, Progressive

cities that are well educated in the United States, how different can they be.

And you don't actually notice the difference, I think on a one day or two

day visit, you kinda notice the difference when you actually spend a longer period of

time there and understand the undercurrent. So Yeah. It

wasn't a shock actually as much as it it was kinda cool. Like, I appreciated

that 2 places in the US could actually feel very different because,

you know, diversity is the spice of life. So actually really, really, I liked

it even though it was different to maybe how I thought. That's cool. That's

cool. The winter must have been a good shock on you. The

winter was a shock in less of a positive way. Yeah. Diversity is a spice

of life minus in weather. Yeah. I'll say

70 degrees sunny year round all day. Were you there during the year? They

had, like, a record amount of snowfall, like, something like Yeah. Fifteen

feet over the winter? I was. Yeah. Yeah. Exactly. Yeah.

Yeah. Campus shut down. Yeah. I was a student then,

and, You know, as I was saying, very healthy risk

appetite. I think everyone was out in the yard, like, throwing snowballs at each

other while there was, like, a record blizzard So it was, it was

fun. It was less fun when the snow was still on the ground in Maine,

June. That was when I was thinking, get out of here.

Do you listen to audiobooks at all? Yes. I

I read more often, but sometimes I do re I listen to audiobooks to conveniently

Do you have any Recommendations?

I really like The Happiness Advantage by Shawn Achor.

It's yeah. It's a book about how,

I think there's a thought process that, you know, like, success breeds happiness,

but this is also, like, work by a behavioral psychologist. Like how happiness can breed

success and just how to be able to be in that mindset more often. And,

you know, it's a weird book because it's actually kind of style as a business

book. But I actually think it's a lot about like personal development. And

so, yeah, that's definitely one I'd recommend.

Cool. Audible is a sponsor of the show. And if you go to the data

driven book .com, you will get, 1 free book on us. And,

if you sign up for a subscription, You get a we

get a you get a subscription and of knowledge, and we get a little bit

of a kickback for them being a sponsor. And

finally, where can people learn more about you and Predabase?

Yeah. Absolutely. So, the obvious and easiest answer there is of

course prediabase.com. I think, you know, we've learned,

the easiest way to learn more is just to go ahead and try it.

And so you'll see things there like documentation, you'll see a bunch of

videos on our, blog page, which are short, 3 to 5

minutes, and our YouTube channel, on prediabase, p

r e d I b s e, actually has longer form 1 hour pieces of

content that are more educational. But I'm a big believer that the

easiest way to actually learn is just to be able to get your hands dirty.

So if you click that try for free button, you'll get a few weeks, and,

you know, credits. We'll give you the GPU out of the box so you can

run all these models yourself, and you can learn firsthand. That's usually the easiest

way, you know, to be able to get Started more. And then if you wanna

learn a little bit more about our underlying technology, we've open sourced

both of the key components. So for how to train models, we have Ludwig,

And then for how to be able to serve models, we have LAURACS. And

so those are the 2 l's that you can kind of use in order to

be able to understand how the tech works under the hood. Very cool.

Thanks for joining us in the show, and thank you once again for your, patience

as we work through some scheduling conf conflicts,

And, I'm glad we had this conversation. You're always welcome back in the

show, and I'll let the nice British AI lady finish the show.

Thanks, Frank, and thanks, Dev. What a

splendid conversation that was. It felt like

navigating through a maze of data with only the smartest chaps as my

guides. To our listeners, I hope your brains are

buzzing with as much excitement as mine is metaphorically speaking,

of course, since my excitement is more of a series of well organized

algorithms. To our dear listeners, if today's chat

has ignited a spark of curiosity t in you, then I dare say we've

done our job. Remember, the world of AI is vast

and ever evolving, and it's thinkers and doers like deaf who keep the digital

wheels Turning. Before we sign off, a gentle

reminder to keep your minds open and your data secure.

Until then, be sure to like, share, and subscribe as the

kids say these days.