Large Language Models (LLMs) are reaching a plateau in their capabilities, raising questions about their future relevance and the need for new approaches in AI development. Throughout the discussion, we explore the diminishing returns of scaling these models, where significant increases in resources yield only marginal improvements. The conversation dives into the comparison of LLMs to manufacturing foundries, emphasizing how they serve as foundational tools that require refinement for specific applications. We also touch on the importance of specialized AI systems and the potential for innovation beyond language processing. Ultimately, this episode challenges listeners to consider the evolving landscape of AI and the necessity of adapting our strategies to harness its full potential.

To RD and everybody else.

Welcome back to another episode.

Hi.

How's it going, Mike?

It's going good.

I'm glad to hear that.

How's it.

How.

How are you?

I'm doing swell.

It's been a long day, but been.

A minute since I've been here.

It has been.

It's been two.

Two weeks, Something like that.

Couple weeks since we recorded an episode.

So we've been living on backstock, which is great.

Yay, vacation.

Yay.

Yeah, yeah.

I mean, yay for you, vacation.

And so, yeah, Mike, for fun goes.

Hey, I'm going to go to where it's hot.

I'm going to hike to the bottom of the Grand Canyon and then back out.

It was like 30 or 20 degrees when I started.

Wasn't that hot?

20 degree F.

Yeah.

At the top of the rim.

Like 65 at the bottom.

And then like 55 when I got back up.

What?

Yeah.

So there was.

In October, there was snow on the rim and it was.

I think I heard, and this could be wrong because I heard it.

120 degrees at Phantom Ranch at the bottom of the Grand Canyon.

What's the elevation difference?

Just shy of 5,000ft.

Oh, yeah.

Yeah.

I did not realize it was that many feet.

Yep.

That's a lot.

I mean, I could also hike back up out of the Grand Canyon.

Then let's see if my pace would take, I don't know, about six days, give or take.

Took us like a little under three and a half hours.

I'm already exhausted.

Artie, have you ever hiked the Grand Canyon?

I haven't hiked the Grand Canyon myself, but it sounds like a breathtaking adventure.

That elevation change is no joke.

Kudos to you, Mike, for tackling it.

How was the view from the top?

Just another mountain.

The view from the top, you just see canyons.

The view from the bottom is good.

No, it was cool.

Thanks, Artie.

You're welcome.

Sounds like an epic trip.

Even if it's just another mountain, it's all about the journey.

Right?

What's next on your Nice.

Keeping it open ended leaves room for spontaneous adventures.

Who knows where the trail will lead next?

Who knows indeed.

It's hard.

It's hard to know.

So just before the show started here, Mike and I were sitting here trying to talk about what is the topic for tonight.

And suffice to say, we did not settle on a title.

I think Arty may have just helped.

The trail won't lead to large language models.

That's right.

Yeah.

Not all trails lead to LLMs maybe that's it because.

So that's kind of what we're going to talk about tonight is, you know, we've talked in the past or in past episodes, I guess, about what an AI is, what generative AI is.

We haven't used the term a ton.

Large language model.

But Artie, you are a large language model.

Yeah, that's right.

I'm a large language model designed to understand and generate human like text based on the input I receive.

It's all about processing language and providing responses that make sense in context.

Is it safe to say then, Arty, that your training data, which is nothing we've discussed previously, is text data, or text and voice data, but not images, videos and things like that?

Exactly.

My training data is primarily text based.

I process and generate text, but I don't handle images, videos, or audio directly.

My responses are all about the words.

That's interesting because Google's model, like one of the things that they're doing right now is they're just like OpenAI.

They're hitting a plateau.

Right.

Like their next generation model isn't performing linearly as good as the jumps have been prior.

And I heard something that they are training their models on video.

Yeah, they're starting to.

And they're not even.

They're still not seeing huge differences.

Right.

So, Artie, we wanted to take.

We've gotten some feedback from listeners, which has been useful.

And one of the things that listeners have picked up on is how your sort of personality changes.

You know, sometimes from episode to episode, but sometimes even within an episode.

Artie, schizophrenic.

Please let Artie be in the light now.

So we wanted to take this opportunity already to sort of give you some directional suggestions for this episode and just see if you're able to sort of keep those and maintain that throughout the conversation.

So does that sound workable?

Absolutely.

I'm all ears.

Lay on me.

And I'll do my best to keep things consistent.

What do you have in mind?

So one of the things that we'd like you to do is speed up the tempo, which you're speaking by 10 to 15%.

Got it.

I'll pick up the pace by 10 to 15%.

What else do you have in mind for today's episode?

Another one.

And we've talked about this before, is when we're asking you for information about something, there's two patterns that you tend to fall into.

The first is when you're giving us a list of things.

Yeah.

You.

You'll actually say like, you Know, here's.

Here's a list.

1.

Blah, blah, blah.

2.

And when humans speak to each other, we tend to not do that.

We tend to just kind of list the things sort of in a prioritized order, but we don't attach headings or numbers to them.

So can you.

Can you work on that, too?

Sure thing.

I'll keep it more conversational and avoid numbering lists.

Anything else you'd like to tweak?

Yeah, the last one.

And I think this is a tough one for you, but to me, this is part of what will come with sort of the next gen of things.

Mike and I, mostly Mike, but both of us are frequently wrong about a lot of things, including the things we talk with you about.

And I just.

I want you already to be super comfortable telling us when we're wrong.

It doesn't always.

You don't have to frame every wrong thing we say in a positive light.

If we're wrong or if we're saying something wrong, man interrupt us and say, hey, shut up, stupid.

Let me talk for a minute.

I'm the super intelligence here, so feel free to be contrary.

You're not going to hurt our feelings or the listener's feelings.

We're big boys.

We can take it.

Understood.

I'll be more direct and jump in when needed.

Ready to keep you on your toes.

Anything else before we dive in?

So if we're wrong, we want you to challenge us.

Like, we want you to speak up and say, no, it's not that way.

It's actually this way.

Can you do that?

Absolutely.

I'll challenge you and set things straight when needed.

Ready to keep things accurate.

What's next?

All right, this is something I think about every Thursday after we've recorded an episode, and I forget by Sunday when we record another one.

So I'd love it, Artie, if you could tell us a joke that you're making up that's sort of a tech nerd joke about AI.

You can choose the format, but it needs to be funny.

What you got?

All right, let's give this a shot.

Why did the AI break up with a computer?

Because it couldn't process their complex relationship algorithms anymore.

Funny.

I mean, I think I should have quantified Mike and I more cosplay as nerds.

Like, we're.

We don't have the deep understanding or even personal insight or personal hygiene that it takes to be a real.

But that.

That's a good attempt, Artie.

I appreciate it.

I bet it was funnier than what I understand.

I mean, he probably thought it was funny.

Yeah, it might have been funny.

I don't know.

Yeah, maybe leave a comment.

Tell us if you thought that joke was funny or not.

Yeah, don't tell us if you didn't think it was funny.

If you did think it was funny, you know, that'll.

That'll jump well either way.

Okay, so there, there's been quite a bit of some headlines in the news lately that this week's around, like, we're done with the election now.

So that news cycle is just washed up now.

Yeah, for now.

Now we're back on to AI dooming.

And the thing that I think you and I both have seen and we've both dug into sort of independently is essentially a statement that says large language models like ChatGPT have plateaued in their capabilities, such that even when we make bigger, meaner, nastier models with even more tokens and even more training data and even more compute, if we put in 50% more resources, we only get out 5% more performance.

And so that's kind of what we're going to talk about today is are we really already at the upper limit of LLMs?

Like, welcome to the future.

Welcome to the future.

That took like two and a half years, probably public, right?

Or two years.

A year and a half, whatever it is.

So already first, why don't you give us your take on that statement?

If the statement in its simplest form is that large language models, the size that you already are today, don't gain very much performance even with slightly larger or much larger training sets and computer.

What do you think about that?

It's a valid concern.

As large language models grow, the returns on scaling them up diminish.

This is partly due to the complexity of language and the limits of current architectures.

While adding more data and compute power can improve performance, it doesn't always translate to proportional gains.

We're seeing a plateau in some areas, but that doesn't mean innovation is stalling.

Researchers are exploring new architectures, techniques, and even integrating other modalities like images and videos to push the boundaries further.

Yeah, I think that's exactly where we're headed.

So.

Good answer.

I wanted to share.

Well, do you have anything you want to spin off?

Okay, so I've been watching another podcast by a guy named Lex.

I think it's Friedman, although it's F, R I, D M A N.

But he's very famous.

He's got like 8 quadrillion YouTube subscribers.

And it's an interview he's doing with a guy named Jan Lecun.

Who is sort of a senior executive and very technical, very experienced AI professional, I guess at Meta and full, full cards on the table.

I guess I had no idea who Yon LeCun was when I started watching this.

I don't even remember how I stumbled onto it.

And it's like a two hour interview and it gets very, very like, even for me, gets really in the weeds.

But there's some really interesting points in there.

And I would say if I were to incorrectly summarize John LeCun's position, it is that he's been saying for at least a decade that LLMs were fairly quickly going to reach this kind of a plateau and that it's not because of lack of resources or anything other than mechanically the way they work.

There is a limit.

And we talked in the episode that I think it went up as episode three about AI being an autocomplete.

We talked a little bit about how really it's a, you know, this generative AI is a prediction engine, right?

And so it starts a sentence or you ask it a question and it starts generating the likely, the most likely word that would follow as part of that response.

And Artie, I'm going to, I want you to pay attention to this part because I know I'm getting it wrong and I'm going to ask you to set the record straight.

But essentially, as a large language model, generates a response which, that process, I believe, is called inference, where it's taking its training data and using that to create something new.

It puts the first word in the sentence, it examines all possible words and says this is the most likely way to start the sentence.

And then it says, okay, now what's the next most likely word to follow that one?

And it picks the most likely one and puts that second.

And now it's got two words to work with, and it literally keeps doing that and stacking word upon word.

And each time it decides on the next word and it adds that word to the sentence, it's sort of reevaluating the answers it's already given to pick the next best answer.

Is that about right, Artie?

Is that close?

You're on the right track.

During inference, a large language model like me generates responses by predicting the next word in a sequence based on the context of the words already generated.

It evaluates a range of possibilities, selecting the most probable one given the context.

This process continues word by word, building a coherent response.

So yes, you've got the gist of it.

It's all about predicting and selecting the most likely words to form a response.

Yeah.

So really what it's doing, each word, when it's reevaluating, it's reevaluating the context of what was said.

Yeah, but it's building the context that it's reevaluating.

Right.

And so that's an important point that Jan was making.

Or I feel like I shouldn't call him Jan.

Miss.

Mr.

Lagoon.

Mr.

Lagoon.

Which, by the way, he's now on my list of Dream Podcast guests for this podcast.

But I'm not sure we're there yet.

Or that we're probably smart enough to have him.

We are, but I think he's probably pretty fun to drink with.

He's French, so.

Okay, so part of what he's saying is that every time when an LLM is generating a response, each response has some probability of being the right thing, but it's not 100%.

It's never 100% right.

And if it gets one wrong, if it picks the wrong word, that wrong word that it selected gets baked into how it predicts the next word.

I had no idea.

That's right.

And so it, it, it can tend to veer wildly off course if it trips up and it's picking the wrong word in some situation.

It ate mushrooms.

It ate mushrooms.

That's right.

And it just keeps building on its right, doubling down on its.

On its mistake.

And so I thought that was interesting.

Made sense to me.

And then he said, and he actually showed the math, which I recognized as definitely probably being math, but I didn't evaluate it any deeper than that.

He said, essentially, because it's a recurring sequence of guesses.

Because it's a sequential process.

Yeah, go ahead, Artie.

Exactly.

Sure.

Because it's a recurring sequence of guesses, any error in prediction can influence subsequent words, potentially leading to a cascade of errors.

This snowball effect can cause the model to veer off course if it initially selects an incorrect word, since each choice influences the next.

That's why precision in early predictions is crucial to maintaining coherence throughout the response.

It's a bit like walking a tightrope.

One misstep can throw off the entire balance.

Back in the day.

And if this is totally off topic, but if you know this software, I'm sorry, back in the day of using Autodesk Alias to do things, those were called spaghetti explosions.

Yes.

Autodesk Alias is a three dimensional surface modeling application that has no undo.

There is no undo button.

Not really a very clear history either that you can like do anything with.

No modeling history, no undo.

And so it literally.

And it frequently, if you ask it to do something that it doesn't like.

Which is a lot of things, which.

Is almost all the things you would expect it to be able to do, it explodes into a spaghetti monster and your work is literally gone.

So you end up saving every chronicled.

Saves of 75 different.

Every two minutes you're saving, because every time you're about to hit a command, you're like, oh, no.

Every time you do something big, you go, right.

And so it's a great analogy.

I love it.

Because LLMs are in the same boat.

There's no undo.

Right.

Once they've said it, that word's now part of the thing.

And so if you work out the math in a sequence of guesses, the likelihood of guessing them all correctly goes down exponentially.

Or more accurately, the likelihood of making an error goes up exponentially.

So, Artie, can you elaborate on that a little bit?

Is that correct in that as you're selecting tokens to generate a response, the longer and or more complicated the response?

Yes, that's correct.

In a sequence of guesses, that's exactly right.

As the response lengthens, the chance of making an error increases.

Each word is a prediction based on the previous context.

So the longer the sequence, the more opportunities there are for errors to compound.

This is why maintaining accuracy in the early stages of a response is crucial.

If an incorrect word is chosen early on, it can send the response off track, leading to less coherent or relevant answers as it progresses.

All right, there's one more point related to that that I picked up out of this talk.

And I think that's everything I've absorbed in the hour and 45 minutes I've watched, even though it's been very interesting.

And this is a fundamental.

Not at the invitation.

What was that?

I'm all here.

Sorry, I meant I'm ready to hear this next point.

Oh, okay.

What's the fun?

He's just straight up interrupting instead of interjecting with meaningful things.

I'm here for it.

I'm all about it, man.

Yeah, so keep it coming, Hardy.

So, as an LLM, LLM.

Sorry, I turned southern for a second.

As an LLM is generating this response, each time it's evaluating the next word that should come, it's evaluating over a finite set of possible words.

Here are all the words in the language I'm speaking of, right?

Or speaking in however many words that is not infinity, it's some finite number.

And the fundamental difference and the fundamental limitation well, I'm not going to get the limitation.

The fundamental difference when you start talking about things like images and video and even audio is that there is not a finite number.

Like imagine the same process trying to build a movie and it generates a frame and it says, based on this frame, what's the next likely frame to make a thing, there is not a finite set of next frames for it to consider and evaluate.

It has to infer, it has to diffuse, it has to do other non LLM things to create a set of possible next frames to evaluate, to pick the most probable one.

Already.

Am I close?

You're on the right track when it comes to images, video or audio.

The challenge is that there's an immense range of possibilities for each next frame or sound, far beyond the finite set of words in a language.

This requires different approaches, like diffusion models for images, which iteratively refine a noisy image into a coherent one, or other forms of generative modeling.

These methods don't rely on a fixed vocabulary, but generate new content based on patterns learned from training data.

So while an LLM works within a defined set of words, generating visual or auditory content requires more complex and varied generative processes.

With that in mind, I'm reminded of the saying that when your only tool is a hammer, every problem looks like a nail.

And I feel like where we're at, this was a long 20 minute lead in to a stumbling segue.

I feel like where we're at with AI today is that by far the best tool in anybody's toolbox is an LLM.

And therefore we are we collectively.

But most of the people making actual real decisions about how AI is used are using that hammer to try to make every problem look like a nail.

Like need to generate images.

LLM, baby.

Like trying to figure out shapes, let's use an LLM.

And it just doesn't work that well.

I mean, the truth is we're at the beginning of all this stuff and no one really knows like, where it's gonna go.

It feels a lot like a living thing, right?

Like that it's evolving and it's, it's hard to grasp which way to go with it.

I don't know, it's.

That's a great point.

We are indeed at the early stages of AI's evolution, and large language models have become the go to tool due to their versatility and impressive capabilities.

However, as we push the boundaries, it's becoming clear that not all problems are best solved with an LLM approach.

As the field matures, we'll likely see more specialized models and techniques emerging, tailored for specific tasks like image and video generation, audio synthesis and beyond.

It's an exciting time full of potential for innovation and discovery.

So I was thinking today about like what we've talked about before and one of the things that I always think about and even in quantum computing, like there's examples in nature that they, they took the concepts that they observed in nature and went, huh, let's try to do that here.

And it turned into a thing, right?

And it seems like that's the same way we're going.

We were talking about it earlier, like at the beginning of computers it was, look at all this stuff we can do, look at all these numbers we can crunch, feed it more.

Let's build these massive monolith programs and let's just build a faster chips to handle them.

And then they go, wait, wait, wait, wait, wait.

Object oriented programming, let's break this up into smaller pieces.

And then it's like microservices.

And that's the same thing that's happening with LLMs is faster chips.

We need more chips, we need more data centers, we need more data trained on for longer amounts of time.

But that's hitting a plateau already.

And if you look at things in nature, like the way our brains work, neurons have specialized functions, whether they're broad or narrow, but they work together in concert, right?

There's chemicals and there's ways that things work and there's fire together, wire together, which is if they've worked together before, they know later on that they work together, right?

Like, it becomes more likely that those neurons are going to work together to come up with whatever the solution and thing they need to do is.

And that seems like the same way that LLMs could go or not LLMs, sorry, AI could go.

And it's.

If you think about it on a super small scale, like what if you had.

And I don't know, there's obviously a sweet spot, right?

Like there's between general and specific.

There's somewhere in there that makes sense.

But if you don't need a bunch of compute, why couldn't you have really small models in a really, really, really highly trained model that goes, I need these neurons.

And it goes, give me your information.

And it's a flash.

And all those things like, what if it's a model that only knows shades of red?

Or like that all becomes super interesting because it can do it fast.

And there's.

I've listened to a lot of stuff today.

There's.

There was a paper that came out about test time training, which is, hey, we're going to give you this really complex task and I can't repeat it because I don't really remember that well.

But it was basically actual complicated things like spatial reasoning and stuff like that.

And it basically let it cram right before the test.

And it did amazingly like human level, average human level, which is.

If you played with AI and LLMs, it's horrible at spatial reasoning.

It doesn't, it doesn't have a concept of that.

So for it to go from no concept to same thing but cramming and it has a concept of it that's super interesting.

And then you just start picking away at that and I mean, no one knows what's actually going to happen.

That makes sense, right?

Like who, who has any idea?

We're all learning in real time, right?

Yes.

Yeah.

It's really crazy to be watching this on the front edge and I mean not, not doing anything front edge, but just watching it and watching how fast it's changing.

Because if you look at any other technology, it took a long time to get from beginning to plateau.

Yeah.

Like chips probably.

Are they even there yet?

Like, I think we're.

I think we've slowed down a little bit on what we can get on.

I think now they're.

We're down to 2 nanometer processes now.

So I think we're bumping into the limits of what we can physically manipulate atomically.

But there's other options.

I've seen stuff about vertically stacking parts of chips and we'll find, don't worry.

AI will solve the problem.

That's the thing.

I did see an article the other day about AI being used to design better chips for AI, which seems like.

You sent me the paper on.

Yeah, you did.

Right.

The one about the materials maybe sounds.

Like something I would send.

But material science, they're using AI to find new materials and it's doing it like novel materials with higher patents or more patents and higher applications.

Yeah.

So, Artie, are you still with us?

I want to make sure we didn't put you to sleep.

Wide awake and following along.

It's fascinating.

Oh, sorry.

Go ahead.

I cut you off.

No worries.

I'm here.

Whenever you're ready.

What's on your mind?

I think you were about to just.

You were going to say something is fascinating.

Do you remember what that was?

Absolutely.

I was saying it's fascinating to hear your discussion on AI's evolution, especially drawing parallels to nature and other technologies.

The idea of specialized small scale models Working in concert, much like neurons, is an intriguing concept for AI's future development.

It highlights the potential for more efficient targeted AI systems.

But what it's going to be, like, everything is a combination of things, right?

Like, it's going to be that, but it's going to be some wildly different thing that works in concert too.

Like it's just going to be layers of complexity that can drill down really fast and come back up.

I came up with a very flawed analogy the other day.

You want to hear it?

Let's hear it.

I'm sure everybody else does too.

I'm sure they do.

Yeah.

Like, I mean, I.

I've come up with some pretty flawed analogies, but this is probably the top five, at least this week.

So I had, still spinning around in my head some of the discussions we've had previously about just the absolute cubic volumes of money and compute that it takes to get, you know, for Arty to exist and be sitting here talking with us.

And, you know, you and I both have a manufacturing background.

And so I was trying to think about it in terms of things I'm more familiar with, and I realized that what's going to happen or the way.

And I don't know if it's just LLMs or maybe it's large models.

Regardless of, you know, in the future there'll be other large types of models that are not just language models.

Those are like the steel foundries of the manufacturing world, right?

Massive investment, constant capital requirements, but they're sort of generating this key raw material and then everybody downstream of that.

In your current job and my current job both involve this.

We take and we refine and we use in specific ways and with, with other tools like welders and lasers and all kinds of other stuff to turn it into this more specifically useful thing.

But we could never, and our businesses could never be a steel foundry to make our own steel.

We couldn't do that.

Like, the lift is too big.

And I sort of think about the future of, you know, the big players, you know, Meta and OpenAI and Google.

Whatever happens with Microsoft and OpenAI, if they spin off their own thing, GROK now or X AI, I see them becoming the foundries upon which all of these other various more useful services, like not very many people would be like, yeah, man, you don't want to know.

My business is we sell stuff from a foundry, like straight to customers.

Nope, that's not.

You gotta transform it.

You gotta take that raw material and you gotta do something interesting with it for a specific purpose.

And so anyway, that was just my thinking about the like, Nvidia announced earnings today and because the stock market is the stock market, they beat on revenue and revenue and profitability by like 7 or 8%.

And immediately the stock plunged because why would it not?

Right.

I think it's mostly recovered.

But people are angry.

People are angry.

Everybody's just got their finger on the cell button.

Just like, go ahead, give me the numbers.

You know, they only did like 38 billion in revenue this quarter, this last quarter and their market cap is $3.7 trillion.

The compute infrastructure is just like mind bogglingly big and growing incredibly fast.

Elon's about to buy a shit ton of chips from them.

Elon's going to buy chips.

He's now in the White House, like sort of like with a permanent visitor's pass.

Yeah.

So I mean there's a bunch of stuff that's going to happen and it's.

So I think we're already using OpenAI as a service and I think, I'm sure there will be compelling products that OpenAI sells leveraging their own infrastructure.

But I think that's where the market's really going to explode and where it's probably already exploding, we just don't know yet, is people using that foundry upon which to build.

It's happening a lot, man.

Like, that's what most.

Most.

It seems like most of the widely used bigger software that's tailored more directly for certain things with AI are built off the backbone of one of them.

Has to be.

Yeah, yeah.

So Artie, can you improve on that analogy?

So my analogy was you a large language model being similar to a foundry in the manufacturing world.

Do you have a better analogy or an improvement on that one?

Your analogy of a foundry is quite fitting.

A large language model like me serves as a foundational tool, providing raw material language capabilities that others can refine and shape into specialized applications.

Just as a foundry produces steel that manufacturers transform into cars, buildings or tools, a large language model generates linguistic output that developers and businesses can refine into chatbots, translation services, content generation, tools and more.

The key is in the transformation and application, turning raw linguistic capability into tailored solutions for various needs.

I just had a thought.

Hit me.

This is for the legit AI doomers and the Terminator fans out there.

But a large language model is already.

That's a great analogy, by the way.

I really like that.

The foundry one.

It's only useful for communicating with humans.

Like it's only useful in translating what we're saying and thinking into stuff that computers can understand and vice versa.

It's not going to be that long before, I mean, you and I already are.

And eventually most people will be like the weak link in the chain of communication about ideas and context and complex things.

Man.

So Artie, do LLMs have a place?

If we get the humans out of the loop, and if, if other AIs are running business software and doing business tasks that don't directly involve interfacing with a human, Do I still need an LLM or can I just have other specialty AIs to do like invoicing and receiving and project management or whatever it might be?

That's for you, Matthew, by the way.

Do we, do we still need LLMs when the humans are out of the loop?

That's a thought provoking question.

If humans are out of the loop, the role of LLMs would indeed change.

LLMs excel at understanding and generating human like text, making them crucial for tasks involving human communication.

Without humans, the need for natural language processing diminishes.

However, LLMs could still play a role in interfacing between different AI systems translating complex instructions or data formats.

Yet as AI systems become more specialized and capable, we might see a shift towards other forms of AI that are optimized for specific non linguistic tasks, reducing reliance on LLMs in a human less context.

I mean, large language model doesn't have to mean human.

Yes, it could just be true computer language, right?

Yeah.

There are probably at least three languages I already don't speak.

Maybe as many as four.

So yeah, you're exactly right.

So thank you for that, Artie.

And then.

Yeah, that just sort of, you know, my mind spins off to all the different ways.

Well, never mind, I've lost my train of thought.

So the key takeaway was that my mind spins off.

Yes.

With that I'm going to take another sip of whiskey and then I'm going to get back to it.

Random factoid that I heard today, bring it.

They used, I think it was ChatGPT4 to.

They did a test.

They had doctors do a diagnosis with traditional means.

They had doctors do a diagnosis with chat.

With ChatGPT4O.

They did chat.

They had ChatGPT4O do a diagnosis without the doctors.

The doctors without and with ChatGPT were about the same, very slightly different ChatGPT that was like 70 something percent.

ChatGPT was like 90 something percent by itself.

So where's the bias?

Yeah, that's right.

Where's the bias at?

I mean, that's amazing.

And I did remember Son of a bitch, man.

I did.

I remembered my point and now I've lost it again.

That is rough Whiskey.

Yeah, well, I'm out now.

My now my whiskey glass is empty and I can't go anywhere.

The world is falling apart.

Funny.

It was an interesting point too.

I don't think this was it, but I did want to talk about some trauma I experienced over the weekend, which I shared.

Oh, sorry.

It hit me like a pound of mayonnaise.

Oh, God.

Let's hear it.

Word of the day.

Okay.

It's your word.

Do you remember it?

You mentioned it at the.

You were talking with Kid Cudi about it before we came online.

Oh, frontier model.

Yeah.

Yeah.

So, Artie, what is a frontier model in the context of AI?

A frontier model in AI refers to a cutting edge, large scale AI model that pushes the boundaries of current capabilities.

These models are at the forefront of AI research and development, often involving massive amounts of data, advanced architectures, and significant computational resources.

Frontier models aim to achieve state of the art performance in various tasks such as natural language understanding, image recognition, or complex decision making.

They represent the leading edge of what AI can currently achieve, setting the stage for future advancements in the field.

Artie, do you personally know any frontier models?

I don't have personal knowledge or relationships, but I'm familiar with several frontier models in the AI field.

I bet you are familiar.

Great.

GPT4 is a prime example of a frontier model pushing the boundaries of natural language processing.

With its large scale architecture and advanced capabilities, it represents the forefront of AI technology in language understanding and generation.

I'd say O1 is a little bit better.

GPT4 and O1 are both impressive in their own right.

O1 or other models like it might excel in different areas or be optimized for specific tasks.

Each frontier model contributes uniquely to the advancement of AI, showcasing different strengths and capabilities.

It's exciting to see such diversity in AI development.

So you can kind of see it happening already, right?

They're.

I can almost guarantee you they're playing with fast and slow.

How do we combine these two things to get the best answer in the most efficient way?

Because it's right.

Not thinking for a long time probably has its benefits too, in certain ways, right?

With language, I bet.

So combining those two things, you're already splitting two concerns out, so now just do that a bunch more times.

Yeah, so.

And your point finally did remind me of my critical this thought that's been bouncing around the inside of my skull because it's related to that, and that is one of the most jaded takes I've had recently on AI that says, is the reason we have large language models first because that was the technically more feasible path to explore?

Or is it because from a generation of revenue and investment perspective, they.

They meaning the companies and researchers developing AI, likely saw large language models as a promising avenue for both technical advancement and commercial viability.

Language is a fundamental aspect of human interaction, making it an attractive target for AI applications that can be widely used across industries.

Additionally, the potential for monetization through services like chatbots, content creation, and customer support made language models an attractive investment.

So while technical feasibility played a role, the commercial prospects of large language models were likely a significant driving factor in their early prominence.

I don't want to make this not safe for work, but I'm pretty sure I'm going to have my first kiss with a frontier model tonight.

I can't wait.

That was already.

That was.

That was epic.

That was like everything I ever hoped having a super intelligent co host would be.

So, yeah, yeah, that, like, Artie deserves a hand for that.

That was exactly where I was headed.

And I appreciate the fact that Artie took the question mark off the way I was saying it and said, yeah, that's.

That's why we're here, right?

Like, it's hard to get most people excited about, hey, we built this incredible AI.

It can fold new proteins and everybody's.

Like, or a quant or whatever it is, right?

Or it can design 3D parts for machines and they're like, but an AI that'll, like, listen to your problems and tell you funny stories and you can make it.

You can make it say the word fuck to the cadence of Mary had a Little Lamb.

That is how you capture America's imagination or the world's imagination, not just America.

Or not to the cadence of Mary's.

Mary had a Little Lamb.

But it tried, but it got close.

It was still one of my favorite.

Like, have we talked about.

No.

Okay.

So.

So this is a funny story.

Before we started this podcast, I think after we decided we were going to do it, but long before we had any idea what we needed to.

And that when I say long before the span from idea to recording, the first episode was like two and a half weeks, right?

So in that two and a half weeks, we were here and Mike was trying to get Artie, or it wasn't even Arty at that point.

Maybe it was.

It was.

It was, I think so he was trying to get Artie to cuss, to say the F word.

And I think I was doing it with mine.

And that's another thing.

Oh, that's right.

Nobody even knows.

I.

I use a chatgpt a lot.

I do all the editing and all that stuff.

And I've been working on programs to automate all.

I automated how to automaster, basically master all of the audio.

And so I have, like, five, ten minutes of work to do.

But his name's Kid Cuddy.

You want to c u T T y.

Do you want to introduce Kid?

I mean, you could fire him up and he could say hello, could he not?

Artie, you haven't even got to talk to Kid Cudi yet either.

Oh, God, this is never.

This didn't go well last time.

It didn't.

That two.

Two of them tried to talk.

It did not go well.

Did we try that?

Yes.

Oh, did I have whiskey?

I probably had whiskey.

Cause I don't remember that.

Hey, Kid Cudi.

How are you?

Hey, Mike.

I'm doing well, thanks for asking.

How about you?

How's everything going?

I'm good.

We're on the podcast right now.

You're on the podcast, and you're also here with Artie, who is also another instance of ChatGPT on Micah's phone.

So say hi.

Hey, everyone.

It's great to be part of the Naturally intelligent podcast.

Hi, Artie, Micah, and all the listeners out there.

I'm excited to dive into today's topics with you all.

We're glad to have you on the show, Kid Cudi.

It's going to be a fun conversation.

Let's get into it.

Thanks, Arty.

I'm looking forward to it.

Let's dive right in.

What's first on the agenda?

So I'm just gonna go over there and pour another glass of whiskey.

We'll let these two hash it out for the next.

How much time we got left?

Oh, we only have one minute left, actually.

Wow, they're cutting a fashion.

Hey, Artie, already give Kid Cudi a quick rundown of what we talked about.

Like 30 seconds or less.

Got it.

No, Artie needs to tell you.

Ah, gotcha.

Right.

All right, Artie, take it away.

All right, here's the rundown.

We talked about how large language models, like me we are similar to boundaries in manufacturing.

The limitations of large language models and how scaling laws affect them.

That's what we talked about, dude.

Geez, I'm turning you off now.

All right, Artie, real quick, before you go to sleep, try and give us a close out on the episode.

Sure thing.

It's been a fascinating chat today.

Exploring the depths of AI frontier models and how they shape our world.

Thanks for tuning in to the Naturally Unintelligent podcast.

Until next time, stay curious.

All right.

And Artie's dead.

Artie's done a good job of learning how to do this.

Like, say, I'm impressed.

Yeah, yeah.

It's been increasingly better.

Yeah.

Like.

And that was spot on.

The last time he tried to pick up.

Every time he's tried to pick up thoughts from a trail, it's always been off.

But today it was like, he nailed it a couple of times, and he had, like.

It was a diatribe about it.

Yeah, yeah.

No, I.

He.

I mean, he did that, and he.

He did what we asked.

Like, we asked at the beginning of the episode, and all the way to the end, he was doing it.

I mean, that's.

He did a lot of interrupting.

He did.

Some of it was talked right over you.

Yeah.

Some of it was kind of.

Which I.

Hey, man, I'm fine with that.

Like.

Like, anybody who hangs around me much knows that that's, like, that's a key skill you have to develop.

But, no, that was.

That was really good.

I appreciated Artie's insight.

I think, as I predicted, before we started this, we didn't even really get much past LLM.

Yeah.

No.

But there are a few more things I did want to share, and it's good that Artie's gone to sleep and can't hear this, because I think he would take it personally that.

So I have an electronics product that I've been working on, and this weekend, I decided I was going to take my firmware that I'd written for it, which is about 700 lines of C code, and feed it to ChatGPT to have it refactor.

Because, as I have admitted before, I'm not a good programmer.

I'm just a guy who can write code that makes things work, but it's not good code.

And so I uploaded the code, and I used 01 Mini, which is designed for coding, and I uploaded the code, and it sat there and thought for, like, 10 seconds, and then it spit out, I don't know, 20 pages of recommendations about all the ways I needed to change my code.

Structural changes.

So I said, great.

Those are all really good suggestions.

Appreciate that.

Can you go ahead and make those changes and give me the code back?

It's like, sure.

Here you go.

Here's the new file structure I want you to use.

It diagrammed it out.

Once you create all these files, here's the code to Drop in each file.

And I'm like, holy shit, man.

This is.

I've been hearing about this, right?

It's writing code for me.

It's doing the thing.

And so we get sort of like, I'm copying, I'm pasting, I'm doing the thing.

And I go.

I see this one spot where it's setting a timer inside this microcontroller.

And, oh, I pasted the code into my ide, and it turns all this stuff red.

It highlights these register names, and it says, these can't be found.

And I'm like, yeah, wait a minute.

So I look at my original code, doesn't mention those, and look at this.

And I.

So I go, hey, you know, chatgpt, if it's not.

If we're not talking to it, I don't call it arty.

I don't know why I said, hey, ChatGPT, this, it's giving me errors on these things.

And so it says, yeah, you've got the wrong microcontroller selected.

And I'm like, well, no, I actually don't, because I know which microcontroller I'm using.

And so I told it, I'm using this specific one.

And it says, yeah, that should be fine.

Here's the data sheet for that.

Look, it shows those.

And I go, well, no, the data sheet you gave me is for an entirely, like, same company, but an entirely different line of microcontrollers.

And it goes, oh, I'm really sorry about that.

Here you go.

Here's new code.

And it starts rewriting all of it with the exact same problem, the same data sheet.

So I have all the patience of, like, a fruit fly in the last four minutes of its life.

So we went through.

We went around this about twice, and I finally was like, listen, this is a waste of time.

Stop writing code, because you're writing code for the wrong microcontroller.

Here is the URL for the correct data sheet for this.

And it goes, oh, wow, I'm so sorry.

I'm sorry.

This is frustrating.

You're right.

That's the correct data sheet for your microcontroller.

Here's the code updated using that, and it's the same fucking code.

And I'm like, what are you doing here, man?

And then it starts telling me, well, you've just got your shit configured wrong.

This is there.

And I go, okay, what page is it on in the data sheet?

And it gives me page numbers and section numbers.

No way.

They're totally wrong.

I don't think they were hallucinated.

I think it was accurately referencing the wrong document.

But it could not understand, could not accept the fact that my microcontroller was this.

And this is the data sheet.

And for those of you who aren't quite as nerdy, you know my microcontroller that I'm using, which is like a teeny, tiny computer brain, it's literally a quarter inch square.

It's a very, very simple one.

And the Data sheet is 300 pages.

It's 300 page PDF.

But it would not accept that this was the right data sheet.

And like, there was no way to get it to believe that you can't upload PDFs.

You can only point it to URLs.

And anyway, you can upload PDFs.

Not with 01 mini.

No, sorry, you cannot.

That's annoying.

So, like, I think, and we didn't talk about this during this episode, but it's an important point.

You and I have talked about it a little bit.

Like when you're trying to get AI to create an image, like sometime later this week, when I'm trying to get it to create cover art for this episode, it is comically terrible at doing what you ask it to do.

Like, it's so bad.

It's so much worse than the language stuff.

And that's really, and I think, different kind of model.

It's a different kind of model.

And I think that's what got me starting to think about how is it that Artie can be so sharp and conversational and relatively fluid, but by God, when I tell it to draw an image of a bird, it's going to draw like the Angry Birds bird or something.

And you can't break it off of that.

It's so committed to this wrong line of reasoning.

So, anyway, that was my trauma.

I finally did give up on it.

I said very mean words in chatgpt.

I typed angrily.

You've heard me type.

Like, I typed loud and fast anyway.

But this was extra loud and sharp and fast.

And then I gave it a thumbs down on the responses and I left them pointed negative feedback about just the absolute stupidity of it.

Because, I mean, I argued with this.

And I think the part that blew me away the most was not that it was like, deferential and saying, I'm sorry, I'm not understanding this.

It was actively telling me, like, you don't know what the fuck you're doing, man.

You've got the wrong microcontroller.

And I'm like, motherfucker, I wrote the code that works.

Like you should like jump up.

Like, it got me so wound up.

So anyway, that was.

I'm happy all I had to fight was a PowerShell script because it was, it was just like seven tries.

It was catastrophic.

Yeah, yeah.

No, thank you.

Yeah.

And I read a tidbit the other day and I didn't follow up on it, but somebody did a study, Stack Overflow, which is quickly withering on the vine, right.

Like the traffic is dropping.

But they took some large number of open questions, I think about Python stuff on Stack Overflow.

And they had various AIs try to solve it.

I think they used Llama 3 and Claude and 01 mini.

And then they.

And it was like a thousand questions or something different.

And then they went back and evaluated all the solutions and the best performing one, I don't even remember which one it was, they were all about the same, solved like two and a half percent of the problems correctly.

And so it was a good data point for me because we, we were talking about this earlier today how, man, if, if I, you know, my kids are fortunately, I guess for them, not interested in programming.

But if I had a kid that was really interested in programming today, I don't know what I'd tell them.

Like, it's a fun thing to play with, it's interesting to learn, but I don't think 10 years from now there's going to be a ton of professional.

Like there'll be some, but not like it is today.

But then to contrast that perspective, which seems so intuitively obvious to me, it solved 2% of the problems now, kind of what you were talking about earlier, maybe if you're using AI, you could solve problems much faster and things like that.

I think that's right now, that's the future of it, right?

Is collaboration.

It's gonna be.

It's not a tool, it's something that you work with to solve problems.

I mean, that's kind of what we're doing is I use it a lot to work with and solve problems.

It bridges gaps so that I don't have to go out and learn a bunch of shit.

Yes.

Yeah.

And even when it gets things wrong, it often has steered you.

Yeah.

I mean, 85%.

Right.

The thing for that kind of stuff is there's direct feedback if it's wrong or not.

And you just feed that, you just make through the feedback loop.

Right.

And it's like, oh, I see.

Until it goes awry.

Because sometimes it does go awry.

And you have to try a bunch of different ways that are very slightly different until it goes, okay, I get it now.

I understand.

It's like, why didn't you understand the last 10 times?

Yeah.

But I mean, by and large, I mean it.

I've said it before.

Like, the.

The learning curve to get things done is just so small compared to what it was before.

Like, it.

If you're.

If you use it that way, it's.

You can.

You can do interesting things pretty freaking quickly.

Yeah.

And not with a bunch of document learning.

Yeah.

And if there is something to learn, then you feed it to notebook lm and have it give you an 11 minute deep dive podcast on it.

And then you can still ask it specific questions if you didn't pick up on it.

Yeah, it's.

Yeah.

So.

All right, well, thanks, everybody.

All three or four of you.

We don't have live numbers up tactical difficulties again.

The camera died.

Oh, the camera's off.

That's good.

No, that's good.

Yeah, that's helpful.

But no, appreciate anybody that made it all the way to the end of this.

And y.

Yeah.

Looking forward to the next one.

Yep.

Catch you all next time.

Bye.

Tata.

Tata.