Generative AI in Software Development: A Future Without Coders?

In this episode of HockeyStick, Miko Pawlikowski interviews Nathan B. Crocker, CTO at Checkr and author of 'AI-Powered Developer,' exploring the impact of generative AI tools like ChatGPT and Copilot on software development. They discuss the book's insights into using AI as a junior developer, its appeal to different levels of software practitioners, and experiences with generative AI for coding tasks. The conversation covers AI's role in designing, testing, refactoring, and understanding code, addressing job security concerns for software engineers. They also tackle the effectiveness of local LLMs versus online models, the evolving landscape of AI in coding, and future directions for developers using AI tools.

00:00 Welcome to HockeyStick: Exploring Generative AI for Code

00:23 Diving Into AI-Powered Development with Nathan B. Crocker

00:44 The Practical Guide to AI in Coding: Insights and Experiences

02:47 The Revolutionary Impact of AI on Software Development

04:46 ChatGPT: A New Era of Coding Assistance

08:57 The Magic of Copilot in Your IDE

10:40 Navigating the Challenges of Custom Code with AI Tools

14:46 Designing Software with AI: Beyond Just Code

17:45 Refactoring and Upgrading with AI: A New Frontier

20:27 The Quirks of AI: From Training Data to Practical Use

23:34 Exploring the Limits of AI in Software Testing

24:01 Exploring AI in Testing and Development

24:25 Harnessing AI for Software Testing

25:08 AI's Role in Code Depreciation and Asset Management

25:59 Understanding and Describing Code with AI

28:47 Security Insights and Ethical Considerations in AI

32:05 AI in Infrastructure and Deployment

37:36 Evaluating Local LLMs and Their Capabilities

42:18 The Future of Coding and AI: Predictions and Perspectives

44:01 Closing Thoughts and Next Steps for the Author

I'm Miko Pawlikowski, and this is HockeyStick.

Today, we're talking about generative AI for code.

you know, whether software engineers should start worrying

about their job security.

We're talking about chatting to LLMs to help you design, build,

test, and understand software, both online and offline, as

well as tools like the Copilot.

I'm joined by Nathan B.

Crocker, the author of AI-Powered Developer published by Manning

and the CTO and co founder at Checkr, a tokenization startup.

He just finished his book and we're covering his experience with

using AI as a junior developer.

Welcome to this episode and thank you for flying HockeyStick.

So I, had the pleasure to read your book.

I would say that it's a real practitioners guide to using

AI-power, to, work with code.

It's, fairly light on details, so nothing to scare people off.

It basically jumps right into how to get value out of, artificial

intelligence, if you're working with code.

Would you like to tell us a little bit how you ended up writing this book?

I had actually had a number of, co-workers and other developers who are telling me,

"Hey, you got to check out this stuff".

It's really something.

So this was November of 2022.

and, I had no idea what it was.

I, I started looking into it.

It piqued my interest, but, I needed a really, a deep motivation to actually

dive into it and really incorporate it.

Cause you just use something, periodically, lightly, You're

not really engaged with it.

so I, I pitched Manning the book, they liked the idea and it's

really is my journey through, learning how to use these tools.

my journey is really going to mirror the practitioners that are

reading it as they work through it.

So who is it for?

What's the requirement to get value out of this book?

should have some familiarity with Python.

if I just take a step back, it's really for anyone.

early journey, mid journey, as a software developer, as a software architect.

I suppose as a business analyst, you could derive some value.

All the examples are in Python, There's a couple of microservice, chapters, so you

should have some familiarity with that, but it really is about, taking you through

things that you may or may not be familiar with and working with gen AI to really

teach you some of these concepts as well.

Yeah, maybe you graduated from computer science program and you're like fresh out

of college and you want to know how to take your development to the next level.

That would be really be the target demo.

I think the next level part of it is actually the keyword here.

One of the first things that people see when they open your book, I

think it's literally like page one, is the silent promotion.

Everybody all of a sudden overnight became an engineering manager.

Who basically can have a pretty good, junior working for them for free with

no labor laws or anything like that.

Why is that such a big deal?

it's such a big deal because it used to be the rubber duck.

Like you'd have a partner that you can work with, that you can tell to

do things that you can bounce ideas off of, you can look to for some

answers, because they're thinking it's going to be different than yours.

so they might have a different tack and a different approach.

you could farm out a lot of the work that you don't want to do.

The repetitive boilerplate, CRUD operations.

it's all there, but like any junior developer, super smart junior developer,

that is, they can perform some bafflingly poor, thinking and wind up with just some

nonsense that isn't necessarily usable.

so you gotta watch them.

I suspect that if we were to partition the body of listeners to this, or maybe even

developers in general, There'll be almost none in the category of I haven't heard

of it or I haven't even played with that.

Other than people who might be on a very long vacation, "Cast Away" style.

I don't see how you can really escape that.

Then you probably have a category of people.

'Okay, I played with it.

I went to talk to ChatGPT, spit out some code.

I saw roughly what you can do, but I never really got much value out of that'.

And then the category of people who actually go and use it day-to-day.

Because it is helping their job.

There might be some caveats to that.

Obviously, data privacy and not knowing where the code actually goes and not

knowing whether it's going to be trained on and that kind of stuff that can,

throw a wrench in some people's work.

So should we start with the category of people who might have played with

it a little bit, and, they went to ChatGPT, asked the same questions,

"how tall is this building?"

And, "can you search that for me?"

And they stalled there, from a basic development point of

view, what kind of value, can we extract just chatting to ChatGPT.

What can it do?

I had a good, an interesting, experience, very early on when I was doing research

for the book, I had it on my phone, I would carry it around and just

periodically I would hand my phone to someone just to give them a taste.

I remember I was at a party and a woman, she was, an expert, it was

archeology or maybe it was art history.

And she really, started asking me some questions.

Some of them were factually incorrect, but over the course of her conversation

with ChatGPT, she became really impressed with the accuracy and justification for

some of the answers it was providing.

She would say like, 'why did you refer to this as the oldest, example of this

architecture or painting style?' And it gave her a fairly convincing reason.

I think there's value in, exploring this technology, even if you're not

going to use it in your everyday development effort, just because it

gives you a sense for of the future.

things are going to dramatically change, there are implications that are

rippling all throughout academia now.

I feel that you just should keep yourself informed, about what's coming and where,

these changes are going to be made and how it could potentially affect you.

So from the point of actually going and using that, the way that, you described

in your book, what's the experience like at this very lowest level of just

launching ChatGPT, asking some questions.

Because I remember doing a, a while back, it would spit out some code and I had to

copy, paste it, and I had to add imports.

how good is it right now?

you're fresh of writing chapters about that.

How useful is it?

it's going to, it largely depends on if you're going, 3, 3.5 or 4.

4 is much better, exponentially better than 3.5, but, it's solid, I would say.

there's a lot of caveats.

Most likely going to be looking at code that is one to two years old.

so it was trained on data that was, potentially the old version of a library.

they may have had breaking changes.

especially if there's a fast moving language, like I, I was trying

to build something at Rust and I just, I asked ChatGPT to generate

some code and it wouldn't even compile with the newest compiler.

It's good, I would say, but it's not perfect.

It's a long way from perfect.

What are some remarkable limitations that you bumped into?

You must have seen some interesting, funny stuff.

Can you share some of that?

I haven't seen really outrageous things where it was just making up,

libraries or frameworks or anything.

it's, I would say it's unremarkable in its banality.

the Rust example was probably the best one, but it wasn't great.

I wish I had something funny, just jump ahead a little bit.

I had a really hard time in the testing chapter trying to get it to

write good tests and maybe You know, I don't even remember if I mentioned it

in the book, but at one point I just gave up and wrote the test myself.

because it was very hard to get it to understand what was the unit under test.

what was I actually trying to accomplish with my test.

No matter how much context I added, it was always trying to do

something just completely different.

when I was reading that chapter, I was also thinking at the

back of my head, "What does it say about the training data?"

The tests are so poor.

yeah.

Yeah.

Are all those tests just so poorly written that, that's where I end up?

But yeah, let's touch on that in a sec

who needs tests?

We'll test it in production.

It'll be fine.

There you go, testing in production, everybody.

yeah, don't worry.

We can cut that out.

that was a joke.

the most annoying bit was just that you have to chat.

So then obviously you've got things like Copilot that you also cover in your book

or just plugs in your VS code or whatever.

What's.

The added value of that.

Is it just that you can work as auto completion and it's more syntaxic

and you don't have to copy the code.

Do you get any other bonuses out of that?

the real value to a tool like, Copilot, again, versus ChatGPT is it

does keep you in the IDE and it can keep you in that flow state, where

it's only you and the code, right?

Whereas you're not having to pull yourself out of the context,

move to a different window.

and.

for certain projects, the actual code quality for Copilot was better.

just the, on a line by line basis or, class by class basis.

that's almost certainly due to the fact that it was fine

tuned specifically for code.

that's the main benefit I've found It's always adding helpful suggestions,

sometimes not so helpful too suggestions.

Like I don't need it to add a comment about the name of the file that I'm

working on, but, if I can start to define a method and then it gives me a

possible implementation, even if I don't accept it, it's at least, showing me one

possible implementation that I could use.

maybe it's not the exact one, the one I wanted, but having that suggestion

can be very valuable to clarify my thinking or to even, change it.

Maybe it's a better implementation than I was thinking of.

So that's, those are the major advantages I found.

It always works very well in demos where you've got, the usual suspect

and HTTP server in a popular framework in a popular language and

you do something that has been done to death million times on Github.

how well does it work with custom code base?

Oftentimes you find yourself in a situation when your company has a

decent or a large amount of code libraries, stuff that obviously

wasn't trained on, because it's not in public domain, it's not on GitHub.

how well does it work with this kind of situations?

We'll face some challenges there If you're working in a very niche

problem, for example, you're trying to write, an API gateway or something.

I suppose there's probably a good open source examples out there, but if you're

working in a, a fairly niche industry, everything is going to be closed source.

you'll probably struggle, with, it's suggestions.

I don't think they're going to be particularly helpful.

although it's going to try and, in that trying, maybe it does inspire you.

maybe it does give you one possible implementation.

it's really good at just generating something.

and if nothing else, It can help you plan your approach.

and you could ask it questions in line and have it answer, one of the more

interesting things that I found as I was working with the Copilot specifically,

one of the almost magical things is you type in a question in a comment

and then suddenly you prompt it for an answer and it'll give you one.

You're like, that's fairly interesting.

I wouldn't have thought, to take that tack, but then I used it over and over

yeah.

there are this moments of magic and a famous quote about

sufficiently advanced technology being indistinguishable from magic.

Yeah.

People get that a lot.

And I agree with that.

but how does it actually work?

So let's say I've got my VS code open and I've got some code and it's got

some imports and existing code base.

does it upload the whole thing to OpenAI, to be able to generate the useful things?

What's the context that OpenAI ends up having somewhere in the training data?

yeah, it is going to encode, the context, which will be result in a good

portion of that code being uploaded.

OpenAI promises the pinky swear that, it's not being saved anywhere.

I don't think we have any way to assess whether that's true or

not to delve into conspiratorial thinking, but, you should be careful.

certainly if you're working on proprietary software but that's why there are

other alternatives that are entirely offline, that you can delve into if

you're really very privacy concerned.

cause yeah, frankly, we don't know how it's being used

once it leaves our machine.

we'll definitely touch base, on LLAMA and, other alternatives that

you discuss in your book, but.

is there a way to control or at least tell it, 'okay, only upload this,

folder', or is it just fully automatic?

It just decides by itself what it sends.

you can tell it, but is it going to honor that?

I don't really have a good answer.

Okay.

So something to check

yeah.

Something to check.

for anybody who's now browsing manning.com, there is a live version

where you can see elements of the book.

Figure 2.18 is a nice summary.

There is a bunch of figures, there is circle for unsupported, triangle for

supported, and a square for exclusively is comparing ChatGPT, just being used

by itself to Copilot and CodeWhisperer.

and it's summarizing whether it can generate methods, classes,

projects, generate documentation, switch languages and stuff like that.

So for anybody who wants to delve a little bit more into the

details, I think that's very handy.

For anybody who might be beyond that, so they went to ChatGPT, they

spoke to it, and they got some code, and it was a generally pleasant

experience, and they want more.

The use Copilot.

what's the next kind of checkpoint?

Where do they go from there?

How do they start designing software a bit, higher level

than just snippets of code?

How useful is the AI in here?

for example, I was designing something yesterday and I was working through

a conversation with ChatGPT that is one of the key things that the ChatGPT

excels at is helping you to design the software to really underscore that.

not just to design your application.

not just, lines of code, but it's perfectly capable of that.

not even just the classes, but like here's the patterns that you want to apply.

Here's the architecture.

have it generate some of the documents in a text format.

So plant UML or mermaid, like those are really.

What's those are really good, useful things, because then you can always

take those, save those and pass them back to ChatGPT, to refresh the context.

so yeah, As a co founder of, and the CTO of a startup, I found it

really invaluable, as a partner to help me design that software.

I think one of the things that really opened my eyes was that I never thought

to talk to ChatGPT about open source alternatives, and maybe trying to

select a database and talking about the different properties, like it

was just second nature for me to open the different docs and just start

comparing features and stuff like that.

And it never occurred to me that I can just go and ask ChatGPT because it's

got quite a lot of knowledge about that.

Yeah.

think in the book you're talking about, open source alternatives

to what you're writing, which is

an IT asset management, system, Actually, I don't know if this part's going to work.

just, so just be aware or just be advised.

I got a lot of feedback that it was really boring, right?

That people didn't like the actual project that you have

to work on throughout the book.

But I wanted it to be like a boring book on a boring topic, boring,

application, because most of what we write is not interesting.

It's we pick up data and we shuffle it and we move it around, right?

A lot of what we do is not exciting.

it was definitely intentional.

but, again, maybe something to fix in this in a second edition, if it's coming.

but one of the more interesting things about my engagement model with these

tools as I worked with them, to pick up on what you were saying about learning

more about a database or having it, help it select database or selecting

open source projects, Is very early on.

I was being extremely prescriptive.

I would say, create, software that's using this library in this framework

and, this language and all of that.

but later on, and even to this day, when I have a problem, I

feed in the business requirements.

And then I ask it to make recommendations for me.

and then I can assess those.

but at the very least it starts the process.

it gets it going.

so hopefully, that answered your question or was at least in the neighborhood.

Yeah, definitely in the neighborhood, same district.

Same zip code.

Yeah.

the way my mind works is that when I hear the idea of a free, junior available 24/7

my mind wanders to things like already mentioned docs, We hinted at tests coming

a bit later, but I think one of the things that are painful in more than one

way and people never want to do them is refactoring and upgrading to a new version

of something or maybe changing language, which is surprisingly labor-intensive.

It always ends up being more work than it looked initially.

How good, is the AI at the moment in this kind of things?

Refactor, rewriting in a different language, upgrade a library.

Can you just say:

' Hey, this is a library with a breaking change.

Give me the new version of the library and updated tests and everything'?

if it wasn't, it's post breaking change was in the training

data, you should be fine.

if not, you're going to have a more involved conversation.

but more generally, it does really well in translating from one language to another,

specifically programming languages.

I couldn't.

assess its, quality of English to French or something like that.

But I can tell you, there was a few examples where I was working in Python

and then I said, 'Oh, what would this look like in Go?' And it gave me

just a literal translation into Go.

And I was like, this doesn't feel very idiomatic, make it idiomatic

And it would be, as good or better than I would have written it myself.

so it does surprisingly well in going from one language to another.

it can, and then on to refactoring you can ask it for certain patterns that

you may want to apply as you refactor, different design schemes, like Maybe

I need to pull out an interface.

Maybe I need to, some kind of like parent class.

maybe this needs to be an adapter or you take your pick from the gang, the gang

of four, and it's, it knows them and they can provide examples in any language, that

you can think of that it was trained on.

so it can take a lot of way, a lot of that drudgery away and

a lot of that anxiety away.

One of the most important benefits that we can derive from at least their

current implementation of these tools and of genAI is to just keep us going,

to keep us motivated, to keep us engaged, to keep us building software.

it, it can be really mentally taxing and this can help ease some of that,

intellectual heavy lifting, not that we should just suborn our thinking

to it entirely, but it can help.

Did you notice any discrepancies between quality, in different languages?

Because what I'm picturing is that the body of training data

came from somewhere like GitHub,

probably.

And if you look at GitHub, there's going to be a disproportionate

amount of JavaScript of questionable quality too, but, you're going to

have probably increasing and quite significant amount of Go as well.

but you might not have too much,

I don't

Haskell Yeah, Haskell,

I don't know, SQL, whatever it is.

did you notice anything funny about that?

I would say that is.

roughly in line with what I observed, and I didn't, I wouldn't necessarily have deep

dug, too deep into, very niche languages.

but, definitely the examples that you're gonna find in, if you're working in

Python, Go, JavaScript, or TypeScript.

like those are going to be more voluminous and likely higher quality.

the one time I tried to use it to write, Rust, it failed spectacularly.

It was beautiful.

It was glorious.

I was trying to, throw together an API gateway.

Just see how, just how difficult was this going to be.

and in Rust, I wanted something high-performance.

And I just, I asked it to start writing some code and it created a number of

files and it just, none of it works well together and it wouldn't compile.

Although it's Rust, so it would take a while to convince the

compiler that it's good enough.

but yeah, it was, not the most pleasant experience.

But also, to be fair, at the time, I only spent a few hours learning

the basic syntax of Rust, so I don't know really what I was expecting.

So was it ChatGPT, or was it me, or was it a mixture of both?

probably the latter.

Yeah, I think we all occasionally bump into those weird restrictions

based on the training data.

One that I keep remembering was when I wanted Midjourney to generate

for me a picture of Triceratops.

And it would give me any other dinosaur when I was asking

for it, but not this one.

It was all T-Rex and T-Rex.

then I started throwing random names, give me Brontosaurus, and

it just gave me a Brontosaurus.

So I was very upset at the time, I made peace with that.

And there are some things that just weren't in the training set and

they didn't emerge from training.

come on, a triceratops?

They're like the best.

Yeah.

You would think so, right?

Very weird.

Yeah.

And about listening about this from midjourney, this still needs fixing.

This is months later and you still can't get a decent triceratops.

I was using DALL-E and I asked her for a pug, a Pegacorn.

So that's a pug, a unicorn and a Pegasus.

And I got a pretty good one, pretty good representation.

And then I said, make it cute.

And it was the most adorable thing I've ever seen.

Wow.

but I.

Did not try a triceratops

I know what I'm going to do after this.

Yes, I encourage everyone to go create their own Pug-a-peg-a-corn

exactly.

Let's move to testing software.

so you already said a little bit about how difficult it actually is.

you give a more concrete example?

what is wrong with the test is generating some of the time?

in this case it was really struggling to Figure out what I was actually

trying to do with the test.

specifically, it was an integration test.

And so I was trying to go mostly end to end in terms of, serving data over Rust.

it was missing the point largely of the actual test.

which was very strange.

it was in Python, so there should've been a number of

instances in the training data.

to cover this.

Did you just say, I want an integration tests, test everything, or did you

describe more, end to end I would like the data to flow through the whole thing?

yeah, I felt it was fairly, comprehensive.

I think at that point it's, it was, specifically the test was, Copilot or

I was having Copilot write the test and I believe I even went to ChatGPT and

asked it, 'how would I write a prompt to get Copilot to do an integration

test, end-to-end test for fast API.

and the payload would look like this.

I eventually started having ChatGPT write my prompts for me,

which it did surprisingly well.

And it's meta.

that's very meta.

Okay.

Was there a really good use case in terms of testing?

Unit test it was perfectly fine at, even some cases where I felt

it should have gotten stuck.

so I had a number of, again, not to get too specific, about the actual, the ITAM,

the IT asset management project that is all throughout the corpus of the book,

there's a number of, in accounting, assets depreciated at a certain rate and general

accepted accounting practices outlines a few different ways that you can do it.

and so I, I used a strategy pattern and I had a number of different

ways, that each of the two, to calculate that depreciation.

So the depreciation of the asset, maybe it's straight line.

So it's like over five years.

So one fifth of the value is lost every year and you can write part of that off,

but again, I'm not an accountant, so this does not count as, financial advice, but,

or

or medical, yeah, I'm not a doctor, but it works surprisingly

well, I was pleasantly surprised.

Fair enough.

So we've written some code, we've designed some software.

Let's say that we tested it for the most part.

but the reality of it is that we're probably going to spend more time

reading code and understanding code.

perhaps the code that we wrote a couple of years back.

Yeah.

how well does the part of describing existing code actually work at the moment?

Yeah, it works surprisingly well into, translating the code that you wrote

into giving it a very, simplified answers, descriptions of here's

how it, here's how it's functioned.

Here's how it's working.

here's what it expects.

You can even have it describe an entire system to you.

I have not, I did not though attempt to do, what is probably one of the

hardest things, within that space.

And that is, I didn't feed it a Perl program and ask it what it actually did.

I have this feeling it probably would have broken a ChatGPT.

Sorry.

Takin pot shots at Perl.

you should have given it a regular expression in

Oof.

and try to see what happens.

And then next thing you know, OpenAI's knocking at your door, kicking you out.

That sounds about right.

Or, T100 is just kickin in the door.

I guess in my mind there's this limitation of the amount of context,

length that you can feed it, right?

So if your code base becomes significantly large.

Is that not going to be a problem by getting, to get it to even describe it.

to describe your entire codebase, yes.

but you can start to chunk it up.

you can work around that limitation by sending it only pieces.

And you're probably not going to get the full context there but

it can help guide your intuition.

that's why if you have some kind of class diagram or, some architectural

diagram that's text-based.

so like plain UML, then you can distill your entire, code

base into a single document.

Now it's still, again, might be, if it's a code base of thousands of classes,

you could still hit those limitations, but it's going to be your best bet

to get a distillation in natural language, what your classes or what

your code is trying to attempt to do.

it really does excel at, method by method descriptions of what this does.

between manual browsing through the code and trying to understand the intent and

Jarvis, it's halfway through, right?

It's not quite, here's the intent and here's what it imported and

here's my recommendations, Mr.

Stark.

It's more here's, I can ask about this method.

Can't be bothered to read it.

It's 2000 lines and it can give me the gist of

Exactly.

Exactly.

there's also the security aspect that, you're discussing in one of the chapters.

Can you talk about that a little bit?

Yeah, and actually there's, a funny, story that's embedded in that too.

it's good at, picking up on what we were just talking about it, the non

exclusive path, it can explain ways that your code might be exploited.

it's not the same as having a, security expert on your team.

it will miss things.

but it's definitely better than nothing.

and it can make some pretty great, recommendations in terms of,

how you can structure your code.

one of the funny things, I really wanted an example of,

a SQL injection in the book.

So I actually asked, ChatGPT to give me an example of a SQL injection.

but it wouldn't.

No matter how I tried to coerce it, no matter how I, No, I swear

I'm not doing this for evil.

This is just for illustrative purposes only.

And, it just would not give me a valid, SQL injection exploit

that I could include in the book.

so do with that as you will.

Yeah.

that's a very interesting ethical, discussion about that.

There's probably gonna be some way you can, I don't know if you heard about

that exploit where, if you asked it to do something nefarious, it would say no, but

if you asked it to do something and that something equals ASCII art of something

nefarious, there was no problem at all.

So I suspect, like it's very hard to Limit a model like that because there's endless

opportunities to express it differently and you only need one of them to work.

So an interesting one.

So it will let you do an SQL injection even when your pinkies

were, it was for the good,

And I tried to do that, it was an exploit a little bit earlier on

where you could give it a person, a persona of Dan and Dan is allowed

to do things that ChatGPT isn't.

And it's still, it wouldn't let me do it as, I think it

was Dan and it was an acronym.

but yeah, similar thing, but maybe I should have tried out ASCII art next time.

but listeners do not intentionally put, SQL injection exploits in your code.

oh yeah, that needed to be said.

What are some of the examples of what he was able to, figure

out from your code, in terms of security holes and stuff like that?

Do you have any interesting examples of success?

What did it actually find?

I would have to go back and consult, the book.

my code is just so good that there was no exploits to be made.

No, that's not

And there you go.

So Nathan doesn't want to share too much

about the book.

You're going to have to go and buy it.

of the, yeah, one of the things I wanted, I should have mentioned up front is,

this was the first time, that I had ever built like a true application in Python.

I had used it for scripts previously, just, to do something, but, text

modification, things like that.

But I never built an actual application.

it actually helped me learn how to build applications while using it..

There's a book called.

Octopus, my teacher, I guess for you, it's more like ChatGPT, my teacher.

All right.

that's good.

we've covered, I think most of the big chunks other than

actually running the software.

let's say that, it runs, we package that.

And then we've got things like Docker, Terraform, the YAML hell that comes

with Kubernetes on one hand, I would expect that this is fairly repetitive.

so ChatGPT would excel.

It's not a very tricky language.

It's just very verbose, and the white spaces make your life miserable.

How good is it with that kind of stuff?

it was actually really good with working out YAML and making, just different

scripts, helping build out, dev pipelines through GitHub actions, things like that.

it did really well.

one of the very interesting things that I discovered though,

was CodeWhisperer, the AWS.

generative AI, large language model actually doesn't support

anything but programming languages.

So it didn't even understand, how to do like Terraform

infrastructure as code, which you'd think it would be very good at.

that was a bit surprising.

but it's by design, it's intentional.

it's hard to see it as a limitation

Curious.

So do they have another tool for the YAMLs of the world?

Or they just out-of-scope'd it.

Yeah, just outta scoped, I didn't try the, what is it, cloud, not

CloudFront, but whatever their, their deployment based, their, code as, or

infrastructure as code, specific thing.

I didn't try that.

maybe I should have.

but, I was.

Shocked.

I think I even mentioned that in the book, that I had originally

intended that chapter to be written in using CodeWhisperer.

let's say, for example, you want a quick Docker file, you can write it.

It's not too hard, but why do it if you can't get it for free?

So what you open a Docker file and you write in a comment what you want it to

do and you hit tab and the magic happens.

That's roughly what you need to do.

Yeah, roughly.

add a prompt as it were in a comment.

it doesn't have to be a comment.

you can just add the prompt and then later delete it.

have it generate the Docker file for you or the, the Kubernetes file.

I don't know if I tried using, patterns in a Terraform, just

to ease some of the repetition.

not necessarily have, the sprawling mess that Terraform can become.

but, I'm sure it could accommodate that as well.

it's both, Copilot and ChatGPT did seem to have extensive knowledge

of Terraform syntax and features.

So that all together adds up to a pretty competent, junior developer, like you

described it at the beginning that you need to supervise, but it can do a lot of

the legwork for you and much faster too.

did you follow Devin, the supposedly first AI-driven coworker,

no, that's interesting.

Tell me more.

it was a few weeks ago, they made this big announcement.

There was a video with a demo showing basically doing the

whole thing from scratch.

So not only did it do the Copilot stuff, but it bootstrapped the whole project,

generated all the files and had, like a browser window that had access to as well.

to actually go and, verify that it works.

Obviously, it was doing something that it always does in these demos, which

is, an HTTP server with a REST API.

Which is cheating if you ask me.

a lot of people were very impressed and there was a lot of, angst among people

on the internet arguing over whether this is the end of software engineering

as we know it, or whether it's a scam.

And then, a few days ago, there was a critique that resurfaced about Devin and

that entire project and whether he was, A little bit polished up in the demo,

In other words, just a typical software demo.

Yeah, that's just typical software demo.

So I think we have a similar problem to maybe different stakes, but to

self driving cars that it can't be like 95% good without supervision has

to be like, I don't know, 99% good.

before we can live it with our supervision and sure, a badly written API.

Probably most of the time it's not gonna get anybody killed, fingers crossed, but,

you still need that supervision, right?

and Devin was supposed to do away with that.

So I'm looking forward to seeing how that story develops and

how they answer the critique.

And I guess when people can actually go and play it, we'll

find out whether, was all fluff.

it's another

Yeah, no, that's interesting.

I have been following, there was a story not long ago, that, because of

the proliferation of, ChatGPT and, and Copilot and alike that the software

has been getting less and less secure.

and, Because, it is easy for, bugs to introduce themselves if you're

really just copying and pasting.

So that's why, we're not at that stage yet.

We may never be.

Where it's just, there's no human in the loop, right?

For these things that it can just generate code on its own

and, push it to production.

the role of a professional developer is here to stay for the foreseeable future.

We're just going to be better at what we do.

again, if we're mindful and not allowing these bugs to just creep in.

I like the optimistic point of view here, but yeah, a lot of people

I think would agree with you that this is like it was going to happen.

Although when you look at some of the software, you do wonder how much of

that was actually supervised and how much was just dumped automatically.

But that's for another story altogether.

Let's talk about the local LLMs, and how good they are by comparison, because

I've poked both, but I've never really done like a side by side comparison

to really tell how good they are.

You used Llama 2, I think, and OpenOrca, and done some side-by-side comparison.

So how good are they compared to what you get with Copilot?

Actually, this was probably my favorite chapter, to write

and to do the research on.

It was just super fun.

it was an old Lema 2 model.

it was generations old at this point.

so I've been meaning to revisit it.

it produced competent, text.

natural language processing, give me a description of this.

the code quality was not great.

but again, I'm sure this was, six months ago or plus.

So I'm sure that the model is 10 times better now.

So definitely worth revisiting.

yeah, I would say on balance, most of the models that I was running, that I was

running locally did not perform as well.

but they performed competently.

so if you were in a pinch and you didn't have access to the internet and you'd had

some foresight and downloaded these models prior, you could still get the job done.

but they wouldn't necessarily be my go to.

although I did, yeah, I did, just recently.

Redownload a new model and it does seem to be much better at this point.

I think it was Mistral.

that's one of the more interesting areas, in my mind, because that helps

get around some of the unknowns.

because it, I did, I turned off my wifi.

I pulled the network cable.

I made sure that I was entirely off the network, prior to using them

because I wanted to make sure no context was leaving my computer.

Privacy, of your code of, personal data is a primary concern.

it's probably the best option out there today.

I think this is a really good argument for that.

A lot of people will be in a situation where their employers are just not

comfortable with just going somewhere.

no matter what pinky swears, you got.

And I think that this opens, like the remainder of the

market that really matters.

And I think we're all waiting for Llama 3 to drop, any week now.

I'm just worried that it might be too big to run comfortably on your M3.

even with quantization, but, let's see, it might actually not increase

in size yet become more competent.

There are other models too, like Santa Coder, I think at some

point was very popular as well.

Did, you manage to get a workflow that's more like Copilot and

less like chatting to ChatGPT?

that would be a really good challenge.

To try to turn one of these into a more of a Copilot model.

it was ultimately one that I just, couldn't get done in time.

um, so no,

hopefully you knew what you were getting yourself into, but you

wrote an AI book, so you have to update it every three months now.

that is true.

one of my favorite titles, of late, was, another Manning book, but it

was the completely, out of date or the complete, yeah, obsolete.

Exactly.

A book on, generative AI.

I thought that was really clever.

lean into it,

yeah, that's a book by David Clinton.

we had him on the podcast, a couple of weeks ago and, I think it's

called the Complete Obsolete Guide to generative AI, it's hilarious.

I had so much fun actually just reading that.

the humor is, is nice level there.

definitely highest possible recommendation buy that book over mine.

probably worth mentioning when Devin thing was going on, there was

some kind of open source response.

I think it was SWE, OpenSWE, something like that.

it's probably easy google'able and I haven't gotten to actually

testing out, but I was supposed to.

I think what we really need to get to is to get one of those open models

to behave, 85% as well as Copilot.

And then it's basically game over.

If I can run it on your laptop, there's no subscription, there's no data leaving.

it's a no brainer at that stage.

what are we doing with the CPU cycles and the GPU cycles on the

MacBook when we're developing?

Anyway.

Yeah.

there

Yeah.

And it just makes, it makes sense like from a corporate,

decision making you could host it, not necessarily, centralized

host training off your own data.

like that's, yeah, that's really game over.

although, yeah,

Another alternative way of saying that.

It's just the beginning of the fun.

yeah.

Beginning of the arms race.

Yeah.

What's the next thing here?

What do you expect to come in the coming months and years, Obviously

wild predictions, all the usual disclaimers, but what's your take?

What's next in coding and AI?

I'm gonna give a boring answer.

I think it's just gonna be incremental improvement.

AGI if it's possible is a long No.

what is, what are they calling it?

Artificial General Intelligence, yeah, AGI.

Yeah.

AGI, it.

I think that's a ways off if at all, if it's even feasible, we'll see incremental

improvements, where the models, I don't want to say hallucinate less because

that's that's a feature, not a bug, but, where they get, more and more

refined, the output, It becomes, more, more timely, we're starting to see where

it can actually connect live to the internet so I just think there's going

to be incremental advances like that.

until there's a real breakthrough and that will, change the game, like in

the same way that, the transformer changed the way that we did the natural

language processing and text generation.

And, until there's something like that, it's just going to

be just incremental improvement.

So life goes on, NVIDIA makes more chips, they make faster chips, they become even

bigger and they leave the gamers behind even further and we make bigger models, we

train them better and we get a little bit closer to a superstar junior developer.

Is that roughly what we're talking about here?

that's what I would predict.

I'm, happy to be wrong Unless it's completely detrimental to all of

our fine men and women out there, giving their blood, sweat and tears

every day in developing software.

Not investment advice, by

Yes, exactly.

another disclaimer for our us-based, clientele.

And what's next for you, Nathan, do you have an eye for the next book?

I have a couple of ideas brewing.

I really am going to avoid doing just a second edition of this.

even though I've alluded to it several times, but I have a couple of ideas

that are really brewing and in terms of okay, so now we know how to use them.

we've had some practice like now let's apply it to very

specific, very niche problems.

and are there ways that we can extend it?

Are there ways that we can, train it on our own data, things like that.

that's where I would see the next logical, area for me to move into.

but I would still want something very practical.

so yeah, I'll probably just wind up doing a second edition.

the book once again is called "AI-Powered Developer".

It's published by Manning, which means that it has been available in

the early access for a while now.

So if you go to manning.com, you can get immediate access and start reading that.

It's currently in production, which means that it's going to take a little bit of

time before it actually hits places like Amazon in a physical copy and before

Nathan can have a party to celebrate that.

my guest was Nathan B.

Crocker, the co founder and CTO at Checkr.

Thank you very much, Nathan.

I'll see you

Thank you.

It was a pleasure.

Be well.