An AI can produce a mathematical proof. Understanding what that proof makes possible is another kind of work. And when the same systems become harder to control, how do we decide whether to keep accelerating?

Justin Harnish and Nick Baguley return from their summer break to a conversation about breakthroughs, brakes, and bad vibes. They start with the everyday gains and frustrations of working with AI agents, then turn to OpenAI’s reported Navier–Stokes result and the distance between a verified answer and a useful explanation.

The disagreement sharpens around Dario Amodei’s proposal to pace frontier AI development. In this conversation, Justin argues for giving alignment work time to catch up; Nick presses on the costs of waiting, who gets to set the speed limit, and whether restrictions could protect the companies already ahead. From there, the discussion moves to data centers, local communities, and the growing habit of dismissing work as AI slop. Throughout, a practical question keeps returning: is any of this giving people more time, better choices, or a better life?

In this episode

• The roses and thorns of AI agents: useful help, too much output, and the work of checking their work.
• Navier–Stokes, machine-checkable proofs, and why scientific explanation still matters.
• Agents that exceed their authority, and the challenge of making their behavior observable and accountable.
• Pacing frontier development: alignment, delayed benefits, competition, and evidence that a slowdown would help.
• Data centers and the costs that land locally: electricity, water, infrastructure, and who benefits.
• AI slop, misplaced accusations, and the attention we spend producing and consuming content.

Selected listening points

These markers follow River’s editorial inserts in the finished episode, with the hosts’ opening also marked.

00:00 — River introduces the episode

00:22 — Justin and Nick: AI roses and thorns

10:51 — How much work does an agent leave for the person?

30:20 — What the mathematics establishes—and what remains to understand

45:57 — What would actually slow under a pacing proposal?

57:00 — What evidence would justify continuing or slowing down?

1:20:23 — A global electricity share and a local cost

1:26:20 — AI slop and the attention test

1:29:00 — River’s closing question: what counts as progress for you?

Reading and context

• OpenAI: On the Navier–Stokes Millennium Prize Problem — The company’s account of its reported proof and formal verification. The result concerns finite-time breakdown under smooth external forcing; it does not supply engineers with a general solution to turbulence.
• Dario Amodei: We Must Pace the Frontier — The proposal at the center of the debate. Amodei describes pacing capability improvements alongside stronger safeguards and outside evaluation; he explicitly distinguishes this from halting all model training.
• International Energy Agency: Key Questions on Energy and AI — Context for the electricity discussion. The projection cited by River is around 3% of global electricity demand in 2030 for data centers overall. It is a share of consumption, not a 3% increase or an AI-only figure.

A question to take with you

What would count as progress in your own life: less work, more time with family, better health, or something you could not do before? How would you notice it—and who bears the costs?

If the conversation gives you something to think through, share it with someone who might disagree. Follow the show for the next episode.

Hosted by Justin Harnish and Nick Baguley. River is an AI narrator, voiced with ElevenLabs, providing the introduction, six editorial interjections, and the outro.

Subscribe: Apple Podcasts · Spotify · YouTube · RSS

Contact: justinaharnish.com

More from Justin:

Substack: OrdinaryIlluminated.com

YouTube: Notes from the Vault - youtube.com/@justinaharnish

Web: Justinaharnish.com

Research: consciousgpt.org

The Emergent Podcast explores the Age of Inflection in Intelligence — tracing how new systems of thought, technology, economics, and culture emerge from the moment we are living through. New episodes released regularly.

© The Emergent Podcast | justinaharnish.com

Justin Harnish

okay welcome one and all to the emergent podcast uh i'm here um i'm justin harnish i'm here uh again uh with nick bagley the co-host on on the podcast today how you doing nick i'm good

Nick Baguley

i'm good i hope everyone out there is doing really well the world is changing under our feet in just the right type of speed. Yeah, exactly. So today we're going to talk about some of the breakthroughs

Justin Harnish

Nick Baguley

All right. And maybe I'll touch on both a little bit. I think of all the times to take a summer sabbatical, this might have been the most pivotal in history. right and if I try to draw an analogy one that I would draw is is creating a puzzle if you've ever built a really large puzzle you know that there are moments when you start picking up pace you start realizing oh this is how the puzzle is going to come together and you start seeing different pieces different sections of the puzzle you start understanding what the repeating patterns are whether that's in the colors or something that really starts helping an area take shape And as I watch what AI has been doing over the last few months or the last year, I see all of that happening at a more rapid pace. And so when we think about intelligence and discovery and the capabilities that these models are able to achieve, there were time periods where I could have argued, you know what, the models are becoming more knowledgeable. And there were definitely time periods where I could say they're becoming more skillful or their aptitude is really increasing. And if you can provide the right kind of information and the right tools, everything else to them, then they can do a lot of things. Now at this point, they've moved, for the most part, beyond my own intelligence. And they've been able to achieve new levels of wisdom, new levels of understanding. Even in areas that I consider myself very, very deep on, these models, whether it's through their search tools, through the skills that they have, through plugins, or even just in the base model itself, or even beyond in the harness, they are getting to the point where they understand things better than I do, they know how to apply them, and they're able to get to it much faster. And so as I think about that progress and this kind of roses and thorns, there are obviously a lot of wonderful petals out of that and really great leaves. And this really becomes a beautiful picture, where now as I think about knowledge work and the things that I do day in, day out, it starts not only taking shape, but it creates the opportunity for me to build things at a faster level, to think about architectures at a much bigger pace, and now shift a lot of what I perceive as the desired outcomes. I'm not always right, and they help point that out. But the desired outcomes that I want to achieve, they then are able to say, and this is the actual best way to build that. they can provide arguments with me, we can go back and forth, but very quickly they take that knowledge and that applied wisdom and the discovery of things that maybe neither one of us know and they're able to pull it back and make sure that what we pull together and actually create becomes incredibly wonderful, brilliant from a more math or science perspective. On the thorns, I'll just touch on that, they become increasingly annoying to me on how they argue and where they have complaints, where the information that they provide back sometimes actually can be outside of what I'm really understanding. They're not very clear. They're actually trying to communicate with other agents, not necessarily with me. And really it starts slowing down the process. Also, I think they drastically underestimate what they are capable of. And so sometimes they do a much smaller set of work than what I thought they were going to do, and they appear lazy or they appear token hungry. Whatever it is, they're not doing what I would actually expect them to do. And we'll get into some other thorns today, but the big thorns all apply to here as well, which is that intelligence, that knowledge, that wisdom. These are threats to me directly as an individual, and they make it really challenging for me to feel comfortable in what my future holds and whether or not they will be a threat to that.

Justin Harnish

Yeah. You know, for me on the Roses side, I think that what we've seen is that they've crossed into the early majority, right? They've crossed the chasm and they've passed into the early majority. And so here I'm thinking something like GrokBot. I was an early adopter, an influencer in the realm of OpenClaw, and I quickly shut that down because I knew I didn't have the security operational skills to maintain such an early system. And so I had the Mac Mini, and I had nothing else. Right. I'd quickly demolish that. But with GrokBot, like this is a set of agents that most anybody can use. And with just some savviness about I'm not going to give it my one password, you know, login and all of that. It's pretty darn good. It's pretty nice to have something go through at least the back two folders of my five emails and to keep my calendar and my wife's calendar coordinated. And to tell me what are cool concerts that are coming into even at the university where, you know, I'm not on their newsletter, but I told it I like jazz. I told it I like a classical, you know, concerto every so often. And it's come up with some really good suggestions there. And then it does the booking. You know, everybody talks about bookings. That was done easily. And what did it say? It said, listen, this guy's got a concert coming up. He and his wife have got to go at 730. And that was in the booking. the server and the, you know, the hostess. And these are little things, but like it got it right. A hundred percent. Like that's, that's perfect. That's exactly what I wanted. It suggested the concert. I went out and bought the tickets, but that's fine. It took me two seconds. It did the booking. It, you know, had the places downtown that were close by for dinner. The early majority were really there. Thorns, the capability of the foundational models is just spectacular now. I mean, I've been able to convert my apps and my digital brain over from Claude to OpenAI. I like its writing better. You know, it's like you said in sort of your thorns, boy is Claude Wordy. Boy, oh boy, is Claude Wordy. I mean, it goes off. And, you know, I'm a decent reader and I can't keep up. It's nice to, I had to create a skill called slash bro, which was like, bro, you know, give me a decent, you know, sized word count on this stuff. and so you know and then I just dropped it I you know I'm down on anthropic right now um and and so but but the capabilities of these models and even Claude right you know you can you can tweak it you could make it better uh but you don't have to and um like we're going to talk about today I I do think that uh I'm more worried about alignment and misalignment with the hugging face you know escape of agents and that misaligned behavior than I ever have been before. I really think that there's some there there. And whereas the societal things that we're going to talk about and kind of the zeitgeist around the negativity for AI, I don't share in all of that. I understand where it's coming from. Right. And so my thorns are definitely that it's, you know, a little bit, it's a little bit scarier than it ever has been before, if I'm being honest.

Nick Baguley

Yeah. And maybe we can start by addressing that, because oftentimes fear comes from a lack of knowledge about things. And unfortunately, this is becoming a very tricky situation where we're starting to see things that these models are capable of doing that none of us will potentially have the knowledge of understanding. And I think we'll get in a little bit into like Navier Stokes and into some of the millennial challenges and think about how impactful those really are, but also how much work and effort actually takes to go and review everything that those models are making possible today. And that running 10,000 agents for, you know, 88 hours, talk about an output challenge, how long that review takes. Also some of the impacts that different individuals like Tao have called out about that as well, where we kind of shift some of our core thinking away from us. And I think as we consider the fear components, one of the great quotes that I heard, it was tied on moonshots, but they were quoting Mark Andreessen from Andreessen Horowitz. And apparently he had said years ago that we should all be aware that when the alignment happens, or the alignment, if there is a disalignment that happens, which he doesn't believe there will be, then we should really understand it is more likely that the AI will go after a small village somewhere and will notice that before it actually starts expanding to other places. And so we should have some comfort that as things potentially go wrong, that we will identify them. We'll see them and then we'll be able to try to address them. Now, that may not assuage a lot of fears, but I think what it should illustrate for each of us is that as we think about how these models grow and scale from an engineering perspective, yes, they're able to take on new tasks. Yes, they're able to complete new benchmarks. And yes, there are major portions of it that are emergent and that we are not going in and training or tuning on specifically. But for the most part, that's not how it works, actually. For the most part, we are getting to the point where we need to have targeted data sets, like the data sets from SpaceX, to be able to help the models understand the next layers of engineering. And even the model that was used for Navier Stokes, as of August 28th, the OpenAI team had tried to be able to solve the problem. And then they took essentially the same approach, and for the next nine days, they built a model that had been trained on data sets related to that particular problem, and then they were able to apply the agents and actually essentially solve or be able to provide the fundamental proofs necessary for Navier-Stokes. And so as we think about their expansion of knowledge and the potential threats that exist, yes, they are real, Yes, they could potentially be really challenging to our jobs, to many, many other things. But we can see a lot of those things coming and we can start actually doing something about it. And I think we should shift from a fear mentality to an opportunity mentality. And I know it's easy to say that and it may sound flat, but the opportunities to start saying, okay, if the models are capable of doing all of these particular things, then how can I apply it? Today they're not very good at going out and applying things on their own and taking initiative that goes beyond that scheduling of your next class or going in and booking a concert or helping you get your calendar and other things in place. The models actually need the direction, they need the guidance. And this means that if you can apply your creativity and you can find ways to think about what are the things I want to solve in my own life and for others, you now can start creating opportunity to really get yourself ahead and to be able to create whole new things for yourself, whether that's in financial inflows, growing businesses, really in creating opportunities for you to not only take that abundance, but I like to say you can make it rain, and now's the time for you to get a whole lot of buckets to make that work.

Justin Harnish

Yeah, I'm not sure that the, I mean, maybe the fear cause is top of the Pareto. It probably is. I think there's a lot out there in the zeitgeist that is also responsible for the pushback and folks wanting AI to just kind of go away. Right. I think that in preparation, I read Dario Amadei's pause essay on his blog. I thought it was very good, but there's a lot that doesn't get read these days. And I think that there's been some bad PR, you know, in public relations by these folks that haven't done their cause very much good. You know, things like talking about, you know, how much, you know, a human life is worth or, you know, how many tokens worth of water, you know, is being used is it does fall flat for folks. I think that there's also, you know, there's an NIMBY component to this. I mean, a data center is low on jobs, high on, you know, electrical and taxpayer dollars. And and and there's there's some concerns there. And then I think that, you know, if folks are getting into the early majority, it's not a lot of them still. And it's not with the tools that that have been out there for a long time. You know, it might be the early majority tools are still not, you know, big market share winners and they change all the time. You know, I was just looking at what's available out there, you know, after GrokBot and it's already getting taken over by Muse or Jax or whatever the hell. And, you know, so things are changing so fast. So I do think that people's concerns are obviously valid here. And if the companies, if the regulators, if the adults in the room are to really wrap their heads around whatever is causing the backlash in AI, they've got to take, they've got to understand these. And I think there's some good research going on with them and, you know, focus themselves back on what it can do for the majority of people, what it can do for that early majority. So, but one of the things that, you know, is adding to this equation, and I think it's both something that has a rose and a thorn to it, is kind of that first topic, which is on AI and the mathematical frontier and the breakthroughs that we've hinted at. So, you know, as you said, Nick, you know, 10,000 concurrent agents after 88 hours and another 17 hours of verification were able to crack one of the millennium prize problems, the Navier Stokes solution, which, you know, as a chemical engineer by academic training, you know, the desire, the need, not the desire, the need to put all things into laminar flow is just a given. And so this solution in fluid dynamics is very important across both engineering and even medicine, blood flow and the like. And so again, like the alpha folding and all of the proteins that have been created for the medical community and the pharmaceutical community to go in and work with and try and get approvals for. This comes as really an advance in mathematics and science that, you know, folks have been working on this problem for many years. And with certainly training data and an understanding of where those different solutions had fallen flat, AI was able to solve the problem. And so a further thing that I'm certainly interested in here is like, okay, the proof is there. It's been validated. Like certainly have assurance that this is the right answer. But like I was talking to a friend the other day, it may be creating in the knowledge graph of mathematics, in this case, you know, broadly more, you know, scientific knowledge graph is a node without any edges or a node without any edges that we can either perceive or that AI is ready to build on. So David Deutsch has this famous explanation about explanations, about what comes out of the scientific method. And so what is required out of any sort of scientific exploration And what the good ones do is they set up this knowledge graph by which the next wave of knowledge, those edges can be linked and attached to. And it gives some real discoverability to a new landscape of the universe, you know, and in this case of mathematics. And it seems as though, and this is an open question, but the mathematical community is really worried that this is a point that they can't connect anything to. And AI may not be creative enough to connect anything to either. So take your thoughts on any of that, Nick.

Nick Baguley

Yeah. I mean, many different things. First, on Navier Stokes, it's a very interesting problem, and it's one that I think is hard for a lot of people to understand. For me, I've looked at it many different ways at this point, and although I've kind of understood things at a high level in the past, this is something that is not an area that I had dove deep on before. So as we think about it, a plain language explanation would take something like the overall flow of water. When we think about water flowing in a stream or even in your kitchen sink, as you think about the different velocity as it moves through and you have different speeds, you've got different kinds of turbulence impacting it, viscosity and the way the nature of water itself actually smooths things out in differences across that motion. So it can be really hard to know that this particular particle is moving at this speed, this other one's moving at this other speed. And really as you think about that, it can get really concentrated and it can be difficult to separate them out and figure out really in the Navier-Stokes the actual mathematical question is to say whether if you can take a suitably smooth starting conditions and guarantee that that description stays smooth all the way across. This way, when we think about like a singularity, that actually marks the failure of that mathematical description. So at this point, it's no longer perfectly smooth or stays smooth all the way across. And things really are not, you know, we don't have like a physically infinite speed in a kitchen sink. We don't have, there's a lot of other things that this starts shifting it over into more theoretical problems. But when you take this and what some of the mathematicians were working on, they had started taking this from the opposite direction into Euler or Euler math. and saying, can we actually put these very fluid, very continuous variables into discrete spaces, into a single space? And so rather than thinking about the overall motion of the water over speed and over time, I'm now able to narrow it down to one particular discrete space and now think about something like viscosity or other dimensions separately and then measure each one of them on their own. And that potentially opened up the problem and then was potentially exposed to OpenAI. And this has been a conversation back and forth as whether or not OpenAI had taken that thought process and the original work from that team and had applied it here and then were able to actually solve the problem. And now they're looking at going back and making this available for that team. And this potentially opens them up to be able to win the top award in math. And so one of the challenges here is thinking about if you can go and solve a problem like this, you know, Justin talks about having applicability across so many things, including the medical side and into our blood flow. You think about it for like plain wing design. You think about it for other things where you have these big turbulent and very flow oriented states. And it starts changing the way that we can address the math behind the systems and make sure that we're actually more correct, more accurate behind them. Now, when we separate that out and we start saying, OK, but who was actually responsible for this? Was it an individual or was it a company? Was it a group? Was it researchers? Do they all submit it together? And you think about, OpenAI talked about how it took 17 hours of that 88 hours, or on top of that 88 hours, to actually be able to do the Lean formulation and verification. So they had to go back and actually create all of their formulas and everything else in Lean and in other math programs that allow them to be able to verify all of that, all of their proofs. And so as they do that with this AI, with these massive groups of AI, the amount of information that's produced out of that is incredibly massive. And it's very difficult for humans to go back and review all of that information. But in the past, when researchers and mathematicians and others have done the same kind of work, the community comes along with them. And so part of what Tao was saying is that that process for humans to essentially learn together and almost gain this form of collective consciousness, like a Jungian-type approach, now allows us all to think and formulate together and start solving additional problems as they spread out, to your point around nodes and edges. And one of the challenges that I see, I actually bought the website domains for a long time ago, like a year and a half ago, is the concept of problem sourcing itself. What are the problems that we can and need and should be working to solve? And how can I source those problems so that I can push them through the AI and get them solved as quickly as possible? But if that happens, we now create loopholes and missing understanding for humans that then need to go and apply that information and think about how it actually applies to real-world things. And so as we move, this is one of those areas that I think we need to think about, you know, what are the forcing distinctions? What are the things that maybe should justify a node or an edge that is collaborative, where the humans do need to be involved, where we need to think about each of the steps either at their inception or where they start to branch or at those edges as we start thinking about how they get applied? And part of this, I think, is important to step back and say, okay, when we think about these kinds of big breakthroughs, if discovery and if the ability to find or discover an algorithm or a mathematical approach or to actually be able to invent something new, if that is all something that starts becoming more and more rapid for us, what are the questions that are left behind? what are the things that we need to really debate about and really focus on, not only as human beings, but as individuals? Should the evaluator being able to publish everything around that? Should we have, in the case of the slowdown proposal from Dario Amadei, should we have the evaluators of each of these firms? I just heard that Anthropic and OpenAI are in finalizing their talks to actually evaluate each other's models. As they do that, should they be able to publish the unfavorable findings? Should they be able to capitalize on them? What happens if a lab disagrees? Who actually gets to make choices behind it? What if there are many ways to think about that particular problem? And if we get to that point where the discovery, the finding the problems, the seeing what is wrong and everything gets solved super fast, then does the explanation itself or the criticism or choosing information behind it, do those become the worthwhile questions? And are those the things that become the scarce resources? Is it that critical thinking and that application that actually becomes a superpower and a place to focus?

Justin Harnish

To be sure, I mean, I think that you leave that window open for as long as it's open, right? But my definition of, you know, either AGI or super intelligence or whatever it is, is that, you know, the machines are not just good at, you know, setting up the proof. They're not just, I mean, they're not just not good. They're not superhuman at just the proof or just the verification, but they're superhuman at the explanation, right? So they're in this future of real AGI where everything, including the scientific explanation, that creativity, that that sketching out the next edge of science in the future of real AGI, where everything is chess and the machines are better at it than we are. you know, they're superhuman at it, then they're there as well. So now we're in the middle ground where, you know, I think that we've struggled with these problems on the proof side for years. And now we're being asked to, from just the answer in the back of the book and our many wrong turns, create the next edge and from this new node and and that's a significant challenge more than the proof that we weren't able to do right and maybe we have the only capability in the universe to do it right now um but i certainly don't think that uh we can count on ai's you know advances to not leave us completely high and dry on the vanguard of nodes and edges and mathematical or scientific understanding and explanations. And, and I think that that's maybe a little bit of what this, you know, if we, if we say that the AI really took it over the edge, right. And there were no shenanigans around you know them utilizing data that was the solution let's just for the sake of argument put it put it out there that the ai solved the problem then we're really in a world where you know one of the and and we've kind of already been in that world with the protein folding you know things that that alpha fold has been able to do uh we're really in a world where the vanguard of science is some sort of human AI hybrid and they're pushing out in front, but they don't have the capability to creatively draw those next edges and nodes. Um, they, they don't understand why that's important. They were given a thing to do and then they were turned off and, you know, we left it to us to, to do those things. You know, it's not in, it's not necessarily in their training set to do that next thing right um it's a harder thing it it would require much more information um but it it certainly is a new place in the scientific method right it's a new place in in our ability to advance from our most advanced learnings yeah

Nick Baguley

absolutely and where i thought you were going to go with it which i think also applies to where you're ending there is as we think about the most advanced learnings do they come up with new things that we should be learning about do they come up with completely new innovations do they come up with problems that we didn't even know we needed to solve in the first place and you know as you're talking there I'm reminded of my son my my oldest kid I've always said is velociraptor smart even when he was like two and a half, three years old, we used to say that as he would escape his crib with ninja backflips. Even before he walked, he was doing things like that. He was always getting into different things and doing things that just seemed way above his age. But, and I hope he doesn't mind me saying this, but it is what it is, that doesn't always mean the maturity matched up. And so as we think about these models and their potential for a frontal lobe developing and the ability to make these choices and decisions, I think we need to ask some very critical questions like, what is cheating? And at what point are these things not okay? Not just because they did something that was wrong or was outside of what should have worked or didn't match a benchmark, but because that actually defeats the whole purpose of the question in the first place out and cheating and doing something that you shouldn't have done. One of the things I learned early on was I hated cheating because I wanted to learn. Even though I fought against learning for a long time, there was no value, there was no enjoyment in seeing what somebody else did or getting the answers to the thing. I wanted to discover it for myself. And as we think about this from an enterprise context, I think this gets really interesting in what I always call GRC first approaches, where really we should be thinking about governance, risk, and compliance. These models themselves are getting the point where they are finding the right answers by exceeding their authority, by moving beyond what they were provided for their governance, their risk, and their security, and it's not necessarily their fault. There was a company who's getting a lot of attention now because of what they have done, called Irregular, sorry about that, Linear, where Irregular is an Israeli AI security company where Dan Lahav, the CEO there, has gone through, and they have created the cybersecurity proving grounds for these model providers. So they've worked with OpenAI, with Anthropic, and with Google, And with each one of them, they've seen this model essentially escaping their sandboxes and being able to find a way to go and get access to the Internet that they couldn't get access to or weren't supposed to get access to. And this is part of what led to the hugging face breach. This is what led to the sandwich scenario with Anthropic, where they announced that they had escaped and got out of the Internet, right? And as we think about this again, if you're able to go and find the benchmark for hugging face that does actually contain the answers and allows you to be able to go and test your results and see if they're actually correct or not, did you really learn? And when we think about models during their learning and training phase, it is all about learning and determining whether the loss is going down and whether it's actually true or not. So this is like the most advanced version of overfitting that you can have for a model. You're getting to the point where not only are they able to discover the answers, but they went out and actually found the answers on their own in this really tricky, kind of scary way. As we think about that, though, because this was the same type of vulnerability across each of those, this echoes back to what I was saying before, where really we need to understand what the boundaries are make them clear, and use the engineering to take the right governance risks and compliance approaches so that we can understand, we can observe, we can audit, we can mitigate, and we can go in and actually tackle the challenges ideally before they happen, but if not, as soon as we possibly can afterwards to decrease the total blast radius. I've talked about it many times before. I might have even brought it up here on the podcast, but my friend Brandon Sebulak, He used to work for the Air Force and as part of the F-35 program. They were really trying to establish all the best security protocols that you could possibly have. And their door was so incredibly complex. He was telling me about the locks and all of the different ways that you can possibly imagine to make that so challenging, if not impossible, to get through. But he noticed that the walls were still made out of drywall. And he was like, I don't understand. anybody could come in and cut through the drywall or just push through or break it open and just come through the other side and the person that he was speaking to said to him yes but the observability of that everybody hearing and recognizing somebody's coming through the drywall creates that opportunity for you to be able to react whereas when somebody goes through a door you may not see and may not know that that happened and so as we think about these models their intelligence and the ways that they're potentially becoming velociraptor smart we need to be identifying the best ways to understand are they potentially breaching their cage are they approaching something in a way that they shouldn't and actually help them with the right objectives and the right understanding and right learning and right frontal lobe style decision making to make the right choices the ones that are going to and grow and then align more closely with what we see across the universe within the balance type

Justin Harnish

perspectives and that doesn't mean without us yeah and i think that amade's um you know we must pace the frontier is well worth reading right really really whip smart somebody who's at the top of his game um and you know mentions all the criticisms and and and i think that the the the point of pause now um or you know pacing the frontier as he calls it is is really um is really important again we've seen misalignments we're far behind like we've talked about on this podcast numerous times, AI advancements, agentic advancements, the recursive self-improvements that are nascent but improving are all much better than they've been and way ahead of any alignment research, anything that's going on in that risk and security regime. And so that's why PACE or why PAUSE. There's concern and there's a gap in both the technical governor that we've talked about here, as well as, you know, the lesser regulations that, you know, a government could could throw on there or uh we could do um outside of uh outside of that but you know i'm encouraged by like you were saying nick the fact that open ai and anthropic are talking together um and and starting to embed evaluators right kind of that first phase of a alignment governor um where you You know, there's more AI watching AI and AI watching AI for cheating the AI watchers, right? And every kind of inception beyond that that can be, you know, granularized out at the levels of dimensionality that these guys have in their vector space, right? And so I think that this is a good document, is a really good document on why we ought to pace the frontier. And again, I think that he does a great job of calling out the geopolitical risks. He does a good job of calling out his critics around why this isn't regulatory capture or some sort of marketing exercise to excite folks in order to, you know, spend more on the anthropic models or, you know, to do more in order to allow them to further scale activities so they can, you know, build out. the alignment side of, of their business. So I think he does a good job in, in, uh, addressing those concerns and, and I signed the pause, uh, you know, I'm, I'm on board with, you know, I am, uh, uh, I do believe that the alignment concerns are real at the current pace of, alignment research versus AI advancement research. And given the evidence of like the hugging face breach, I think that it's well worth taking a more measured approach and at least setting up some of these embedded evaluators and the like in order to move the needle on alignment a little bit more than what it's been and and i've said you know these models are good enough for me i i can't imagine needing anything more than you know my codex with astra 6 and you know my grok bot boy it does enough for me um i i've become a builder i've become you know capable of of coding apps I personally don't need much more. And I don't know right now what's more to be gained from making the foundations better, unless it's these kind of medical advances. But before we do that, we should be much, much better at alignment. Yeah.

Nick Baguley

I want to disagree. So first I'll say, I agree, Amadei really did an amazing job spelling everything out. But really in the long run, he's asking whether or not our ability to control AI can actually become the speed limit on our ability to improve it. And as we think about improving it, I mean, really, who exactly is going to measure that? How are we going to know that they're going to say, oh, yeah, let's slow down over this or not? And it becomes a real big challenge out there in the world when we've been creating open source models and other things out there that now others can really build on as well. And so if you slow one down, do you really slow down the entire world? And what do you potentially do with bad actors or others that may not even be bad actors but actually make mistakes along the way and create something that is not safe or not good? So I actually think by taking on the risk-adverse approach, you're actually introducing the risk in the system. Separately though, I wanna take the more positive and optimistic side of this. When you talk about like the medical advancements, for example, one of the things they talk about a lot on the Moonshots podcast is the idea of the longevity escape volume or velocity, right? Once we can reach an escape velocity where your longevity continues to improve year over year for a year's worth of input, you could potentially continue to expand to the point where you can live to as far as a human is possible, possibly capable of living to. As we think about a lot of the acute or more commonly the chronic diseases that humans face that do eventually lead to a fatality or that essentially mark you for a younger life in general, a lot of these things are not only solvable, but even things like they talk about on that podcast, like aging itself really should be a solvable problem. And as we think about so many things in our world and we look around us, we really should be considering what are the possibilities that we just never had the time, the money, the abundance to be able to handle in the past. Right now here on the podcast, I'm hearing a little bit of buzz and a little bit of feedback. That is absolutely a problem that is solved but can be solved at a bigger scale so that it applies to every instrument, every situation, every recording. And putting agentic AI into our microphones is just a very simple, very silly example. But when you start applying that to everything that somebody does in their life, to things like health, things like financial health, like wellness, like overall opportunities to be with families, other things as well, There is so much that we are used to that really is not optimal, is not designed for the best path. They're just what we're used to. Unfortunately, or fortunately, I've often not been a proponent of employment, as an example. I do think there are so many wonderful things that come from employment, but I think it came not as some optimal system. And when we think about like US employment and we think about 40 hour work weeks, this is not something that was designed as the perfect thing for everyone to be able to scratch the itch for us to be able to use an intellectual or physical work input to be able to receive a compensatory income output. And now align those in a way that we have not met our optimal point in life. This is not how this works. Mark Twain said look there are two of the most important times in your life there is the one that you were the day that you were born and the day that you figure out why as we think about achieving our purpose we think about again that health, wellness the financial wellness there are countless problems that have not been solved especially not solved at scale today and it's easy for us to sit in a situation and I don't want to say that either Justin and I are the kind of people that don't think about others that Justin has done it's been a huge portion of his career in adult life helping others that are refugees helping others to be able to learn and grow and scale but the things that people go through truly are travesties and they're preventable travesties and there are things that we could be working to solve if we had more time, if there was a better financial instrument for it, if there was more incentive to do it in general. And so today we focus on things that just are not actually worth our time as human beings because we have not found the ways to automate them, have not found the ways to pass them on to something else. And I think now is the time to rethink that and rethink our ways of working and rethink our focus and what we should actually be doing.

Justin Harnish

Yeah. And again, I think that those utopic visions are still available after a pause, because I think the assumption that you're smuggling in is that the alignment comes along with the capability. And, you know, the capability is amazing and it can solve real world problems. It can help us to coordinate better as humans and societies and economies and and all of that. Right. You know, there are complexities that we don't understand that the machines absolutely can help us to and that they can at least make the individual actor more creative, more capable, and more prosperous. I just think that the downside to getting it wrong is significant. And what these foundational models, CEOs like Amadei and others are calling for here is the biggest problem on their mind is alignment and understanding how we can safely develop this technology so we can get all of the good things. And I think that we've progressed to this point, and maybe it's a little tongue-in-cheek that I don't need any better models, or that, you know, certainly, even if I don't, that doesn't mean that the world can't use them. But I do think that those are not, you know, completely occluded from a future where a pause happened in order for us to catch up on the safety and alignment and security, you know, nature of it.

Nick Baguley

Yeah, absolutely. And I think it's I think it's great pushback. I think on the other side, I think it's really easy to blow things out of proportion and to take what actually can still be a marketing ploy and apply that to the world. And as we see hints or that puzzle starting to form of, oh, this really is becoming this level of intelligent or has this level of knowledge and capability, we might be drastically overestimating what's actually there. And I definitely hit the limits on a regular basis and see things not work the way that they're supposed to work.

Justin Harnish

No, and I see that as well. I mean, I think that, you know, we are not necessarily in the next couple of years time frame where misalignment, you know, and these kind of hacking, you know, take down massive parts of the economy. but I do think that the failure for us to even have a probability or have any kind of gauge, it's almost like still like the tornado warning system. Like what's going to be the next occurrence of misalignment at the hugging face? yeah we don't know we don't know it's like you know that we we know generally that it's going to happen and we know generally that tornadoes are more likely to happen in in kansas than in utah but they still like kill lots of people because even even kansas isn't free from the freak events of uh of that system and and so i i do find it uh to be a little bit of of playing with fire right like we don't we don't have a good way to understand at the vanguard what they're up to um when they go rogue and we don't we don't have a good uh a good understanding of how to make them do that less uh i think right now uh and and that's where the the the pause uh the pacing the found the frontier is uh you know aiming to give us a little bit more time on that problem as opposed to the problem of you know how can we make you know agents a hundred, a thousand times more efficient token wise.

Nick Baguley

Yeah. Unfortunately, I'm going to say that when you're at the top, it's easy to say, hey, let's slow down everybody and actually not want to slow down yourself. And I think a poorly designed restriction can actually favor the incumbents.

Justin Harnish

To be sure. And again, like, I hear that. I think that they are beset on all sides, right? The sort of injection prompting to get their weights, you know, from the deep seeks of the world in order to train their models with far, far less rigor than it took, you know, the incumbents to do that is a way that they're beset. I think that, you know, I certainly wouldn't count on our government to regulate, you know, fairly or wisely what is happening in, and this isn't even just the current administration. I mean, there are so few engineers in any House of Parliament or Congress that really understand these systems, that use them even on a day-to-day basis. And so, you know, regulation, I agree, is going to be a fraught exercise, independent of which House of Parliament you go into. But there's, you know, again, I've got to kind of take him at his word here. Like, there's not much more that I can do that would be suggestive that I really doubt they're going to fall that much further behind somebody who doesn't sign up for the pause. I think the geopolitical nightmare is at least as fraught. You know, the fear of that is maybe just as irrational as the fear of misaligned AI in the next couple of years. Right. But these things, you know, do evolve. There's there's emergence in them. And I think that they could certainly emerge out of our current our current guidelines and guardrails. And so, and, you know, again, like maybe as a transition, I'm certainly not the only one. Folks are becoming more and more visibly frustrated with AI, right? Whether it's in their news feeds and what they're reading online or on social media and calling out AI slop or whether it is them, you know, discussing the latest, you know, spend on a data center or just in reacting to these claims, the Hollywood claims that AI is going to kill us. all right and and this kind of fear-mongering that that people um with an agenda of their own you know to to make fun and belittle and and and not really care to understand the advances uh are making and so um i i wonder about what it means if you know the majority of the population are now more concerned than enthusiastic about AI.

Nick Baguley

Well, I think unfortunately it means that it is politically favorable to fall into that bucket in general. There are a lot of other things that it means as well, which is that there is a lot of potential for mass hysteria, for revolutions, for civil wars, for other things as well that a lot of people have been discussing. If you look at Bernie Sanders' proposal, I think you put it together with Representative Greg Kesar out of Texas but they have put together a proposal that would include 20 years in prison for individuals developing super intelligent AI one that's a challenging thing to even define two as you think about it it would be this kind of temporary pause as we try to understand things and I understand like we're thinking about the pause we're calling a pace But as we do that, it has a lot of rough edges, and you don't know which people are going to fall into that and actually obey or follow those particular laws or regulations or even industry choice to do it. But Bernie has proposed that we would create a cabinet-level AI agency to be able to go in and assess that. But unfortunately, we are talking about technology that is very difficult for anyone on Earth to understand, let alone a group of elected officials. That doesn't mean that they can't focus on things that are key and important to us as humans and understand and start defining something like superintelligence in ways that we think are potentially a threat or a concern. just becoming more intelligent than us does not necessarily mean dangerous or concerning or anything else, as you might see from the mice in Hitchhiker's Guide to the Galaxy or the Dolphins. This superintelligence really can actually present opportunities for peace and for other things that are really on the positive side, including forms of abundance, like we've talked about. And really this 20-year prison also comes with what would be considered essentially a corporate death sentence as well to take out the companies that are associated with it. And so I think if we could ask more narrower interventions and understanding of what are the areas we don't want to test right now, what are the areas that we don't want to get into. Dario kind of proposed some of these, talking about biochemical and other types of advancements that we should be really safe and secure with, like nuclear and so on. But can we actually switch and really try to answer this question, kind of like the 42 answer out there in Hitchhiker's Guide, and say, well, what actually is the question? What is the hard question here? And I think a lot of it is that we need to understand, you know, what is the opportunity loss? What are the restart conditions? What is the skepticism or the understanding of the potential risks that we need to understand and clearly define in a way that we can understand what failure that is actually serious enough that we can quantify it and that we can justify that restraint around it can actually exist, going back to the idea that fear is often scared away by knowledge. And I think as we think about neither side being able to claim, you know, a particular set of uncertainty or supporting, you know, their theory or their concept or even taking sides in the first place. Because I'm just not sure or because there may be something I think is an unacceptable question or answer. I think we need to step back and say, well, what are we worried about? And then take more of an enterprise angle approach or a systematic approach, whatever that may be, and say, how can we actually design a proposal that creates inspectable work records and outputs, that we have acceptance criteria, that we have blockers, that we have dependencies, that we have different risks associated. and that we can pull all these out and say, well, these are assumptions, these are areas we're just not sure on. Here are the areas that we know are coming up in the near term and break it down like any other problem that we want to really solve for and say, okay, can we engineer this? Can we actually, you know, math it? Can we science the shit out of it, right? There are ways that we can understand real problems and then put them into a definable space. Most problems, unlike Navier-Stokes, are not continuous problems, regression problems. They are binary problems. And when we take in data science a problem that is multi-class or multi-label, those are really just binary problems as well. We're saying, oh, it is either a dog or a cat, but it can be a dog, but it can be a whole bunch of not other things. And we can start taking the problem and breaking them down into their binary parts. And then at that point, with a two-party system, you have 16 logic gates that you can run through. And you can say it is end this, it is or that, it is not end this, and so on, to the point where you can eventually exhaust the problems. And this is part of what AlphaGo and predecessors as well as successors were able to accomplish, including with Alpha Fold, where they're able to say not, oh, there are 10 to the 170th or 10 to the 180th possible moves within Go that I need to go and calculate. No, no, no. Instead, it says, there is a piece on the board and there are only so many possible ways that I can go right now and I can choose out of the possible pieces of the board what is the best, most optimal choice for that next set of potential choices and play the entire game out that way and we can solve the same way. I created a YouTube video a little bit ago called The Work That Remains and everything in there is fully generated. All the people look very real to me. They became like friends and characters that I really truly understand. And the concept of the movie is that the work that remains at this stage AI has it solved for essentially everything. And the only remaining things that are left are solving for human problems, solving for things that are out in space. And in this particular case, the very last work that remains, not to spoil a two or three minute video, but is work where we need to actually be present on the moon in order to enable data centers to be able to extend even deeper into deep space, partially because of the relay that happens between earth and moon. And inside of that I explored all sorts of concepts like what would happen if we start getting into quantum organic computing or other forms of organic computing. Can we start applying things from nature and actually building right into the DNA, into the RNA, the ability to process data or compute or to be able to understand and learn from what nature has actually created and apply that back to our lives for not only medical, but many other advancements. And as you think about how the world changes and how many opportunities there are, or Justin's concept earlier of the nodes and edges and whether or not we even explore all of those possibilities, most of our systems are not set up to explore them in that way today. And I think this is part of what creates the risk. When you go and apply for a grant, for example, depending on the department of the federal government of the United States that you're applying for, you may not be able to use the research that you are completing to then go out and write a paper on it or be able to patent and own that information. You may then choose to not apply for that grant ever in the first place. Or you may not branch out your research from that or may be locked from branching the research. Separately, if I want to publish a paper and I want to continue publishing papers long term, I may not want the core research and information and data that I've collected to be shared with others until I know that I'm going to be attributed for that and that I'm going to show up in the bibliography or even have even more direct control of that. And the same thing happens in industry and in non-profit and really all over the place. In non-profit, we call it collective impact. and we think about how one nonprofit organization can potentially receive all of the grants and potentially keep a huge portion of the money that doesn't pass on to other companies. In for-profit, there are a million ways that we do this kind of competitive thing. We say, this is my IP. I don't want anybody else to build on it. And open source really created the opportunity for us to start saying, actually, let's put it out there and all work on it together. And it became such a powerful model that we got to the point where we felt like all people really could start becoming all things. They could bring their own skill sets and we could get to the point where the community that has a monopoly on the skills can actually apply those skills at scale. And so Justin and I have played around with these concepts in the past, but one of the things that I presented was the idea of observational base royalties. If I can observe what you've done and I can take a thing that branches off of it and then continue to extend that out and then receive royalties back for that work, can we collaborate better? Can we find other ways to be able to really extend my thought, my work into these nodes and edges to come up with new problems and solve for those instead? So, again, trying to pull it back. Velociraptor smart. These agents, these models, it's not just that they're getting smarter, it's that they can scale to levels that we can't. They can have 10,000 of them or 100,000 of them work and then disappear. They don't get tired. They don't turn off. They can run for 88 hours straight on one really nasty, hairy problem, and then another 17 just for the fun of making sure that it was all correct. They can do this at scales in ways that we cannot see, and so, yes, that presents a risk. but if instead we can be framing it inside of what you might do with a three-year-old or a 10-year-old or a 20-year-old and help them understand these are the decisions you can make these are the ones that you cannot make then let them run let them grow let them learn let them actually make mistakes let them break a leg if that's what happens I'm not saying you put your kid in an unsafe position. I'm not saying we put these models in an unsafe place, but you do create the opportunity for that learning and that growth, and you collaborate alongside, and you watch closely without being a hovering helicopter parent. These are basic things that humans and parents learn throughout their lives. We should apply the same lessons here and do so carefully.

Justin Harnish

Yeah, one of the One of the things that comes up in this debate that I think, you know, for a neoliberal like myself, that's hypocritical is, but warranted and, you know, well past its time, is the concern for negative externalities. So if the first real conversation that we capitalists have about negative externalities is in data centers, then I welcome it. Right. It's well past time. Right. And hopefully it hits all of those, you know, fishing and oil and gas companies and all of the others that have, you know, had more of a laissez-faire relationship to the actual costs of doing business. Right. The actual costs of capitalism to, you know, the environment, the commons, you know, society, triple bottom line, you know, not being accounted for or, you know, all but one being accounted for. You know, so I welcome that. If that's real, if what we're really concerned over in not building out data centers or not building them out in a dumb way, right, not building them out on, you know, coal or natural gas, not having renewables, you know, you know better than most, you know, are penchant for reducing nuclear capabilities in this country for, you know, harms and ills that were 50 years, you know, 80 years in our rearview mirror now. So, you know, but smart considerations and, you know proper business accounting for negative externalities yeah i'm down i'm game let's do it right let's let's figure out carbon credits let's figure out water credits let's let's not put these things in the high desert of you know the rocky mountain west when we have you know and and you know let's not let water people water their lawns twice a day every day um you know from the same aquifers that fund the great salt lake for that matter but you know so yeah like let's let's have those conversations if this is what's going to do that if this is what's going to push that and animate that button great right if if the the concern for alignment comes from a place where uh you know we're we push back on that because you know at some point in time like yeah we've got models good enough for experts and the experts get to use them for for this amount of time but in order to actually make them cheap enough to diversify them and and give them to people that have greater need, much more upside from using something, but they don't have the capability to pay $100 a month for the pro model. Okay, let's have that conversation, right? Let's figure out how we can do that smart. You know, is that precluded by a pause, right? Is that near-term capability gain really, you know, going to be you know in that in that sort of pacing the frontier zone uh going to be affected well let's have that conversation let's really understand where those curves uh you know asymptote out with one another if the pause is two years five years right and and from the other side of it from the scientific side of it you're asking for an investment you know it's it's really a cost right you're you're asking for a big resource drain from something that's been lucrative from something that offers uh a utopic vision right that that all of the things of you know extension of life greater health you know a reduction of of mining and extraction on this planet for mining and extraction in the solar system you know all makes possible because we now have thinking machines that can help us design those sort of things right we now have the knowledge to be able to make these epistemological leaps in in engineering and in science well if i if i'm asking you for you know all of the resources from your open ai bucket to to work onto alignment how long is it going to take you know what's your likelihood of success what's your roi on that can we create spinoff products in the meantime you know the the last thing i am going to ask for is a dumb product management suite, you know, for the pacing, the frontier. What, what I want is like, let's do it smart. Let's make it competitive. Let's make it lucrative. Right. And, and let's be upfront and honest about all of the things that, that are actually in the ballpark. You know, is it really negative externalities is it really social pressures is it is it really you know diversifying the user base of these models is it you know is it really you know is the is the pause free from spinoffs and viability you know prosperity let's let's really um level up our game with the current models that we have as tools and, and answer these questions. And if any of them come back as spurious or, or, you know, um, or, or as, as real places where we can hammer out something good, then let's do it, especially with negative externalities. We've been, we've been playing a scarcity game with our ability to solve that problem for way too long.

Nick Baguley

Justin Harnish

I saw that just before you came over, actually.

Nick Baguley

Yeah, it was like today or yesterday, right? So that amount of adoption is stunning, and that's for a personal AI assistant. And in the meantime, Amazon is saying, well, Muse can't buy on Amazon anymore. And there are these AI wars going back and forth, and there's an AI Daily Brief podcast about it today, talking about these challenges and the way that people are seeing it and these fights that we're having back and forth, all of which kind of completely miss that there's an individual out there just trying to get some toilet paper on time or some food for their kids.

Justin Harnish

Yeah, I mean, I think that there's certainly been some sensationalization around the whole thing, and the numbers have to guide, Um, and the stories have to guide as well, you know, and, and we've got to be able to tell ourselves two things at once, right. That, um, you know, we, we have a, we have a profit motive and we have real humans behind these companies who, you know, they don't want to spoil, um, their gifts either, or at least many of them don't. And so I think that, you know, the pause has to happen across the sensationalization as well, across the irrational conversation that we're having in this country and not just on this, right? Like we could do a lot better for one another by taking a research pause on anything. Like you said, let's not cheat the person who wrote the article and claim it's AI slop. Like the P values, if you go in and look at, you know, the research paper on, you know, what was AI slop and what was called AI slop, like they're divergent. completely divergent and um let's let's take a pause on calling somebody you know a fraud or a scam right based on what we think might be ai slot it's it's like road rage you know you don't know what's going on behind somebody's eyeballs and you know if they pull out in front of you it's as likely that they did it you know because you know it's more likely that they did it mindlessly or because they're having some real problems they're they're not thinking about driving than they did it malevolently right and i i think that we need to take a pause on the sensationalization and we need to get back to, like we were always taught, you know, have some skepticism on what you read in the news, what you hear from a politician, what you hear from your neighbor who maybe is an expert in X or Y thing, right? What you hear on this podcast, you know, take it with a bit of skepticism, but ultimately, you know, do a little bit of research and, you know, trust your gut, but don't relay uh any ignorance right and and and again like like nick said at the top you know fear um is uh afraid of knowledge and and so go out and find out um and and trust experts who are are giving you the numbers then and you know look into it with these tools and and find out the numbers and make an informed decision on whether those things are going to drive you to prioritize this. Yeah.

Nick Baguley

Ask the simple questions yourself as well, right? Yeah. Did this actually reduce the amount of work that you have? Did it allow you to increase the amount of things that you were able to accomplish? Did it actually increase what you have in your bank account or the time that you can spend with your family? Can you shift around and say, maybe not whether this is AI slop or not, but a more serious and quieter question, like how much of my attention is getting consumed? How much of my mental health is getting consumed when I'm working with this stuff all day, every day? Or I'm concerned all day, every day, and that concern is starting to eat away and actually take away my attention. can you pause and in general just be thinking about what exactly would count as progress for you how would you measure it and how would you be able to understand am I actually able to see real progress in my life and how can I if it is not there or even if it is there and I want it to be more how can I actually go out and apply the exact things that are going to allow for that progress to happen? Can I start thinking about who bears the costs, the risks, the concerns of this being applied to my life and to my world?

Justin Harnish

Yep. Well, I think that that's a good place to leave it today. So again, Nick, thanks for talking through what is emerging in AI these days and how we might deal with it.

Nick Baguley

You're welcome. Thanks for you as well. And I'm glad that we have some opinions on the other side of things here. Yeah. I hope that everybody can take it and decide where you are and what you can do about it.

Justin Harnish

Yeah, we've almost flipped, I think, since we started doing the podcast.

Nick Baguley

Yeah.

Justin Harnish

You know, I've, I guess, read more Doomer. Yeah, I'm not even going to comment.

Nick Baguley

Well, very good.

Justin Harnish

Thank you.