Lukas Biewald, former founder and CEO of Figure Eight, and current founder and CEO of Weights and Biases, sits down with Cindy Moehring to discuss his experience and efforts to better the future of AI and machine learning. The pair discusses Figure Eight’s improvements of training data, along with Weights and Biases advancements of human-in-the-loop systems and versioning models to build models faster and better. They rounded out their discussion with a brief talk of inclusion and women in STEM.

Learn more about the Business Integrity Leadership Initiative by visiting our website at https://walton.uark.edu/business-integrity/

Links from episode:

• Read more about Appen acquiring Figure Eight here: https://techcrunch.com/2019/03/10/appen-acquires-figure-eight/
• Weights & Biases raises $135M Series C to keep building MLOps software: https://techcrunch.com/2021/10/13/weights-biases-raises-new-capital-as-developer-tools-remain-a-venture-focus-ml-matures/
• Learn more about Lukas Biewald’s podcast, Gradient Dissent here: https://podcasts.apple.com/us/podcast/gradient-dissent/id1504567418
• Listen to “Daphne Koller - Digital Biology and the Next Epoch of Science” on the Gradient Dissent podcast here: https://podcasts.apple.com/us/podcast/daphne-koller-digital-biology-and-the-next-epoch-of-science/id1504567418?i=1000509701513
• Listen to “Emily M. Bender - Language Models and Linguistics” on the Gradient Dissent podcast here: https://podcasts.apple.com/us/podcast/emily-m-bender-language-models-and-linguistics/id1504567418?i=1000534822471
• Listen to the Lex Fridman Podcast here: https://lexfridman.com/podcast/
• Take free courses at fast.ai here: https://www.fast.ai

Hi, everyone. I'm Cindy Moehring, the founder and Executive Chair of the business integrity Leadership Initiative at the Sam M. Walton College of Business, and this is the biz, the business integrity school podcast. Here we talk about applying ethics, integrity and courageous leadership and business, education and most importantly, your life today, I've had nearly 30 years of real world experience as a senior executive. So if you're looking for practical tips from a business pro who's been there, then this is the podcast for you. Welcome. Let's get started.

Hi, everybody, and welcome back to another episode of the biz, the business integrity school. We're in season five, and we're talking about all things tech ethics and responsible AI and all of those related topics. And we are super lucky to have with us today. And entrepreneur Lucas B. Wald Hi, Lucas, how are you? Good, Mr. Nice to see it. Nice to see you too. You all are going to love hearing Lucas's story, I have a feeling he's living the dream of many of us. So we're gonna jump right into the discussion. Lucas is the founder and CEO of not just one, but two companies he first founded and is the former CEO of a company known as figure eight. And a couple of years ago, he sold that company for $300 million to another organization known as AP and congratulations on that. Why you think he may like go off into the, you know, wonderful wall that live his life. That's not the case, he turned around and founded yet another company known as weights and biases. And that is where he is continuing to work today as the founder and CEO. So like I said, Lucas, congratulations. First of all, that's an incredible accomplishment. So how did you do this? How did you end up where you are today, and what got you kind of interested in being an entrepreneur and starting all these companies,

I always loved artificial intelligence ever since I was a little kid, I was just enamored with the idea that you could teach computers to do things and I went to Stanford, I really wanted to study it. And at that time, you know, people were like, AI is kind of all hype, it doesn't work, they talked about the AI winter, in the 80s, people really thought, wow, you know, AI is so powerful computers are going to kind of do all the things that humans can do. And they're going to replace this kind of like how people talk now. But at the time, you know, it was hard to find really good applications were AI worthwhile. And there were like a few early successes, and then the technology kind of got stuck. And so you didn't see like a lot of really meaningful applications for a while. And that's when I was in school, it was in the middle that nobody wanted to go into AI machine learning was a little bit of a, like a backwater, I don't think people would have known, you know what that meant. I mean, now, and a lot of the people, the professors were in AI, but not doing machine learning. They're doing kind of old, older style, AI, which is more kind of logic and symbolic manipulation. And, you know, machine learning was kind of coming out of more stats departments. But I, you know, I just love the idea that you could teach computers to do stuff, it's always kind of really captured my imagination. I feel like, you know, in the long term, I don't see why we won't be able to teach computers to do everything that that humans can do. And I think at that point, we enter a totally different world, maybe not in my lifetime, maybe my daughter's lifetime. At some point, I think, well, we'll figure that out. And I think it really has the opportunity to make a really amazing impact and everyone. Yeah, yeah. And I felt like when I was graduating school, I was actually really excited to just get a job, I wasn't one of those people, it's like, I have to start a company. Really proud to have like a, you know, corporate job and, you know, people cared about my, my skills, I felt really good. I noticed though, that, you know, the only companies at the time that really wanted to hire anyone doing machine learning, were companies doing either kind of Wall Street, like, you know, money optimization, which felt a little empty to me, I'm not really like opposed to it, but it didn't, it wasn't like my dream to go to Wall Street and like, you know, like, help a hedge fund get like a slightly higher return, I actually think is kind of intellectually interesting, but just, you know, for whole career when it wouldn't sustain me personally. And then, so the other thing that was happening was, you know, companies like, like Google, we're doing search optimization and try to show you better results. You know, that time they're really important. And the reason Isn't that that was a good early application of machine learning was actually there's a lot of training data created implicitly. So you know, people look up stuff. And that's actually training that you can use the clicks to teach the computers that you can use when somebody like hyperlinks to something, you know, that's actually like a signal, they're like, when you link to somebody say what it is, you sort of like, implicitly kind of defining what would be good search terms to find that piece of content. So it was fun to work on. But I really felt like the thing that was holding machine learning back at the time was a lack of train training data. And training data is like the examples that you show the computer, that the ML system to then learn to generalize it at the time, all the applications are ones where you sort of were getting the train data for free, but I felt like there was a lot of other applications where you actually have to label it yourself, right? So if you want to do voice recognition, you need someone speaking things, and then typing into the computer, what did that person say, if you want to make a self driving car, you need to like take a picture that the car is seeing and label every pixel in that image with like, what is actually the only way that computers learn to do things. And they require a lot more training data than humans. And so you see this big expense. And I just really wanted to make machine learning actually work. And I felt like, there was this kind of problem in the market that people couldn't get the training data that they wanted, I really knew nothing about how business works. But what I didn't know is that, you know, there's a problem that I wanted to solve, you know, it's a lot harder than I kind of, like going into, you know, built the pretty big company over time. Yeah, yeah. You know, collecting this data for, you know, for many companies,

yeah, the training data. So it was a problem you not only wanted to solve, it was a problem that needed to be solved, you may not have realized the magnitude of the of the needing to be sold part of that equation when you started out. But that's almost like a match made in heaven. Tell us a little bit more about figure eight, I've heard it described your first company, as a human in the loop, machine learning and AI company. In fact, by the way, just so the audience knows, it was included in Forbes list of 100 companies leading the way in AI in 2018. What does it really mean, when you say, human in the loop machine learning and AI? Well,

it's the idea that most machine learning systems in the real world, they don't work, where the machine learning always kind of makes a decision, and then people just blindly follow it. Right, most of the way machine learning really gets deployed is the computer makes a guess. And if the computer is not confident, it gets sent to a human, and getting there, right, it's actually really critical. And, you know, the training data piece is actually really interesting here, where the ones where the computer is struggling, or where it's confused, where actually asked a human to help are actually the perfect examples to feed back into the system to make it get better over time. The reality is, like, for a business, a 60% Confident process isn't gonna be that useful for most things that a business wants to do, like, you know, with, like, with Wall Street, you know, if you can pick stocks, right six for the time, you can make a lot of money or, you know, search results, we don't expect search to always give us exactly what we're thinking of, but for most like real business processes, you know, you do need really high level of accuracy, because something's gonna break, you know, but, you know, the thing about, like, 60 said, accuracy, if, you know, the 60% of cases where you're going to be accurate, then a business might see that as 60% cost savings, right? If it avoids, you know, some other process, the 60% of the time that the machine learning is able to kind of do it automatically, right? So automating 60% of process, with 100% accuracy is really useful. Automating 100% of the process, where breaks, you know, right,

right, it's 40% of the time. Right? Right. Yeah, exactly.

And so that's why, you know, it ended up being the vast majority of our customers used figure it in this way where it would build this human, the loop system, and we would help them you know, set things up so that, you know, you build an animal model, that noses confidence, and then the low conference results gate gets sent to a human who labels it, and then that gets fed back in system so it's nice to hear your process improves over time.

Yeah, yeah, it really does. And then when you finally get it up to a confidence level of, I don't know 95 96% And each company has to decide for themselves what's the risk embedded in that particular process and what you know level of accuracy do we need before we are comfortable rolling it out, let's say and then assuming the risk for the you know, small percentage The times the machine may get it wrong, which again, I think will vary. So the software, one of the pieces of software anyway that I know figure eight had included the Dash Cam application of it, and which was useful in self driving cars. So tell us a little bit about that dash cam application of software, how did that work?

Now it is a lot of cars shipped with a little bit of like automation in the driving. Yep. And the way that this typically works is there's a camera in the car. I'm like really oversimplifying, but like, one thing that happens is there's a camera in the car, and the camera is looking out at the world, like, and it's trying to decide, like, Okay, what's out there, right? I mean, so it wants to know, okay, like, where's the road? And where's the road going? It's also important to know, like, are there like, you know, humans here, because humans, you know, are different than a tree like humans will move, right? You also would much rather like crash into a tree than a human. Ideally, you don't crash into anything, right? And so, so the really the critical step, or a critical step is knowing what's in front of you from a camera. And you might think, like, wow, I mean, if you haven't thought about it, it might seem like computers can do so many amazing things. Why is this one so hard? And I think the reality is that, like, humans are actually really good at this, right? Like we our eyes have evolved for, since we were fish, right to, you know, to kind of navigate the world that we're in and a lot of our brain, what it's doing is actually like taking the photons that enter and figuring out like, what's in those, you know, what's out there in the world. And so, it's just a really hard task. Like, when you think about, like, what makes you know, like a person, a person, there's so many different ways that it could show up, there's different lighting conditions, like you could see, you know, the reflection of a person, like in a mirror, how do you know that that's like, not actually a person that you might crash into? So it's like a super subtle, difficult task. And really, what we did was make it efficient to label every pixel because, you know, this is one of those ones, where if you had to, like click on every pixel and say, Okay, this pixel is part of a person, this pixel is part of one time, yeah, it would take you forever, right? So you want to be good guests. And you want to sort of like build tools that kind of help the operator labeler. Make those labels faster. So you're doing more kind of correction of guesses that the system is making versus, you know, just guessing things from scratch, and then as you guess, a new thing, then, you know, the automated system can kind of make new guests about like, Okay, if you think that person, probably those pixels around are also part of a person, and then you kind of iterate until you get to a point where all the pixels are labeled accurately,

do you think they did enhance the safety of self driving cars, or the further automation of cars, even if a human is behind the steering wheel?

You know, self driving cars, over time will probably be safer than than human drivers? They may, you know, they're not now, obviously, but I think over time, they will be, I think, what what we really what we know, that we did is we helped make those self driving systems perform better. And you know, that actually, you know, there's, there's chance that might have made things less safe if you just willy nilly started to playing. But I do know is that our customers can a lot about safety themselves. And so right, you know, they were really eager to, you know, prove the performance kind of get over a threshold where they felt like, you know, they could use it in different scenarios. And obviously, the first thing that you do is use it to like, augment a human driver that's sitting at the wheel. This is almost like a human loop system, right? You can think of like a Tesla, where you have to touch the steering wheel and like, say that you're still looking at things. And you're, you know, the intention is that you grab the steering wheel, if it's like doing something wrong, that's a human loop system. Right? Because the autopilot is driving, right? Humans intervening when there's,

at some point, self driving cars will probably be safer than humans. I don't think that by and large, humans are there yet in terms of accepting that, probably because things like Uber car crash, and you know that the small examples, they get really blown up and magnified. And that's all people can think about. What do you think it's going to take Lucas to get over that hump of kind of human acceptance?

I think that we the bar is naturally hire for like a new kind of system, because it makes different kinds of errors, right. And so, you know, we emphasize the places where it makes mistakes, but I don't think that the human level of driving is like this, the peak that that a, an automated system can do so I think that you know, the automated systems have to be like a lot better than human systems. But the amazing thing about computers that they keep improving, right, you know, I think what can happen is it's going to keep keep getting better and better and better. And then, you know, over time people accept it. And, you know, there could be generational things like it's interesting. My, my daughter is two and she talks to our Alexa doesn't view that as like weird at all, you know? It's like, I wonder like, maybe, you know, people that grow up with this kind of automation might, you know, be more Is this more of like this more about, like humans understand less well than machine learning systems?

Yeah, you might be really right. It could be very generational, just like accepting tech generally, you know, has been. Okay, so let's turn to your current company weights and biases, I find that an interesting name for the company, how does your company weights and biases help machine learning teams build better models faster?

Well, we built developer tools that makes the developers more efficient. And you might ask, Well, what are those? I'll give you like one example of something that you might not realize, right, that you would need. So you know, when you write code, you version, the code, and you version it for a lot of different reasons, right? I mean, like you, if you don't write code, and you write like, text documents, you probably version this text documents somewhere, you save them, and you're like, Okay, this is the latest one, and this is the date and all that. And you do that, because you know, you may need to go back in time and like revert some changes that you made all those things, right. So same thing happens in software. And so people do save all the stuff that they make. And then when you get to be bigger and bigger teams, the versioning is really important, because everybody's modifying the codebase at the same time, right. And it really unwieldy if you didn't have like systems that sort of like looked at how each person was changing the codebase and sort of merging the changes back together in a sane way. Yeah. Okay. It turns out with models, right, really, the computers are like building these models, and the version control systems break and a lot of ways, right, because it's not the humans writing the code, it's computers, right. And so they the computer, like humans will like, if I make like a V to some software, I might modify a couple lines of code and change it with computers, like, every time they do it, they do it from scratch, right? And so, you know, it's like, you really have to keep track of more stuff, like you need to keep track was the training data, you know, what was happening when that model was being built, like really versioning a model, if it's just the like, output code, it's not enough, right? You just don't really know what what happened there, you want to like a whole track record of like, all the things that went on, and also, you know, people doing machine learning, they make a lot more versions, because it's not like a human has to go in and like make each new version, the computers are making the version. So you'll have like, 1000s, millions of versions, right. And so keeping track of all that is really important to doing machine learning safely. So, you know, versioning models, and, you know, versioning machine learning code is something that that we do to help teams like, you know, both work better together, right, because the two people working on it, they can see each other's doing, it often, like stores the valuable IP for companies, right? Because if like, you have a person who's building on they leave, you want to come in and share what they were doing and pick it up. And then there's a, you know, compliance issue of like, what if you put a model in a car and the car crashes? someone's like, Hey, why did the car crash? You can't really answer that unless you kept track of which model you put into a car, which might seem obvious. But if you don't systematize it, over time, you know, errors will will creep in, someone's going to forget to write down what version when it's in which car. And so we make sure that

as simple as that sounds, you're 100%, right? I mean, and that's a huge issue for companies when it comes to controlling risk, making sure they are compliant and watching out for their liability, protecting the reputation overall. I mean, if that were to happen, and it's a simple little thing, like what you just described, what version did it, you know, went into the car? And if they can't answer that, I mean, they'll lose trust with all of their customers and their customers think, well, what in the world are you doing, you're putting in a system and you don't even know which one and you can't trace it back. So it's a little, it's a little thing that may seem obvious, but it could have really big implications really big. I understand that the tools are developing and the machine learning world for those developers is actually kind of really didn't exist back to your point about machine learning was kind of an afterthought. A backwater people weren't spending time in it. Meanwhile, basic, you know, the software world was had been growing a lot during the time of the AI winter, if you will. So they had a lot of DevOps, right in that space where they were doing a lot of this, and I was reading some of your work. And in an article in TechCrunch, you mentioned that when software code fails, he said it crashes. But when ml work, machine learning work fails, it can behave badly in more subtle ways. And that really piqued my interest. And I was hoping maybe you could explain that a little bit.

Sure. I mean, I'll give you an example. Right, we work with John Deere, to help them in fields identify, which are weeds in which our crops that you want to catch that I decided to spray just the weeds with with pesticides, which is great, uses less, you know, less pesticides, it saves the farmers money, and also is better for the environment. And so you know, the way you train that data is you pull a camera over a field of lettuce that has some weeds in it, right? And you take pictures, and you have people label like okay, here's the weeds and here's the lettuce, right? And so you might think like, okay, everything's great. Yeah, right. And then, you know, one day it snows, right? You can imagine like if you just didn't have any examples of snow. You know, I think a human could probably figured out if they were waiting, right? Also human might like stop and say, hey, you know, I don't really know what's going on, I can't see the weeds here, right? You know, it's really actually hard for machine learning systems to adapt as well to new situations like that. And so what can happen is that, you know, in the snowy condition that was never in your training data, your ml system, rather than like stopping and not doing anything, you might decide to spray all the lettuce with pesticides. And that'd be really bad. That'd be like, heartbreaking, or you completely destroy the field. And that's like the fundamental danger here, right? Because you can never collect examples of every possible edge case that you might encounter. Right. So yeah, I mean, that's, I think that's a fundamental challenge. It's deploying, you know, machine learning systems today is such

an interesting point, while machines are able to do things faster, and we can train them to be better. Everything that a human brain that we have learned over the years are things that machines don't know if you don't teach it, right. So now, now, you're saying they can behave badly, like ruin entire crop of lettuce, which nobody would want. And so you have to think about probably the most likely scenarios that you know, your tool is going to encounter and make sure you train for those. And then also think about maybe some of those black swan events, if you will, kind of the upper left of a of a risk heat map. And for sure, all the ones that are going to be upper right, you know, have great likelihood and great impact. So yeah, that's, that's, that's very interesting. So let's switch topics entirely and talk a little bit a little bit about diversity and equity and inclusion in the AI and kind of ml industry and tech in general. You know, no surprise, I think there's been a lot in the press about historically, tech companies not really being seen as a place where women can thrive, particularly if they're in the engineering, so they're in a STEM field. And you've been a two time founder, so I'm really interested in knowing what your view of that is, and whether and kind of what you've done to try to make a difference. Being aware of that, that at least impression is out there?

Well, I always feel a little shy, like answering this question, because my lived experiences as a white man, you know, went to Stanford, right? So obviously, there's a, you know, serious inclusion issues in tech that, you know, I really want to help with, you know, one simple thing that's like, actually, a real challenge is I think, if you don't get diversity in your first couple hires, it can be really challenging. You know, I've talked to women engineers that don't want to be the first woman engineer on a team. So I think like, when you get some of that's like, we want to do that, you know, like, I appreciate it, you know, make sure that they're, you know, comfortable and happy. And, you know, make sure that you're, you know, the stuff that you do as a company feels inclusive. And I think there's a lot of people that feel like outsiders, and a Silicon Valley Tech company. And it's a challenging thing. We do a lot of trying to survey employees and trying to be responsive to what employees are asking for,

I have to say, I think despite the fact that you are obviously white, and you're a male and you went to Stanford, you clearly have an awareness and an appreciation of the issue. You've used your position to think strategically about how do you get more women in you recognize that if you don't do it in your first couple hires, you could have an issue, right? And that's a lot of where it starts, right. And being a good ally, being a good advocate, being the kind of leader that that people would say is inclusive, so yeah, I don't think you need to be nervous. You actually have a lot to share in that space. Are there any particular female STEM leaders that you admire? Oh, yeah,

well, my old advisor, Daphne Koehler, I just like admire diva. She's kind of a famous person. Now. I mean, she, you know, she won a MacArthur and she runs a company called in Citro, and she started a company called Coursera. That's done, you know, yeah. You know, Emily, Bender is another person that comes to mind, you know, who's who's kind of thought a lot about like AI and ethics. And so those

are great suggestions. I think people can go women in particular, if they're interested, you can go look up your women that you admire and learn a little bit more about them. And perhaps that might inspire them to go into the field as well.

Yeah, I mean, I would say even men could stand to learn from these people. They're

absolutely, yeah, yeah, they can serve as role models, really, for everyone. I always like to leave the audience with some additional resources. If they want to learn a little bit more about this topic, or, you know, watch or read or listen to something else and enhance their education even further. I want to know where you turn to for your best additional learning. So you have any recommendations for the audience on what else they might listen to, or watch or read to learn a little bit more about this?

Yeah, totally. Well, I mean, I don't want to show for my own stuff. We actually do do an interview series of people in the space including the two people that I that I mentioned. So if you kind of wanted to watch in depth interviews of, you know, people in machine learning, we have an interview series called gradient descent. That's super fun. I think one place that I was suggesting to people kind of outside of machine learning that I think is kind of an interesting place to learn about what's going on is, there's a podcast by a guy named Lex Friedman, who interviews a lot of really high profile people in the space and it's it's like super interesting. He really gets kind of like deep and philosophical. And it's pretty

cool. Wow, how interesting.

Yeah, yeah. So I guess I guess those would be my sort of, like, entry points. And there's, there's plenty of my favorite thing would be if somebody like kind of wants to learn, you know, how to do machine learning, and there's just so many free resources. I think the best one I would say is fast AI. So if one person is to this and they go too fast that AI and and take the course that would make me really happy,

okay, and there's like free courses on there that you can

free and it's like, yeah, by a really smart guy. Really accessible. Yeah, I really recommend it.

Very cool. All right. Well, we're gonna leave it there. And I'm sure that at least one person will go to fast.ai. And we'll make sure to capture all of your recommendations in the show notes so folks can go deeper. Lucas, thank you so much for your time for sharing with us your background, your experience and what it is that you're doing in this fast evolving space. Really appreciate it.

Thank you so much. Okay.