Host Mike Graen sits down with Sarjoun Skaff - CEO of Bossa Nova Robotics - to disucss the story of how the Bossa Nova Shelf Scanning Robot came to be, along with topics including:

• How the need to their robots was identified.
• The trials and errors it took before a finished product.
• Additional benefits robot tech can provide.

Good morning. Good afternoon. Good evening,

everybody. My name is Mike Graen and welcome back to another

conversations are retail specifically focusing on on

shelf availability. I am really excited to talk about something

that we've talked about on this channel multiple times, which is

the role that robots play in retail. And I am really excited

to have Sarjoun. Sarjoun, I'm gonna have you introduce

yourself to the audience. Let them know a little bit about

your background. And then we've got lots of questions for you.

So go ahead and take it away and introduce yourself for us.

Mike, thank you very much for inviting me. I'm

very excited. You and I go a long way. We've had this

conversation off-line for so long. And then you told me how

about we have it online, so we're gonna take a risk and dive

in. I am the founder and CEO of Bossa Nova. I founded it in

2005, when I was still doing my PhD in robotics at Carnegie

Mellon University, I guess our claim to fame is that we

deployed robots in 600 WalMart Supercenters to help them manage

their on shelf inventory.

Wow, you got a PhD in robotics, there is such a

thing?

Hey, CMU was the first program to offer it

actually.

Really? Really. Now I see all of these sixth graders

out building robots and fighting in little cages. You kind of

started a trend, right?

Robots have always made kids dream. Big kids

too.

Awesome. Well, tell us a little bit about yourself.

And I'm gonna put a little thing in there. You've got lots of

friends and acquaintances, you and I've been working together

for lots of years. But tell us a little bit about yourself

personally, professionally. And maybe if there's a little

soundbite that most people don't know about you. If you're

comfortable sharing, please, please do that.

Then how about sound bites. So I'm born and

raised in Lebanon, I came to the US at 22 for grad school at

Carnegie Mellon, and could not wait to finish my studies to

start a company. I always wanted to be in the startup world.

Bossa Nova was my first and only startup. It lasted for 18 years.

Wow. And it was amazing. We did actually a first we had multiple

half lives, if, if I may say. And the first one was building

robot toys. I lived in China for three years at the factory to

learn how to make consumer products. And then we

quote-end-quote, graduated our toys from children to big boys

and built those inventory scanning robots starting in

2013. And, and very quickly started working with Walmart and

the development. Grew, that company Bossa Nova over to 230

people across, across which we had raised $120 million. So it

we actually lived... . It was it was great while it lasted.

That's awesome. Well, here's what I'm gonna ask

you to do. I need you to take off you're really smart guy hat

for a second because you have that a hat on right now you can

tell us all about how the robots worked and whether but walk me

back and just talk about you not as a solution provider to the

industry, but talk to me as a customer. Okay, because to me,

one of the things that are really excited about this is the

focus of this channel is really on shelf availability. And

certainly robotics are one of the very strong powerful tools

that help that. But tell me as a customer, if you got to

experience over the last bout six months or so where you went

in to get something and you really needed it and you got

disappointed by the retail we don't need to reach hours name.

We don't want to throw in mud but it happens all the time.

Right? Give us an example of as a customer, you were

disappointed by a retailer not having something on the shelf.

I have to admit that I wasn't as sensitive to

that topic until I started working in it as as a supplier.

But once I did, I couldn't unsee it. The of course in the

pandemic we've all had the horror stories of buying The

shelves are empty, or months on end. But there are a couple of

couple of chains in, in, in where we live, where we actually

stopped going to it because we could never fill the court.

Every time we went to the shopping with a shopping list,

we weren't sure that we're going to come back with a couple of

items missing. And we simply switched switch chains, we

stopped going there, it was out of control. We felt they didn't

have their inventory under control.

Hmm. Interesting. Interesting. Yeah, it's it's,

unfortunately, a very common occurrence. And obviously,

during the global pandemic, you know, all things were upside

down. But even today, you'll go into a normal store, saying, I

know I want this product and you go to that product. And it's

like, they don't have it, then things. If it's a box of cereal,

you can probably very sweet quickly switch and go, I don't

want to get this other cereal, but I will. But as I've talked

about, I'm going in for the black printer cartridge for my

printer, and they have all these printer cartridges, but I don't

have the one for my printer. I can't switch. So we talk about

actually firing the retailer, which is what you did, you fired

the retail business where you chose to spend your money

somewhere else. And both Sam Walton, the founder of Walmart

in the current CEO, Doug McMillion, and says, you know,

at the end of the day, customers decide where they spend their

money, they can hire us or fire us every day based on what their

satisfaction is. So, that's a great example.

And we've been spoiled. We've I think become a

little bit entitled we want any product that we want any way

that we want at any time that we want. So our tolerance for lack

of operational excellence is now thin.

Yep, very true. Well, I thought we would start

this thing by with with something that will help set the

stage of what this is. And several years ago when we had

Bossa Nova over rollouts in Walmart, Walmart actually

created a video that described what this particular robot was

doing in store. So we're gonna play that real quick, and then

we're gonna come back and ask some questions about it. Okay.

When it comes to retail technology, we're pushing

the boundaries of what is possible through robotics and

AI. That's why we're excited about new shelf scanning

technology. We're testing a small number of stores across

the US. It simplifies routine work by using data and vision

technology to find items that are out of stock, missed price

changes, and missing labels. This helps our associates see

where they can make the biggest difference for our customers. It

also helps locate items and checks inventory levels on the

shelf, broadening the assortment, available to online

grocery pickup customers, and assisting our personal shoppers

and filling orders. At Walmart, we're using technology to focus

on tasks that are repeatable, predictable and manual freeing

up associates to spend more time doing what they do best: serving

you. When we combine the passion of our people with the power of

technology, the possibilities are endless

Fascinating, Sarjoun, fascinating. So you are

the architect of the whole thing. That's exciting. Tell us

a little bit about Bossa Nova were to come up with the idea;

you said you just kind of came up with it a little bit at

school. How did you get involved with you know, a retailer like

Walmart, just you know, any kind of backstory would be really

interesting.

We we were the proverbial hammer looking for a

nail we we were technologists and we had a cool tech and we

went looking for a solution which is exactly the opposite of

what startup founders should do. We lucked out at work this time.

We I have some videos I can show I think that will make things

clear and fun. So first, I'm gonna show you the technology

with which it's a bit of chronological and show you the

technology with which we started.

So what you just saw is this bowl-bot invented by Ralph

Hauless at Carnegie Mellon University; a robot that

balances on a ball, quite improbable, but it doesn't fall.

You saw it in motion, it is balancing on a single point,

that single point of contact with with the ground. It is it

remains upright. It is thin and tall. So it's well suited for

human environments. And strong with the elegance of this

motion, we actually went to different industries, and tried

to see what applications it could do. One of those is

retail. And so here's a picture of me at the Intel booth at the

NRF in January of 2013. And I owe a debt of gratitude to

Intel, because they hosted our new version of the bowl-bot and

retailers were lining up the booth out of the booth, they

were all asking the same questions. Can you take pictures

of my products on the shelf? And then tell me when a product is

out of stock? And our naive selves said yes, of course, that

should not be too difficult. Give us a nine to 12 months, and

we'll probably be have have a finished solution. And of

course, it took five years instead.

Of course, always takes longer. Always takes

longer. So it was just fascinating. So the fascinating

thing is, I still don't think that I've seen a robot anywhere.

I mean, as I think about just robots for cleaning your floors,

robots for scanning shells, etc. I've never seen a robot dance on

a single ball, that obviously got replaced by other

capability, which is much more stable, I guess what was it?

So the elegance of the robot attracted the early

attention, which was, which got us started. So it was great, but

it was too complex to manufacture and product dies. In

particular, if the power went away, this thing fell. And we

tried to be ingenious with like safety legs that popped out

impromptu with springs instead of power and all of that. It's

it just was to, it became too complex. And we started to wise

to become wiser about the complexities of the rest of the

application enough business development. At which point we

let's just remove the complexity that's unnecessary

and focus on this more stable platform with four wheels.

Gotcha. Gotcha. But was interesting, what you said

is the retailers weren't thinking about how cool it was

because it was dancing on a single ball. You could be you

probably thought that was the cool part. What they were going

is, "man got a lot of people right now that go down my

shelves all the time and scan for empty shells. Can you

automate a robot to do that for me" that that was the big

takeaway. You are coming in with "isn't this tech cool." They

were coming with you "if I got a problem can it do this for me?"

Is that that kind of the way it went?

That's exactly right. That's exactly right. And

we thought putting a camera on the robot was was simple. And

therefore, naively thought we could do that in a short amount

of time. It turns out that, well as you peel that onion, it's a

very complex problem to solve.

So let's double click on that one a little bit,

we'll get a little bit techie. Okay, so number one, if I think

about a normal shelf, you'll see it's anywhere between probably

six to if you look at top shelf kind of eight feet tall, so

they're fairly tall, right? And I'm assuming what are you trying

to do with this robot that made it so complicated? I'm assuming

it's not just slapping a single camera on it and calling it

good, you got other stuff, lighting and things like that

you've got to consider as well, right?

Yeah, and if a single camera could work, even a

high resolution camera and like a DSLR, we would have done that,

we tried, this is how it started. The the issue is, is is

one you need the resolution to read the barcode

on the price tag. That resolution requires having a

zoomed in camera and therefore your field of view is narrow, so

you need more than one camera. Also, you want to see the back

of the shop because products, shelves may look empty. If a

product is recessed yet it's actually not empty. If you don't

look at eye level for each shelf, you don't see the back of

it and you may miss recessed products. And therefore, we

ended up configuring an optical stack a column of cameras all at

the shelf level and zoomed in so they could see the back of the

shelf and read the barcode on the price tag.

So these these things as we go through that, we

saw the video of an actual robot and in the store, literally, it

looked like there was a bank of lights that turned on as well,

is slowly it wasn't going real slow, but it was not going real

fast, either. You're saying it's able to literally stitch

together the price label, the product, the depth of the shelf,

and some of those kinds of things for the retailer, right?

Yeah, absolutely. And I'll show you, I'll show you

those, those pictures as well. The reason you have a light is

that is to avoid motion blur. So by flooding the scene with

light, you're able to shutter very quickly your cameras and

therefore even though you're moving the image turns out to be

crisp. And And here's an image of a single one of those 10

cameras that what am I saying 15 cameras that we have on the

robot, you can see that you can zoom in and read the barcode on

the price tag. But then you end up with 4000 images per aisle.

And that is multiple gigabytes that are difficult to manage. So

we actually had to develop additional technology to

compress those single images and then stitch them into I long

panoramas. And here's a picture of what that looks like.

Got it. So let me make sure I understand - and

make sure that if I, if I understand for the audience,

this thing is going down the aisle. And it is literally

scanning the shelf. It's scanning the products. It's

scanning the label, and it's scanning what's behind the

products, if it can see it. And what you're able to do is a

bunch of different things, which is if I have a label, without a

product, that would be an out-of-stock, right? If I have a

label with a product, but it doesn't look like it's the right

product, you're gonna work on the background, product

recognition, say that says Tide on the label. But that's a Gain

product there. That's it. We call that a plug. When I was at

Walmart, it was already incorrect product. Some people

said. Assuming it can compare, what's the shelf at the register

versus what's the cost or the price at the shelf at the same

time? Correct? Those are some of the pretty cool things. And you

know, if there's 155,000 items in a Walmart store, he does that

every single day without calling in sick without getting tired,

without getting interrupted, etc. That's just the beauty of

putting robots to storage. They do very autonomous tasks, they

do it very well. They don't get tired, they do it accurately and

all that kind of stuff, right?

That's exactly right. You design it well. And

then once and then it works every time.

So the here's the big question, what's the status

of it now?

Um, it's gone. Unfortunately, we could not

survive the pandemic. And they we pulled the robots out.

Gotcha, gotcha. So the pandemic killed a lot of

stuff beyond just ... scary every step for two years.

It's a little counterintuitive. But we got

out-competed by the crowd, by people. In the pandemic, there

was a spike in online. And in e-commerce in online sales,

those orders were fulfilled from store shelves, right. And our

customer orders essentially hired an army of pickers, people

whose job is to find your order on the shelf. Well, by doing so,

they were also reporting the absence when they don't find the

item out of stock by scanning the price tag just as our robots

were doing. And so they were additional signals that

retailers could use and that made the marginal contribution

of of the robots.

Yeah, not not to debate anybody's decision, I

guess. Here's a couple of push-backs. Replacing robots

with people is not counterintuitive. It usually

goes the other way around. Right?

Anachronistic not counterintuitive, it's

anachronistic.

I don't know what that means. I'll have to look

that up later. But okay.

The rest of the industry replaces people with

robots. In this case, the opposite happened.

Well, here's the other thing and again, not to be

controversial, but I'm gonna throw it out there. So what

you're saying is rather than proactively monitoring my

shelves for every on shelf availability issue which

disappoints a customer or picker either one; instead, I'm going

to let a customer order the product. And then somebody try

and pick the product as the method that tells me I am going

to disappoint you. Going back to that example, right? It doesn't

seem, if we're really customer focused, I would think you would

want a proactive tool that says, Here are the disappointment

factors right now. Because here's the other thing is I

don't think customers order every product in the store,

right? They maybe order, let's say, the top 20 or 30%. I don't

know what the numbers are. But what about those other 70% that

are maybe not picked very often, but unfortunately, they're gonna

be on shelf availability problems, and we won't know

about it, right. So it's, we'll get into diagnosing that don't

want to rehash it and debate it, etc. What you built was

something that makes a lot of sense for the industry. It

really, really does. And obviously, there are other

people out there who have replicated your success with

other retailers, and we're seeing some pretty great

results. So it may not be the tool for every retailer. But

it's certainly for some of the retailers who want to

proactively make sure their products is on shelf. And we've

talked about this before, I don't want to go into that. But

some people use algorithms, hey, I'm selling 12 a day. And

suddenly I sell zero, that's a pretty good signal that says I

have an on shelf availability problem, a lot of your stuff:

spices, things that are in health and beauty that may only

sell one, one or two a month, I can't tell if it's not on the

shelf, or it's just not someone just doesn't have the velocity.

So tools like yours, I think are ideally suited to be able to

proactively do that. In addition, there was some stuff

we were doing early on, to be able to capture other signals

other than just computer vision. Right? Is that was that the

vision for where you want to take this?

We were pretty... Yeah, so we wanted to, there's

so much more that you can do with a sensor platform that's

autonomous. So computer vision was one, 3D data was was the

second. And it turns out that measuring the state of the

store, as built is very valuable. Measuring the depth of

the shelf for shelf capacity allows you to adjust what you

push the quantity that you push from this from the distribution

center, knowing that the depth of a shelf changes over time

because the shelves themselves get swapped. Doing compliance

checks on whether you're ADA compliant, for example, you can

do that with with 3D data. Figuring out what is your space

management, meaning what fixtures to replace, what's the

state of your fixtures - are they damaged do you need to

replace them? And if so, what what did what are the sizes that

fit. So you would know for instance, that buyers will build

a planogram of where the product should go, where which product

go, which product goes where on a shelf, essentially a map of

what the shelf looks like. Assuming that it is a four foot

shelf. In some stores, we may have a column in front of the 4

foot shelf so it's it doesn't always work. So knowing what are

the actual dimensions in a store will allow you to be better

targeted and refined in your in the execution of your planograms

Another sensor you can add is one that I know is dear to your

heart, which is RFID. Once that, you're mobile you can start to

do you know inventory products, locate them as well. And you

don't need to either manually want them or make or have a

heavy installation that installs ceiling transducer and ceiling

RFID sensors for example.

Yeah, we actually we actually you'll laugh at this

she probably won't even know I have this but we were in

Pittsburgh in your office and we were in the very early stages of

looking at a Bossa Nova robot in the store for CV and I think I

sent you a quick email saying going to do RFID as well just as

a joke just to see what you would say. And literally I'm

going to show you a video here's the video right here. This is

actually, it's just a stripped down Bossa Nova robot, so it's

just the actual robots and all that pretty plastic. But you're

seeing it actually read RFID tags and that was a vision and

by the way fast forward to now 2024 where Walmart's basically

saying for 60% of its merchandise - which is all

general merchandise - we want an RFID tag. It was capability that

was before its time. Now, today people are wanting stuff and

spending labor with people collecting that. There are other

automated ways such as this robot to be able to do this,

which was pretty cool. So you were just way ahead of your time

Yeah. Look, fundamentally, I'm I continue to

as usual.

be convinced that a mobile sensor platform is a very cost

effective way of capturing data. No way around it. It's just cost

effective. And the as AI is progressing, the ideas keep,

keep growing on what you can do with data. Now, it's fashionable

to talk about digital twins, for example, you can absolutely do

that. It was one of the early products that we built,

actually, we call it Space View, you could do virtual

walkthroughs. And in a world of images.

Right. That's right. So so let me ask you a question.

This is a little bit separate. But you recently it was pretty

interesting on LinkedIn, you went pretty public with, for

example, your original funding strategy. It felt like you're

almost trying to help some of the new entrepreneurs out there

with approaches, etc. But you specifically landed on something

called RAAS versus SAAS. So robot as a service, or software

as a service, etc. You had some pretty strong feelings on that,

I think the audience is interested in your thoughts in

that area.

Yeah, firsthand. These are first first hand pain

points that we experienced. We, we started by offering our

robots as a service. We thought the opex way to the market is

the lowest friction way. And therefore that's what we should

do. And it was successful at first, it's, it's helpful to

overcome initial skepticism, skepticism, if you don't have a

track record, which was our case. And people don't want to

take too deep of a plunge at first and, and get financial

exposure. So they may essentially rent it. We did

that, it worked. And then as we started to scale, the more

robots we had to deploy, the more capital we needed. And

where does the capital come from? There are a few ways where

you can do that. One way is if you have a very strong contract,

you can take it to a bank, and they get an advance against

that. Or you have to raise it from investors and, and we

raised a lot of it, that's why we had to raise so much money

from investors. But that's not a good way of that's not a good

use of of capital. And it's expensive. The cost of capital

is high for a startup, right. And so we actually changed

models, we figured out what would be a more attractive

model. No surprise, our customers being large. And this

is what I'm saying is not necessarily universal, mid

market customers may or may not have the wherewithal to actually

make big capital acquisitions. But our customers were

multinationals, they definitely had the capital, their balance

sheet was extremely strong. And they were actually winning and

used to buying capital. That's what that's how they acquire

checkout counters. So it is a model they understand well, and

they were really open at about buying our robots, because the

other benefit of it is that if if Bossa Nova disappears for any

reason, they get to keep it so business continuity was

important. They were so that was the case. But for a startup

getting recurring revenue is very attractive from a valuation

perspective. If you look at the stock market, software,

companies get 10x value valuation, get the valuation

that's 10x their revenue. And that is a benchmark against

which you could value your company when you're when you're

raising your raising money for venture capital. So getting

recurring revenue is important. And we figured out a way to tie

it to the software. And we had two sources of of software

revenue. One was the software running on our robots. So even

though the customer was buying the robots, we continue to get

recurring revenue from it because we had the software that

we continue to develop. So to support the development, we

needed that recurring revenue. And the other was the AI in the

cloud that we were offering as a service. The robots were

capturing data that was being analyzed on our servers, and

therefore that was a second revenue of a second recurring

revenue, and that was another SAAS so we shifted the business

model from RAAS to SAAS. And the financials look beautiful. It's

It's a sea change And they look familiar to venture capitalists.

This is a SAAS business model. We pushed it to the point where

the robots no longer touched our financials. They didn't appear

on our balance sheet, the customer could buy them from the

contract manufacturer. And therefore that negotiation

happened on the customer's credit rating, not ours, so they

got better terms as well.