In this insightful episode of the E-commerce Evolution Podcast, host Brett Curry sits down with Tom Leonard (https://www.linkedin.com/in/thomasbleonard), a fractional marketing leader who specializes in operationalizing Media Mix Modeling and incrementality testing. They dive deep into the often confusing world of marketing measurement. Tom and Brett will debunk myths about attribution and we reveal what truly drives customer acquisition. 

For ecommerce brands struggling to understand where their marketing dollars are actually working, this conversation offers practical insights on how to move beyond misleading platform metrics.

—

Sponsored by OMG Commerce - go to (https://www.omgcommerce.com/contact) and request your FREE strategy session today!

—

Chapters: 

(00:00) Introducing Tom & Marketing Measurement

(06:30) Understanding Multi-Touch Attribution (MTA)

(12:22) The Case for Incrementality Testing

(22:20) Exploring Media Mix Modeling (MMM)

(27:30) Navigating Budget Cuts and Marketing Spend

(32:17) Understanding Incrementality Vs. Attribution

(35:45) The Importance of Cost Per Incremental

(40:16) How to Get Started with MMM

(44:09) Final Thoughts

—

Connect With Brett: 

• LinkedIn: https://www.linkedin.com/in/thebrettcurry/ 
• YouTube: https://www.youtube.com/@omgcommerce 
• Website: https://www.omgcommerce.com/ 

Relevant Links:

• Tom’s LinkedIn: https://www.linkedin.com/in/thomasbleonard

__

Past guests on eCommerce Evolution include Ezra Firestone, Steve Chou, Drew Sanocki, Jacques Spitzer, Jeremy Horowitz, Ryan Moran, Sean Frank, Andrew Youderian, Ryan McKenzie, Joseph Wilkins, Cody Wittick, Miki Agrawal, Justin Brooke, Nish Samantray, Kurt Elster, John Parkes, Chris Mercer, Rabah Rahil, Bear Handlon, Trevor Crump, Frederick Vallaeys, Preston Rutherford, Anthony Mink, Bill D’Allessandro, Bryan Porter and more

How much is media contributing relative

to customer base is a really nice place

to start.

And the benefit of running

incrementality and media mix modeling is

informing the model with

some of that causal data.

Well, hello and welcome to another edition

of the E-Commerce Evolution podcast.

I'm your host, Brett

Curry, CEO of OMG Commerce.

And today we have got

a doozy of an episode.

We're talking about the three

horsemen of measuring your

marketing effectiveness. We're

talking MTAs Multitouch attribution.

We're talking M'S. Media mixed

modeling. We're talking incrementality.

It's going to be nerdy,

but I also promise you it's going to

be practical and it will make you more

money. And so we'll hopefully

make it fun as well.

And so my guest today is Tom Leonard.

We are LinkedIn friends first.

So I saw Tom on LinkedIn posting about

incrementality, talking about MMM,

throwing shade on certain tools and stuff

like that on LinkedIn. And I'm like,

this is my type of guy. So I reached

out, we had a call, and then we're like,

Hey, we got to record a podcast.

Let's create some insights

for people on the pod.

And so Tom is a fractional

marketing leader.

He's operationalizing MMM

and incrementality testing,

and I'm delighted that he's my guest

today. So Tom, with that intro,

how's it going? And welcome to the show.

Good. Thanks for having me, Brent.

Excited to be here. And yeah,

some of my favorite things to talk

through, so excited to do it. Good stuff.

It's good stuff, man. So briefly,

before we dive into the

meat of the content here,

what's your background and

how did you become a guy who's

operationalizing MMS and incrementality?

Yeah. And what does that even mean?

That's a good point.

For sure. Yeah, totally. Yeah.

So spent most of my career thus far on

the agency side at performance agencies.

And I'd say the crux of

how I got to where I'm now,

or I've been reflecting back a little

bit more on the why I have such a passion

for measurement. And I was at

a pretty hardcore DR agency,

and it was right shortly after TRUBY

for Action came out when YouTube was

starting to invest in, DR.

Moved into a new role we had created

with a centralized group of basically

people who had different areas of subject

matter expertise and a few analysts

that ran tests across a

pretty large client base.

And I was our YouTube SME,

and worked with a couple

analysts to run a bunch of tests.

And really it was to evangelize how to,

and is YouTube a platform to drive growth?

And it was really interesting

because I started spending a lot of time

on YouTube and then also connect to TV

and broader programmatic video.

And it was this really interesting,

for me, the biggest learning was less

about how to make YouTube as effective as

possible,

but more how to help brands think about

demand creation as opposed to just

demand capture. And frankly,

the difficulty of getting brands

to leverage YouTube relative

to connected tv,

because YouTube sat so close to Google

ads and therefore last click attribution

and see tv, you couldn't click

and was sexier in a deck.

And it was just this sort

of recognition of the

irrational kind of human behavior just

in any sort of industry or any thing

in life.

But it sort of helped frame up this

idea of you really have to do more than

just, I don't know,

represent logic or rational arguments.

You really have to also

bring the easy to understand

clear data. And that's,

I think what draws me to incrementality

testing specifically and why

that's sort of the backbone

of a lot of what I do now.

And I think I use the word

operationalizing, NMM and

incrementality testing.

And really what I mean by that is a lot

of people will run medium mix models or

run incrementality tests,

but oftentimes they'll sit in a slide

or in a report to be shown once,

but never to be looked at again.

And so what I'm really trying to do

with brands now is how do you build a

framework and a repeatable methodology

to get insights from tests,

but not just leave them as

insights but to take action?

Because the only way that you create

value from any of these sort of testing

methodologies and measurement

methodologies is by

acting on the insights.

And so that's sort of what I mean by my

funky little headline of those words.

Yeah, it's so good, man.

And it's one of those things where data

really doesn't matter if you don't take

the right actions from it.

And what's so interesting,

and our paths are similar in that

I got my start in actually TV and

radio and doing traditional media, and

then I got into SEO and paid search,

but I loved video. Video was my

thing, but I love paid search as well.

And then when TrueView and TrueView

for Action came out, I was like, whoa,

these are all my world's colliding.

This is.

Video and there's some search components,

at least some search intent involved

there. And it's direct response.

I've always been a direct response guy.

I believe that marketing

should drive an outcome, right?

Advertising should drive

a measurable outcome,

and that should be measured in terms

of new customers and profitable new

customer acquisition. And

what's really interesting, Tom,

and I think this kind of feeds into

the conversation we're having today.

There was a period of time, so I

grew up reading some of the classics.

So David Ogilvy of course, but John

Cap's tested advertising methods,

Claude Hopkins Scientific Advertising.

And they would do things like they would

run and add in a newspaper or magazine

and people would clip a

coupon and bring it in,

or they would call a certain number and

they would track it and they would have

codes and stuff.

And I remember thinking once I got

into e-commerce, I was like, oh man,

we've got so many tools. The world is

so clear now we have every piece of

data at our disposal.

And now the more I've gotten into it

and the more I've matured, I'm like,

we've got more data. But I don't

know that we've got more insights,

and I don't know that we've

got any more clarity. In fact,

there's maybe more confusion.

And I think it goes back to

what you said a minute ago,

this idea of demand generation

versus demand capture.

We're really good at measuring channels

and campaigns that are demand capture,

meaning they're capturing

demand that's already out there.

That's harder to measure

the demand generation,

which is usually where the magic happens.

And so super excited to dive in here.

I think what might be useful

is let's talk about what

are these kind of three horsemen that

I laid out there, MTAs, multitouch,

attribution, and incrementality.

So let's start with MTAs first.

So Multitouch attribution tools,

what are they and what

is your take on them?

Yeah, big question. Great

question. Yeah, I mean,

MTA been around for a while,

different flavors and ways

of trying to make it work,

especially as so much has changed

in privacy and the tech and tracking

landscape.

But ultimately the goal is to try

to give fractional credit to all the

touchpoints along a customer journey with

a recognition that the last touchpoint

click or last impression is

ultimately not what drove that person

to purchase.

That may be the last or the only thing

that you might see in something like

Google Analytics or your analytics suite.

But there's this general recognition

that that is not what drove the purchase.

So MTA, the kind of promise, which I

ultimately think is a failed promise,

is whether all the different touch

touchpoint and then how can you

value those differently. So

maybe you use first touch,

maybe you use even distribution. The

idea of data-driven attribution was the

holy rail or the Promise many years ago,

and I guess still to a

degree for some is like,

how do you know this channel was more

additive or more necessary and therefore

should get more credit than that channel?

Which I think makes a

ton of sense in promise.

I think in reality it's really hard

and I would argue impossible to do,

especially as a lot of the ability to

track users at a one-to-one level degrades

generally my perspective,

I'm very bearish on MTA,

so that'll probably come

through pretty strongly.

But I guess I don't think the toothpaste

is going back in the tube in terms of

the ability to track a customer across

all these different touchpoints,

especially as the ability to

track through or impression based

touchpoint erodes. And then you

really get reliant on clicks,

which I think then leads to a lot of

all the issues that just last click in

general has.

So I think it's really hard to

make a compelling case for MTA.

I've seen too many brands,

especially trying to

build MTA tools internally

and just be a huge time and resource

suck. And then when you ask to compare,

show the multi-touch view versus

last click, it's like, I don't know,

80 or 90% only had one touch

point anyways, that's all

that MTA model could see.

So is it really that much

more useful than last click?

It's sort of multi-touch when that can

be measured, but usually it can't be.

Yeah, and It never really answers

the causality question either,

which we'll get to when we

talk about incrementality.

And I always kind of tell this,

I think the short story of why MT A

isn't really viable anymore as all the

tracking and privacy changes.

But I think the slightly longer story

is the kind of recognition that just

because an ad was shown or a

click occurred doesn't mean that

that medium was needed or

that channel was needed.

It doesn't answer the causal question,

what would've happened

without this ad running?

Did somebody just happen to use multiple

touchpoints as navigation or was it

more convenient to click on one of

these ads that happened to be served?

But if you're not comparing that to some

sort of control group to really hard

to assign causality to the fact

that there just was a touchpoint.

Yeah, it is so good. And it's one of

those things where I remember again,

early on,

you would look inside of Google ads or

you look inside of Meta or was back when

it was Facebook only, and you

were like, the data's here.

I see row ads and I see clicks and

I see performance and all that.

Then you realize, well, wait a

minute, this isn't fully accurate.

If I add the two together,

that's double my total revenue,

so I can't just rely on

what's in the platform.

And that got worse as I was 14 was

introduced and other privacy changes were

made. But then MTA came

along and it's like, oh,

finally we're going to get to see the

full picture. It's going to decipher,

decode the shopping journey,

and we're going to finally see with a

keen eye in perfection exactly what caused

this ad or what caused this purchase

to happen. And then we finally realized

MTA is maybe just a third

option. It's like, okay,

Google's imperfect, Meta's

data's imperfect, and then mt A,

it's just imperfect too.

So now we just got three imperfect

things to look at and make

decisions from.

And in some ways it leads to more

confusion than it leads to clarity.

And now I don't want to wholesale discard

MTAs because I do believe there's some

helpful insights that can be gained

there,

but it's incomplete

and incomplete at best.

And one of the best analogies I've heard,

and this actually comes from Ben Ter,

who's also a LinkedIn friend,

but I met him in person as well,

but he talks about this analogy of, Hey,

if we're trying to measure what

caused people to watch this

movie at our movie theater,

and we look at all these

results and 30% say they saw a

billboard for our movies,

20% say they saw a TV ad,

but you know what? A hundred percent

say they saw the poster on the

door. So we're like,

let's just cut everything.

Let's just do the poster at the door

and that's it. And you're like, well,

wait a minute. Everybody saw it.

Everybody was walking in the door.

But the movie poster is not

what caused someone to purchase.

It was the billboard and the TV

and some of the other things,

word of mouth and other things

that caused them to come in.

And so this idea of causality,

super, super valuable.

So that really leads us to incrementality.

So talk about incrementality.

What is it and why are you on

a quest to operationalize it?

Yeah, it's really the best way,

if not the only way to

establish that a causal

portion that we've been talking about.

It has a distinct control group,

so it has a counterfactual,

it has what would've happened

without this intervention,

whatever that intervention is.

And there's a handful of ways to derive

that counterfactual that control.

The most common would be geographic

based. So like a match market test.

I've got this market over here that

historically has behaved similarly to this

market over here. I can

see that in an AA test,

the lines sort of move similar

to one another. They're not,

if they're influenced by outside

factors, they're influenced.

In what's an AA test for

those who don't know.

Before an intervention happens.

So just over time are those lines

essentially moving together?

Are external factors or stimuli equally

impacting both sides of that test

so that you can feel confident that

when you do intervene and it becomes

comparing A to B,

the delta is what was a

result of that intervention.

So oftentimes it's my Atlanta

and I don't know Memphis,

maybe some other midsize city that

you've done this market matching for.

Historically, they both

look like this on a line,

all of a sudden you turn off

ads on Facebook in Atlanta,

what happens to your top line that

Delta is what was attributed or

should be attributed to

advertising in Atlanta.

Whereas the flip side of that would be

attribution would say basically anything

that was attributed to that could

be attributed to that would really,

it should just be the gap between a

world where that ad does not exist

compared to a world where that ad

does exist. We can't take credit for

everything.

We can only take credit for as much

above and beyond what would've happened

anyways. And so that's the

basis of incrementality testing.

There's other ways to do it.

If you use a Facebook or Google

conversion lift study because they own

that auction or anybody

that owns an auction,

they can do that hold out

for you at a user level.

They can track all of those users

regardless of if you serve an ad.

Good examples are maybe easier to

describe in a first party data capacity.

If you're running email, you may blast

all of your customers and say, Hey,

I sent an email to all my

customers and this many purchased.

They went back to the website or

clicked it. But if you just said, Hey,

I'm going to serve just to odd

number of customer IDs and not to

even number customer IDs,

I can then just compare,

forget about who clicked on ads,

who did anything.

I'm just going to look at my backend.

I know I exposed these users,

but not these users 50 50 split.

They've historically kind

of done the same thing.

All I did was even an odd and just

measuring the difference between those two

groups.

So really any way that you can

establish a true control that

passes that AA test. So

before you intervene, do they

continue to look similar?

Are they influenced at the same rate so

that you can feel confident that when

you do intervene with new

media, retracting media,

some new sort of test that you are

confidently comparing to what would've

happened in a world

without that intervention?

Yeah, yeah.

It's applying the scientific

method with some rigor behind

what happens when I turn this channel on,

or what happens when I

turn this channel off?

What is the actual impact of this channel?

And what's interesting is I

remember back in my early days

of being in the advertising world,

this was when online stuff was

just getting kind of warmed up.

I was talking to this furniture store

owner and I'm like, Hey, what do you do?

Do you invest in radio ads?

Do tv, do you do newspaper?

And so as I went through Themm like,

Hey, do you do radio ads? And he is like,

yeah, I mean, yeah, I sort of do.

And I'm like, newspaper's like, yeah,

there's a big sale, something will

happen. I'm like, well, what about tv?

And he said, yes. And his

eyes lit up and he is like,

when I run TV ads, I feel

it. People walk in the door,

it happens. And I remember early on

in my online career thinking, man,

that was so unsophisticated. Did

that guy really know what's going on?

But now looking back, I'm like,

yeah, that's maybe all that matters.

That is incrementality in a real loose

easy just to observe with your eyes think

because you had one. Totally.

Which I think people

take for granted. Yeah.

They do.

Yeah.

That's not exciting. That's not

like, where's all your data?

It's in my cash register.

That's where all the data.

Is, especially for smaller brands,

when you have the ability

to feel if something's

working or not working,

if you double spend in something that

you think is working really well because

attribution says it's working really well,

and all of a sudden

your cash just doubles,

even though your attributed number

scales linearly, something has to give,

right?

And what has to give is it wasn't really

causing any additional top line growth.

It was just really good at

getting the attributed credit.

So I think the feeling

it in the p and l is

definitely overlooked.

It's valid, and it is overlooked

though. You're a hundred percent,

especially now that we have

so many tools at our disposal.

And I think another way to look at

this, and look, I'm a Google guy,

YouTube and Google is kind of where

I really got my start in online.

Marketing.

But listen, branded search is a

perfect example here. What happens,

we see this all the time.

What happens if you turn branded

search completely off? Now, I believe,

and this is top of front of the podcast,

there are strategic ways to use branded

search and there's ways to run it and

not waste money, but a lot of people

could shut it off and nothing happens,

nothing. Maybe sales get in a little bit,

but you take meta meta's really working

and you shut it off and you feel it.

Sales go down and that's

an incrementality.

Same is true for YouTube if you're doing

YouTube the right way. And so yeah,

I really like this. And one

kind of anecdote here to share,

we just did a test with Arctic,

Arctic coolers, Yeti competitor,

my favorite cooler, my favorite drinkware

as well. And so they wanted to see,

Hey, can YouTube drive an incremental

lift at Walmart? So they had just

gotten into most Walmart

stores, coast to coast.

So we did exactly what you laid out

there. We had a 19 test markets,

19 matched control markets.

So similar markets.

So think like a Denver and a

Kansas City or the example,

use Atlanta and whatever else

that's kind of comparable. And hey,

let's run YouTube in one

and not in the other.

And let's measure then the

growth in Walmart sales,

and let's do a comparison

between the two in Walmart sales.

And it was remarkable. It

was about an eight week test.

We had three test regions, so 19

markets, but three test regions,

test region. One, we saw an average

of 12% lift in Walmart sales.

The test region two was like 15% lift.

And then our final test

region was 25% lift.

And there were some standouts,

like Oklahoma City was up 40% and Salt

Lake City was up 48%. But it was one of

those things where, okay, now we

look at that and we can say, okay,

YouTube had a big impact. And

what's also interesting, Tom,

is we just ran the YouTube portion at OMG.

They also did a connected TV test

in other markets, not related,

didn't see a lift, didn't

see a measurable lift.

And so it could be lots of

that was not to throw shade on

CTVI like CTV,

so maybe they just did a wrong or

wrong creatives or who knows what.

But it's one of those things

where it's like, okay,

if you do this the right way,

you should see an impact.

And I think touching on the

piece that I didn't mention,

the other beauty or value of

incrementality testing relative to

attribution or mt a is the ability

to see beyond your.com to be able to

see what's happening on third parties

like Amazon, what's happening in store.

If you get that data own an operated

store or if you can get that through

wholesale data, it really simplifies.

There's so much complexity.

And I think that's, again,

one of the rubs that I have

with MTA is all of them,

all of the data you have to

wrangle together to try to

patchwork this kind of story together.

Whereas in incrementality testing,

it's pretty straightforward.

It's what did I spend and how

did I run that spend in these by

market by day or by week, and what

was my sales? What were my sales?

What were my new customers or whatever

metric I'd want to look at with that same

granularity and same dimension.

And that's really it because you're

really just trying to understand the

relationship that calls the

relationship between spend and outcomes,

all that kind of muddy middle

in the middle, trying to

get it at the user level,

which again, not going back into

the tube really simplifies things.

Yeah, it does.

And another thing that was

kind of interesting that

came a light doing this test

for Arctic is all of the ads we

tagged with available at Walmart,

shop at Walmart, find on the

shelves and Walmart, whatever.

We measured everything

though in those markets.

So you could look at Walmart sales,

online sales, so the.com and Amazon.

And what's interesting is the

push to Walmart really worked.

It's a reminder of what you ask someone

to do in an ad is what they're going to

lean towards. Because

in some of the markets,

we didn't see that much of an online lift.

We saw some clicks and stuff like

that, but the Lyft was at Walmart.

But we also saw a pretty

strong lift at Amazon as well,

because I think that just speaks to,

there's some people that are just going

to buy everything from Amazon right

there, tell 'em to go online value pro

proposition. Is it on Amazon? Yeah, yeah.

Yeah. Here in a day or two, it's hard.

To beat, dude. It's hard to beat

same price in a couple days.

I don't have to leave my house. But

yeah, really, really interesting.

And so we'll circle

back to that of course,

but let's talk about then

MMM or media mix modeling.

What is that? How are you using that?

And then how does that kind of relate to

incrementality testing? Because again,

going back to your tagline, Tom, you

did not say operationalizing NTAs.

You said operationalizing m

and ms and incrementality.

So what is MM and how does

that pair with incrementality?

Yeah,

basically a big correlation exercise

trying to suss out without a true kind of

holdout group,

what is the impact and contribution of

each media channel and also what would

happen without media.

So trying to suss out a lot of the

same questions as incrementality,

but basically using correlation as

opposed to having a true holdout group.

So basically,

and I'm sure all the hardcore MMM people

and data scientists will thumbs down

this or whatever you can do to podcast,

but hey, in this period of time,

sales went up and nothing could really

explain that other than the fact that

TikTok spend went up and essentially

doing that at a mass scale over longer

periods of time trying to take into

account anything that could explain that.

So you'll always kind of flag it with

these are promotions that happen,

it should because you're going to give

a model at least like two years worth of

data or two years worth of data,

it'll bring in seasonality and try to

understand those sort of trends. So it's

trying to pull out if not

seasonality, if not promotions,

if not some other things

that we are flagging.

And it wasn't price reductions,

it wasn't all these pieces,

what was happening in media

that could explain that change.

And so that's ultimately

what MMM is doing.

It's a big correlation exercise,

figuring out roughly what is the channel

contribution to a top line revenue or

order number and what's really important.

I think the nicest part or the best

first step with M is trying to get an

understanding of a base,

which is what it's going to be called or

intercept what without the presence of

ads,

does this model think that my sales would

be such that I can then calculate not

a total CAC of just looking at

total new customers divided by cost,

but incremental to media

or remove base from

that equation,

how many conversions were contributed

because of media as this model sees,

which no model is going to be perfect,

no measurement method

is going to be perfect,

but it's a really nice

place to start to say,

I knew I couldn't account all

new customers to advertising,

but what's a good number to use or

to start with? Well, it looks like,

and this will depend on the maturity of

the brand, but a really mature brand,

I mean super mature brand,

the big CPGs might be like 99% base

smaller brand might be something

like 50% because you've got

this word of mouth flywheel,

you've got product market fit,

but trying to get an understanding of how

much is media contributing relative to

customer base is a really

nice place to start.

And the benefit of running

incrementality and media mix modeling is

informing the model with

some of that causal data.

You see that a lot and there's a

really powerful feature of media mix

modeling is saying, Hey, yes,

that's a correlation exercise,

can't pull everything out,

but let me inform the model or at least

restrict the priors it can use or the

coefficient, whatever

you want to call 'em,

what it's searching for to try to find

a fit in this model and say, well,

I did a hold out test. I know

you don't have the causal data,

but we ran this in this channel and that

channel and helping that restrict the

model and giving it data that it can't

have without that human intervention can

be a really powerful flywheel.

So using your incrementality test data,

feeding that back into your MMM

model to make it more accurate and

more causal and make that correlation.

Stronger.

Because the two things that are really

like you're really trying to get,

but you don't get with Multi-Tech

attribution or attribution in general.

And you do get with the combination of

media mix modeling and incrementality

testing is the incremental impact,

the causal impact of what

would've happened without

the presence of ads as well

as the diminishing returns curve,

which we know can be really

powerful and important too,

is what has happened over time as I

spend in that sort of a feature of big

feature of media mix modeling

is understanding where

are you on a diminishing

returns curve? Is there

if I keep spending more,

I know it's not going to scale linearly,

but are there channels

that diminish faster?

Is there more headroom in other channels?

And it really becomes this

true optimization game of

where do I put the next

dollar? Ultimately the

question that every marketer,

every finance team is

trying to answer is, Hey,

if I find $20,000 into couch

cushions, where do I put it?

And if I need to give back $20,000,

where do I pull from to have.

I want to hang out at your house and

look at your couch cushions and find 20

grand? That's.

Great. Yeah, it's easy to

give it back, but yeah, right.

We're trying to figure out what is going

to be the least impactful if I have to

give the money back and cut budgets

and where is it going to be the most

impactful if I have another $20,000?

Because the answer is not going to be

found in what has the highest or the

lowest ROAS in an attributed

view. And in fact,

that can have the complete

opposite impact that you want.

Yeah, yeah, it's really great.

So I want to actually talk about

that point in a minute where

if you've got cut budgets,

which hey, listen,

there's been some uncertainty even as we

record this, tariffs up, tariffs down,

markets up, market down, whatever

consumer sentiment is all over the place.

So if things get a little bit

tight, what are we going to do?

We can't slash marketing,

we can't slash growth.

I think that sends you

into a death spiral,

but we might have to get pull

back and get more efficient.

And so let's talk about that

actually for a little bit.

So where can you be led astray?

I think you just made a post

on LinkedIn about this, right?

Where you start looking at performance,

which feels like the smart thing to do,

looking at ROAS and whatnot, and

you're like, well, great, well,

let's just cut the lowest ROAS

campaigns and channels. We'll be fine.

How does that lead you astray?

And if you want to talk about your

specific example to help illustrate these

points, that'd be great.

Yeah, totally.

I think the other one you're referring

to is I think branded search,

which we were talking about

earlier. And I love using both a,

because it can be really, if a brand

is spending a lot of money there,

it can be a really great place to go

find those savings without impacting top

line. But also frankly, it's

really easy to understand.

I think most people understand that

up and down the organizational chart

across departments, everybody sort

of understands the idea of, Hey,

if somebody's already

searching for my brand,

do I need to pay to get that

click and that conversion?

And I found that just the fact that

it's easy to understand can be a

really good gateway to incrementality

testing because it's easy to get buy-in.

Everybody understands that idea,

whereas it may be more challenging

to express that idea in

other types of campaigns.

But branded search is a good example,

and the example that you're referring to,

kind of a midsize brand that I was

working with went through that exact

exercise, had to cut budgets.

They looked at up and down the campaigns

they were running. It was like, Hey,

we just got to make the best decision

we can with the best available data.

They were basically running p max

non-branded search and branded search and

p max and branded search where had

the best attributed roas Best CPA

non-brand was really hard to justify in

a lower budget kind of environment based

off the attribution data cut that leaned

a little bit more into branded search

as a percentage of their budget.

And over the next couple months,

new customers in total revenue

was declining despite the

attributed ROAS and CPA

looking even better than ever.

And that's where was brought

in, looked at all these things,

saw the loose correlation to

non-brand and new customer

acquisition and top line,

just the general skepticism that

many have around branded search,

especially in a low

competition environment,

which they were in. There weren't many

competitors in the auction that we

could see in Auction Insights. So yeah,

ran a very blunt instrument

match market test,

which at a brand of that size and for a

branded search I don't think is ever a

bad idea. And yeah, no

impact to branded search.

It was about 20% of their budget,

which was substantial that you

can either make the decision,

I'm going to put that 20% back in

my pocket or save it for a rainy day

or give it to some other

place in the org or say, Hey,

I'm going to redistribute this to

something that I see in correlation

data that might help

drive top line backup.

Let's reinvest that in non-brand as

opposed to keeping it in branded. Again,

complete opposite of what

attribution would say.

And you see that a lot frankly with

branded search is an easy one to pick on.

Same with retargeting,

but really anything that's especially

challenging with the black box

solutions that blend,

and I'm sure we could do a whole talk

show on p max Advantage plus some of the

things that bundled together historically

radically different levels of

incrementality can be a real challenge

when you're then measuring on

attribution. But yeah, a

ranty way of saying yes,

finding areas to cut oftentimes

if you follow the attribution kind

of data can lead to really kind

of impactful in a negative way

business outcomes because the attribution

view just does not take into account

what would've happened

without the presence of those

ads like Incre Ality does.

And so can definitely lead brands

astray as they're looking to cut.

Yeah, really interesting. And yeah,

max notorious for leaning into

remarketing or branded search.

If you're not diligent about that, it

can lean into both of those things.

And so got to be mindful of that.

You also quoted something

that totally ties into this.

It's from a shop talk talk that

you went to shop Talk the show,

and I can't remember who said

it, but if you see high roas,

I know something is wrong and that the

auto targeting is just finding existing

customers. Do you remember actually

who said that and unpack a little bit?

Yeah, I forget his name and I could

look real quick. He worked for.

Mic.

The Post Dan Danone, the big CPG.

Yeah, I just really appreciated

that quote because I

mean always wonder if I live in sort of

a bubble of being super passionate about

incrementality versus attributed metrics,

but that was just really refreshing to

hear because I don't think that's the

natural.

It's not.

Thought in people's.

Head spend more.

But I really think it should

kind of spark some skepticism,

especially when your goal really

is to try to drive new customers.

My first,

especially if you think about both

incrementality in the context of a SC

or pex that's blending retargeting

and prospecting by default

and knowing diminishing returns

are my first dollars, yes, they're

going to be the most effective,

but if they are focused on people that

are already buying from me and my goal in

my head is new customers,

I should be shocked that I can

spend a hundred dollars and drive

this amazing new customer revenue

and not think that something is up or

even over time as I continue to spend

our BS meters should probably

go up a little bit more.

And I don't think they do by default. So

I found that comment really refreshing.

Yeah, I think that

really illustrates that,

right where it's like most of us would

think, oh, ROAS is going up great,

we're printing money.

Whereas maybe you should say BS

detector, something's wrong here.

This campaigns leaning into customers

that we're going to buy anyway.

And I'll give two examples here to

illustrate this a little bit more.

And I'll also, since we've been

picking on branded search so much,

I'll share a couple of ways I

think we should use it. One.

If.

Other competitors are

aggressively bidding on,

just know that if you're not Nike and

you're not Adidas and you're not like Ford

or something, it's not a

lock. If it's a new customer,

they could be swayed by a competitor.

And that's generally how we

like to separate it out is like,

let's have branded search for returning

customers and let's make that crazy

efficient or just turn it off altogether.

If.

It's a new customer, then again,

we want it to be very efficient,

but maybe we want it on because we

don't want our competitor to come in and

swipe us to give and swipe our

customer. And so one example of this,

I did a podcast with Brian Porter,

he's the co-founder of Simple, modern,

great Drinkware brand has become a friend

and they did a study incrementality

study and they found, I'll

get these numbers off,

but it was like branded

search was 10% incremental.

So basically what that means is if it

shows that I got a hundred new customers

from Branded Search,

I probably would've gotten 90 of

those if I had shut it off, right?

Only 10% were incremental.

So then what you would need to do there

is you need a 10 x row as on branded

search for it to even make

sense. If it's below that,

you're completely wasting

money. Pair that with,

and you and I were commenting

on the House analytics, HAUS,

Olivia Corey and team did 190

incrementality studies involving

YouTube and they showed with

tremendous amounts of rigor

that hey,

YouTube is probably 342 times more

incremental, meaning if

you see a one in platform,

it's actually like a 3 42 in

terms of incremental impact.

And so wildly different

between those two. But again,

we're just so drawn to in platform

row as man, we'll just say spin,

spin spend on p max and branded search

when really we should be saying,

let me lean into YouTube or let

me lean into top of funnel meta.

I think both those examples

too are really good examples.

To me it also speaks

though to the importance of

cost per incremental almost being

more important than incremental

percent incremental. And that's something

I always use with branded search.

I think you and I have a very similar

feeling around branded search.

There's definitely a

time and a place for it,

and it's one of those things where

it might not matter that it's 10%

incremental, 10% incremental relative

to what Google's attributing.

If your attributed CPA

is a dollar and now it's

$10,

but your margin when you sell a

product is a thousand dollars like

hammer that all day long,

that cost per incremental is still

extremely profitable and valuable.

And same with the YouTube piece.

If YouTube was four times as

incremental as Google said,

but your YouTube was crazy expensive,

it still might not be worth it

even though it's four times.

More.

Incremental than the platform was making.

And that's how I think a lot

about this with connected tv where

connected TV can be super powerful

and maybe more so than linear tv,

but if you can buy scatter

linear TV for a 10th

of the cost of CTV,

well it just has to be more

than a 10th as effective and

it's accreted, it's a positive.

So it becomes more of comparison

of a cost per than just a

blanket.

How incremental is something which I

always think is important to focus on and

call out.

To. Yeah, it's so good.

I mean measuring something in terms of

percentages can provide insights and help

make decisions, but ultimately

it's the cost per right.

Translate that into real dollars

to see if it makes sense.

100% agree with you,

but I think this also goes back

to and use your linear TV example,

and I still love TV and

connected TV and stuff. Again,

I'll use YouTube just because

I've got the numbers in my brain,

but with YouTube sometimes

we'll see a $5 CPM or a

$7 CPM in certain audiences

compared to other channels that are

15, 20, 30, 50, whatever.

Totally. And I'm like, well,

if we're reaching the right person

and if the message and offer are

good, how could this not work? And it's

one of those things where it's like,

okay, we're either one of those is

off, we're talking to the wrong person,

that's the wrong message,

or we're just not measuring it properly

and that's where we need to look at it.

So did you have a thought on that?

You another question on

MM here in just a second.

Yeah, yeah, totally. But it

made me think of the idea of,

I think the reason I'm starting to become

way more bullish on any channel that's

historically been hard to measure

where I think there's that arbitrage

opportunity of costs are still relatively

low because people haven't all moved

in because it's easy to attribute.

It'll be really interesting

with a house example,

does that inspire a lot

more YouTube buyers?

That's something that Google

should have put out way long ago,

but I think it would undermine

undermine search and that's their bigger

business. And I could do a whole

kind of rant and I'll save you that,

but the idea of incrementality first

measurement probably wouldn't be great for

the search business. So probably exactly,

haven't been able to make such a

good point that case on YouTube.

But you think about all the channels

that have historically been harder to

attribute,

that's where costs are deflated just

from a supply and demand perspective.

So when you can move in and get CPMs at

five to $7 and it's really effective,

but most people that are measuring

through attribution don't know it's really

effective, that's a huge win for certain

period of time until everybody's flood,

everybody and the costs go.

Up the market.

I'm sure there's a lot of people that

were not excited to see that study from

house like dang it, that means my costs

are going up. I don't like that at all.

So really good man.

So we talked about incrementality testing

and I think you can use tools like

House and then there are others.

We're just talking about work magic and

there's a number of others you can lean

into. Full disclosure,

they're pretty expensive,

but you can also do stuff on your own too.

If you've got someone that

can measure this stuff,

you can do a little bit of it on your

own. What about the MMM side of things?

What's kind of the easy way to start

there? Is there an easy way to start?

What do you recommend to people.

There? I don't know. I dunno if

there's an easy way to do anything.

I think, well, I guess

that's not totally true.

I think there's some ways to

run relatively easy incre tests.

So I think that's the

easier place to start.

Certainly you can always

ratchet up the scientific rigor.

I think the problem with looking

for an easy MM solution is

anybody could run a model with Robin or

there's a lot of open source packages,

but just because you can run a model,

it could say anything.

It's not necessarily rooted in this

can all of a sudden predict the future

and tell you exactly the

contribution from media.

Whereas incrementality can do

that a little more out of the box.

You may have wildly wide

confidence intervals,

but it answers the question.

It gives you the comparison.

I didn't do it in this market,

I did it in this market.

What is the Delta Media mix modeling?

You could build a model

to tell sort of any story.

The proof is sort of in the pudding of

if I do the thing that the model says,

does it change my top line?

Can I see over time that

when I listen to the model

that improves my top line?

So it's a lot easier to get started

with incrementality testing.

You can run poor man's match

market tests as I sort you can just

sort of pick,

some markets historically behave

similarly and there's certainly some risk

there, but with a model you might

think that it's an amazing model.

I just don't feel like there's a great

place to DIY that together without some

real scientific or statistical

rigor. Or if you do,

you've just got to try to prove it over

and over by taking some big swings. And

that's really,

I sort of feel like you can get away

with the kind of feel it sort of tests

without really running a true

incrementality test or model.

If you're a small enough business and

you spend a decent amount on Facebook,

maybe you're not willing

to turn off Facebook,

but are you willing to drastically

increase spend and see if you can feel

something at the top line? Okay, then

what happens if you cut it in half?

What happens?

And start to understand those curves on

your own is probably a less risky way

than trying to, I've never done

anything in R and I'm going to run

or done any sort of medium amount.

I'm going to try to run one.

That's probably a risky proposition.

Yeah, it's a really good insight. I'm

glad you answered the question that way.

I think, yeah,

leaning into the poor man's incrementality

test or just leaning really heavily

into a channel and measuring your top

line if you've got a small enough business

to look at that, but probably if

you're going to lean into MM M1,

you need a couple years of data and so

to be able to make some correlations and

you probably need to lean in to

someone or a tool with quite a bit of

experience because you can do that astray.

And on your comment on cost too.

I mean it's all relative and a lot of

times where you're going to need a medium

mix modeling is when you're spending

a significant amount in a significant

number of channels,

which you're probably only doing

if you are spending a lot total,

which you're probably only doing if your

revenue can support that high level of

spend,

which means that a tool may not be

all that expensive relative to the

opportunity you could derive from

it, which is where I always net out.

So I'm paying 10 or 20

grand for a tool monthly,

but it's allowing me to

redeploy millions in ad spend.

And it totally in completely

makes sense. So Tom,

this has been fantastic.

I'm just watching the clock.

I know we're kind of coming

up against it, but one,

I recommend people follow you on LinkedIn.

You put out some awesome content.

I love reading it.

Thank.

You. People should definitely follow

you on LinkedIn and you are, is it Tom,

what is your handle on LinkedIn?

You are Thomas B. Leonard.

Thomas B. Leonard. That's

probably confusing.

I'm very self-conscious of LinkedIn, so

I'm glad to thank you for saying that.

I think it's good, man. I think it's

really good. I like it a lot. Yeah.

Yeah, it's been fun to start

doing connecting with folks.

Definitely an area that had a lot

of excitement and passion for,

it's fun to have these

sort of conversations,

so I appreciate you reaching out a

while ago and that we could connect.

Absolutely.

Man. Absolutely. So then if

other people were like, Hey,

I just want to talk to Tom because maybe

you can help my brand or my business,

how can they connect with you and who are

you looking to or who do you feel like

you can help?

Yeah, definitely appreciate that.

Yeah, reach out on LinkedIn.

I spend time there. I love reading

everybody's thoughts and content. So yeah,

reach out on LinkedIn mostly we work

with consumer facing brands that

are trying to understand where to

put the next dollar or where to pull

in the scenarios. They have to really

kind of rescue people from attribution,

trying to better understand where they

can get more with their ad dollars.

I think to your point that you teed

up now is such an interesting time or

anytime that there's margin pressure,

there's more scrutiny

on a marketing budget.

Really want to try to help

empower marketing teams to

feel more confident with

what they're doing and ultimately the

finance teams to feel more confident with

what marketing team is doing. Hundred

percent. That's where I love to plug in,

but also just love to talk about this

stuff probably more than I should.

So always open to the conversation.

Yeah, I talk about that a lot.

I've read analytics and measurement

books on vacation and my wife

is like, what is wrong with you? And I'm

like, it's interesting. I don't know.

I like it. And so totally, we are

just a different breed I suppose,

but I love that.

And then I think this is a great way to

end it where if I've got an extra dollar

to spend on marketing, where do I put

it? If I need to cut a dollar of spend,

where do I cut it from?

And that's really what

this approach is about MMM

and incrementality. And so

I think their necessities,

I think attribution is broken and or

misleading in so many different ways.

There's some correlations there, so we

don't have to throw it out completely,

but I do believe you need to lean

into MMM and incrementality for short.

So connect with Tom on LinkedIn.

And with that, we'll wrap.

Tom's been fantastic. Thanks for the

time, the insights and the energy. Yeah.

Thanks so much Brett

time. Glad to connect.

Absolutely. And as always, thank you for

tuning in. We'd love to hear from you.

If you found this episode helpful,

someone else in the D two C space or

marketing space, and you think, man,

they got to listen to this, please

share it. We mean the world to me.

And with that, until next

time, thank you for listening.