Do you split test? How do you do it? What kind of things do you split test and what do you do with the data? These are some of the questions we answer in this conversation with the awesome Matt Bacak about taking split testing for email marketing and taking it to the next level.

Are. You. Ready? 

SOME EPISODE HIGHLIGHTS:

(0:12) Join our FREE Facebook Group.

(3:15) Does Matt really make animals out of paper?

(7:23) Split testing vs optimisation.

(10:52) Testing is like a game - can you win?

(11:34) What should you split test?

(18:23) What is radical testing?

(24:29) Examples of radically different split tests.

(28:41) Tracking and managing data for split testing.

(35:19) Be better than you were yesterday.

(41:21) How to connect with Matt Bacak.

Split testing vs optimisation

A lot of marketers say they split test. But what most people do, in Matt’s opinion, isn’t split testing - it's optimisation. A direct-response copywriter, for example, might 'split test' two different headlines. But if you're only making slight variations, that's an optimisation technique. So according to Matt a lot of marketers are 'optimisers' rather than 'testers'. 

When Matt talks about testing, he means trying things that are radically different. For example, different angles, variations, or offers - such as a beautifully designed page created by a graphic designer vs a basic, ‘ugly’ one created by a direct response marketer.

Testing is like a game - can you win? 

To Matt, split testing in email marketing is like a game. Can you 'win' against someone else or yourself? Testing is closely linked to increasing profits, and Matt doesn't go one day without split testing anymore. 

He'll often run tests that 'fail', where the thing Matt thought was going to work doesn't. But he'll still gain knowledge and get answers to the question he asked himself on a particular day. 

Matt's been doing email marketing for 25 years. And this is how he makes it fun - by testing radically different variations. 

What should you split test? 

We asked Matt how he comes up with ideas. And a recent test Matt did was around trying to work out who out of some big-name players has better study points when it comes to webinars. 

Typically, when people find something that works, they tend to repeat it several times. In fact, as a rule of thumb, Matt starts paying attention if he sees something 3 times or more. Because that means the thing works. So, as an example, he decided to test a few options he'd seen for the 60-minute webinar reminder email against each other.

And every time Matt tests, he’ll learn something new. He likens testing to driving down the road at night time. You may know the destination, but you can't always see what’s in front of you until you shine your lights on it. And when someone shines a light through something they’re doing, Matt feels compelled to pay attention. In fact, Matt even looked at what type of emails the US presidents send. And a fun fact he shared is that President Obama had 25 people on his split testing team while he was in office! 

So for Matt, split testing is all about shining the light brighter on the things that get repeated often. He'll monitor people who are hard to beat because they're the ones on his radar - the ones he's paying attention to. And he'll always test things that are already proven to work.

What is radical testing?

We asked Matt to tell us more about the idea of radical testing. And he explained that the easiest thing to change when split-testing two emails is using different subject lines. However, he's talking about two radically different subject lines. 

And when looking at the data, it's not about open rates. Open rates are for egos, Matt says! What matters is that you get the right people to open your emails at the right time and do the right thing.

Sure, you might send an email and a lot of people open it. But if someone feels like your email wasn't meant for them when they open it, they're going to unsubscribe. They'll immediately find themselves in the wrong frame of mind at the start of the conversation (and before they even read the email). And that's not the reaction you want! If you send an email that sounds amazing but turns out to be a let-down, you'll get different reactions from people. And none of them will be what you want!

So the easiest thing to do when testing (the thing with the most dramatic impact) is to take radically different subject lines. This means you're completely changing the words and having radically different conversations.

Examples of radically different split tests

For example Matt once sent out a question to his audience about calling webinar replays 'reruns' instead - just like with TV shows. As he describes it, Matt will always come from 'a place of zero'. With every test, he assumes he doesn't know anything. Because he doesn't know how something will go until he tests it. And he finds he's normally wrong 80% of the time! Often, what he believes works won't work, and what he thinks won't work usually does! So in the process, Matt will question everything.

For example, something experts say in the marketing world is that you should always use 'you' instead of 'I' in your emails. But what if this idea can be challenged? What if maybe this 'rule' only applies to specific scenarios and situations? What if it's not always the right thing to do? 

There's a lot of 'assumed wisdom' in the marketing world that's been diluted over the years. One we often mention is the idea of reciprocity (originally introduced by Robert Cialdini). This principle is often applied to lead magnets. A free lead magnet should evoke reciprocity from your subscribers because you're giving something out for free. But it's not really free, is it? Because people are giving you their email addresses for it! 

Tracking and managing data for split testing

We also asked Matt how he and his team keep track of the data from split testing. And Matt pointed out that in order to get good data from email marketing you need time and enough test subjects (i.e. statistical significance). Matt and his team will typically test at least 10,000 people per test and run five different variations in order to collect solid data.

Back when Matt first started, emails were being sent out every 2 weeks. They were long emails that might generate sales for 2 whole weeks. But now things have changed, and it's typical for marketers to send out emails every day that only make sales within the first 6 hours (and more so in the first 2). After that, the email has often run its course.

When it comes to tracking, because Matt's been doing email marketing for so long, he still likes to still use Excel spreadsheets. He has someone pull the data out once a week for activities carried out in the previous week, and Matt will generally take a look at monthly reports. This means he's looking at 'true data', where any variations, such as a fluke in the tech or impacted deliverability, can be taken into account.

When looking at his reports, Matt will focus on specific metrics, such as subscribers' clicks and total dollar amount. The 'winners' for each split test get highlighted so Matt can see how many people received that subject line, how many clicked on it, and how much money that resulted in. Every single winner from split testing goes into what Matt calls ‘the winner file’. And typically Matt looks for subject lines in that file that won more than 2 or 3 times because those are the subject lines he should automate.

He'll also look at data and accounts every Monday. So he definitely has a good grasp on the numbers in his business. And if the numbers are down, he'll investigate. 

Be better than you were yesterday!

Generally speaking, Matt will decide what to split test based on who he wants to ‘beat’ on any given day. He's been testing since around 2006, and before that he thought he didn't need to - that what he was doing was working every time. At some point though, he realised he was leaving money on the table because of his ego - he was too scared to find out the truth! 

When he did the maths and worked out how much he could increase his conversion rate, he decided to start split testing. And he's been doing it every single day ever since. Because even on the days when he doesn't beat his own 'winners', he still learns the answer to the question he asked himself. 

It's worth mentioning that compared to when Matt started testing, things are much easier now - you can use segmentation in your email marketing platform, for example.

And one last gem that Matt left us with is that if you want to win at the email marketing game, you want to adapt. The greatest way to adapt is to have knowledge that nobody else has. And you gain that by beating yourself. And Matt's goal in business is to be better than the guy he was yesterday.

Boom!

Useful Episode Resources

About Matt Bacak

If you want to connect with Matt, you can find him on his free Facebook group.

Related episodes

Email Mistakes That Almost Got Me Fired (Avoid These) – With Chris Davis.

When Is It Too Early to Segment Your Email Subscribers?

How To Use Email Subject Lines To Do More Than Get Your Emails Opened.

FREE list of the top 10 books to improve your email marketing

If you want to write better emails, come up with better content, and move your readers to click and buy, here's how. We put together this list of our Top 10 most highly recommended books that will improve all areas of your email marketing (including some underground treasures that we happened upon, which have been game-changing for us). Grab your FREE list here. 

Join our FREE Facebook group

If you want to chat about how you can maximise the value of your email list and make more money from every subscriber, we can help! We know your business is different, so come and hang out in our FREE Facebook group, the Email Marketing Show Community for Course Creators and Coaches. We share a lot of training and resources, and you can talk about what you're up to.

Try ResponseSuite for $1

This week's episode is sponsored by ResponseSuite.com, the survey quiz and application form tool that we created specifically for small businesses like you to integrate with your marketing systems to segment your subscribers and make more sales. Try it out for 14 days for just $1.

Join The League Membership

Not sick of us yet? Every day we hang out in our amazing community of Email Marketing Heroes. We share all of our training and campaigns and a whole bunch of other stuff. If you're looking to learn how to use psychology-driven marketing to level up your email campaigns, come and check out The League Membership. It's the number one place to hang out and grow your email marketing. Best news yet? You can apply everything we talk about in this show.

Subscribe and review The Email Marketing Show podcast

Thanks so much for tuning into the podcast! Thanks so much for tuning into the podcast! If you enjoyed this episode (all about email marketing split testing) and love the show, we'd really appreciate you subscribing and leaving us a review of the show on your favourite podcast player.

Not only does it let us know you're out there listening, but your feedback helps us to keep creating the most useful episodes so more awesome people like you can discover the podcast. 

And please do tell us! If you don't spend time on email marketing, what do you really fill your working days with? We'd love to know!