The Transcript

Split-Testing 101: How to Know Which Words Work

Brian Clark: Hey there Rainmaker’s. Welcome to the show, as always. I am Brian Clark, founder and CEO of Copyblogger Media. Robert is off today, somewhere in a metaphorical coal mine getting his work done but that’s okay.

Today we’ve got a special guest, that will more than compensate for the lovely, deep tones of Mr Bruce. Her name is Joanna Wiebe of Copy Hackers, which is a site you should be paying religious attention to, if you are not already.

Joanna completely crushed it at last year’s Authority live event, talking about copywriting and testing the difference between this word and that word, this button and that button. It’s fascinating stuff. And of course, we have been talking a bit about split-testing in the previous episode, so I thought, “Let’s get someone in here who really kind of lives and breathes both aspects of this.”

She is a creative copywriter, and yet she understands the importance of figuring out scientifically what works, and what doesn’t.

Joanna, thank you so much for being here.

Joanna Wiebe: Brian, thank you so much for having me here because I have been admiring you for so long. It always gets me nervous to hear you say this stuff. I feel all nervous now. Like, “Oh-oh, what if I disappoint him?” Anyway, no, it’s great for you to have me here. Thank you.

Brian Clark: Oh, that’s just silly. Come on now.

All right. Now that we have got that out of the way, why don’t you share a little bit about your story. Kind of how you got to being this really go-to expert and running it through Copy Hackers like you do. You’ve got a story that got you there in the first place, and I know everyone does but I want to hear yours.

Joanna Wiebe: Right. Mine is like a lot of people, where you kind of fall into things.

I was a creative writing student and undergrad, which I really liked. I went to Japan for a year to teach, while writing a book, and I didn’t write a word. I came back and almost went to law school. Then some stuff happened in my life and I had this kind of switch where I knew I didn’t want to do some of the things I had done.

At that point I got offered a job at an agency. What they were calling a creative writer and I was like, “Yeah, that sounds good.” So I did that for a couple of years and then moved over to Intuit. The tech company for turbo-tax, quickbooks and all that stuff.

I worked as their senior copywriter for about 5 years and it was really in that time when I was calling myself a creative copywriter, I figured out that copywriting isn’t about the creative all at. But when you are in an agency, that’s a better title to have, than a copywriter, which just sounds dull and boring or something.

Brian Clark: Yeah.

Joanna Wiebe: Which is like a crazy idea, right? So that’s where I was and I went into it. There was this huge testing culture, and I was like, “Tell me more. I like it.”

When I was going through the whole law school thing, my favorite part of the LSAT was the puzzles. Like, “Figure something out and see how it works.” And this has always been very interesting to me. So when I moved over to this testing culture as a copywriter, and had everything so informed by data, it was just like a revelation to me. I loved it.

So I was there for about 5 years and I worked with Conversion Rate Experts for a while too, which is one of the first conversion rate optimization consultancies. They are incredible. They are out of the UK.

Then I went out on my own and started Copy Hackers. Basically, it’s to help the smaller businesses that don’t get access to the same sort of resources that the Intuit’s of the world get. For those people that can’t work with conversion rate experts because as much as it’s worth, they just truly don’t have the budget right now to afford for somebody to come in and do their optimization for them. So that’s what Copy Hackers is for.

I have been at that for about 3 years, and last year I got to talk at Authority, which so far, is one of the highlights of my career. It’s been very cool.

Brian Clark: Awesome. That’s a great story.

A couple of things come to mind. Sonia Simone will always take a creative writing degree because you can teach people the testing and the principles of copy and all that, but you can’t teach initial drive and talent to actually want to write. So that’s why Robert was a poet and now he’s a VP of Marketing. That was a tough case but don’t tell him.

Joanna Wiebe: He was a poet?

Brian Clark: He still is actually.

Joanna Wiebe: Well yeah, I guess that never stops, right?

Brian Clark: Yeah. But yeah, that’s how we first met. And the other thing is, I did really well on the LSAT and unfortunately did go to law school. Actually, I don’t mind that I went to law school, it was the years of practice before I quit that was annoying. So I think you took the right path.

Joanna Wiebe: Thank you.

What Every Real Copywriter Should Focus On

Brian Clark: Okay. Let’s get into this. We want to really talk about split-testing fundamentals. You know, the technology keeps getting easier for normal people like, you know, creative writing majors and ex-attorneys to use, but we have to have the proper perimeters here.

So we’ve been talking a lot about getting people on an email list, building an audience, as a precursor to maybe starting a new business or a content marketing initiative for an existing business, whatever the case may be.

So when you are building an audience, and you are starting a new site, you have your ideas about what’s a good headline, what’s good body copy and what’s the right button text. You know, all these things, but really where do we start when we are trying to build an audience in a new context?

Joanna Wiebe: I think that for us, it’s going to be a new audience but naturally, as we all know, the audience exists, so where are they, what are they doing and what are they looking for?

I would start just by going out and doing that initial research. I’m sure everybody does this, but may be they don’t consciously do it with tactics that are specific and documented along the way.

So like with everything we do in any sort of optimization, which is business optimization, or conversion rate optimization, or whatever, optimize your copy.

It all starts with going out and listening to your prospects. You may not have a single customer or subscriber yet but they are out there. They are talking and they are doing that all online in this very documented way. By that I mean, either in forums or they are leaving reviews on the Amazon products that they buy, that are related to what you want to sell. All those sorts of things. Going out, learning and just soaking them up, which every copywriter I think knows, but I don’t know that the world necessarily knows.

Brian Clark: Yeah. Well it’s interesting and I am so glad you started there because that is where the battle is either lost or partially won, before you ever think about testing. And you are right, pre-Internet, every copywriter, direct response to creative agency type, that’s the first job and they do this insane dive down into “Who are these people that we are trying to reach?” and I hope everyone still does that. And I say it every time I can but sometimes I think when it comes to content, as opposed to pure sales copy, people think maybe it’s not so necessary. It’s just whatever I want to talk about. And then it doesn’t connect with anyone and they are like, “Why?” And I’m like, “Well.”

Joanna Wiebe: Yeah. Exactly. “Why is my blog dead?” Right. There’s no, “Why is nobody reading it? Why is nobody signing up for it? Why is nobody coming to it at all and I am telling their friends about it?”

Unless you are Seth Godin, but then you already have a brand established, so then if you don’t have this established brand, I think it’s very hard to build anything, if you are not building it. I feel like I am saying the most obvious thing right now but if you are not building it for that audience, that means you have to absolutely know that audience intimately.

You know, if you are trying to attract students or write copy for students, you go and sit among them and listen to them, and they don’t know that you are listening to them.

Brian Clark: Yeah and now we have social media, which is the biggest eavesdropping thing in the world. It’s all free.

Joanna Wiebe: You don’t even have to leave your desk. You can do it all right here in this little box around you. It’s brilliant and I think that copywriters do that. But as usual, maybe I’m going to rant for a bit here, but we do things and we just accept them as part of how it has to work but I feel like there’s sort of this sense that you should get a bit of a pat on the back in marketing today, if you go do research.

I’m seeing that in a lot of blogs people are like, “Oh, good on you for picking up the phone and calling customers.” And I am like, “How else are you going to do it?”

Brian Clark: What?

Joanna Wiebe: Sure, good on you. Just like good on you for putting your pants on this morning. Good on you for doing the most basic thing.

Brian Clark: That’s right. Continuing to breathe.

Joanna Wiebe: That’s confusing to me, especially as a copywriter.

Brian Clark: Yeah, now I know, and it’s just a matter of perspective, because when things become democratized outside of the way things are done rightly in an existing industry, honestly that’s why Copyblogger came into existence 9 years ago.

The tactics I was teaching weren’t any different. It was the context in which they were applied.

It’s funny that the newer people eventually seem to get it, and yet some of the old school people still don’t get it. I find that odd too. Maybe when you are hitting the mailbox for your own career, you just can’t shift your mind to social and online. But that’s another episode.

Joanna Wiebe: I know, right? I’ve got lots of thoughts.

The Starting Point for Building Any “New” Audience

Brian Clark: Okay. Let’s assume that we all did the crucial up-front homework to know this prospective audience, better than they know themselves. That’s the goal at least.

Joanna Wiebe: Yes.

Brian Clark: We have made some educated guesses about them and the benefit that we are trying to give in exchange for their attention. Headlines, copy and all that kind of thing. It’s still a new site though. We’ve made some educated guesses but if we are smart, we wrote many headlines and then went with the one we kind of felt was best but we don’t know for sure. And this might be where some testing may come in handy but if it’s a new site and it’s not a huge traffic generator, how do you solve that problem?

Joanna Wiebe: Yeah, that’s a big one. When we are talking about testing, a lot of people will say, “Well I’ll just push some more traffic at it. I’ll go buy traffic.” But we don’t necessarily recommend that. I think the bigger thing that we would say to do, and the statement that we have, between Lance and myself, “The lower your traffic, the bigger your change is.”

So if you decide that you want to test your headline, like you said, but you don’t have a lot of traffic, for example you get 300 visitors to your homepage a day and you want to test that, your traffic is just going to be really low to make that happen.

So when you are testing one small thing, the impact is unlikely to be big enough to cause a big win in your testing tool like Optimizely or VWO or whatever it is that you are actually using to run the test, or even if you are using an Unbounce landing page.

Something like that that you are testing in, you need to have this huge difference that’s measured between your control and that new variation. So a small change on your new variation is unlikely. In most cases, I’d say 99% of the time, a small change like a headline, and as important as headlines are, just that alone is unlikely to bring you the huge numbers. The differences that you need, right?

Where if you have five people convert or 25 people clicking through on your homepage a day, let’s say per variation, you need the control to keep having that 25 people clicking through but you need the variation to get like 75, 100 people clicking through. You need huge differences in order for that test to ever get to a point of completion.

If it’s 25 versus 30, the numbers are way too low to say anything about it, right? So for us, the big thing that we try to recommend is if your traffic is low, make sure that new creative you test against the control, is dramatically different.

Brian Clark: Yeah. I kind of figured that out myself. Like I said, we have got split-testing tools built into the Rainmaker Platform, which are really simple to use and it’s tempting to just jump right into. Then I started looking around online going, “What’s going to make this statistically significant?” and I ran straight into that problem.

Now when we test on StudioPress or Copyblogger, traffic is not a problem. It’s huge, right? It’s just an issue that I really wanted to address with someone like you.

The Number of Conversions You Need to Make a Good Call

Brian Clark: Okay. So before we maybe look at some alternatives, what is the lowest amount of conversions per option that you would consider, something even paying attention to? I found some stuff online but I don’t want to say anything. I want to hear it from you.

Joanna Wiebe: There’s a lot online. At Unbounce’s CTA conference last September, Peep Laja from Conversion Excel quoted somebody who said “350 paid conversions per variation” and I’m like, “In crazy land. No way, for a small business.” Forgive me if I am getting this wrong, but it really does sound like the takeaway is, “Oh, crap. If I am not Amazon, I can’t test.”

I don’t think that you have to have that many, like 250 or 350 paid conversions per variation, in order for it to be a statistically confident test. For me, I look at things like the data is good and you have to listen to the data. Follow what it says and pay attention. Don’t jump to your own conclusions or say, “Oh, it looks like it’s trending up, so we are just going to call it a winner and go.”

Brian Clark: Yeah.

Joanna Wiebe: But at the same time, I think waiting around for these...