How to Successfully Run an A/B Test on Your Email Campaign’s Landing Page

BY  KAYLA LEWKOWICZ We love A/B testing here at Litmus. A/B testing subject lines, calls to action, or preheader text can be a great w...

BY KAYLA LEWKOWICZ

We love A/B testing here at Litmus. A/B testing subject lines, calls to action, or preheader text can be a great way to optimize your emails for opens, clicks, and conversions.

But what about after the email?

It’s common practice to send your subscribers to a specific landing page once they click. So, how do you know what’s going to push a micro-conversion to a macro one? There’s more testing to be done!

We talked to Alex Birkett, Growth Marketing Manager at ConversionXL, to understand the ins and outs of optimizing your landing page as part of your email campaigns.

WHY SHOULD YOU A/B TEST THE LANDING PAGE OF AN EMAIL CAMPAIGN? WHAT ARE THE BENEFITS?

A/B testing is the only way to tell if the changes you make to a page are actually working or not. You can lead with your gut, and many people do this, and you can be quite successful. Or, you could be leaving a ton of money on the table.

Me? I’d like to know if I’m leaving money on the table. It’s like that old quote, “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” A/B testing, or controlled online experiments, help you know which elements of a landing page are hurting or helping your conversion rate, and they help you squeeze more out of the traffic that you’re sending to the page, therefore reducing the cost of acquisition.

HOW DOES A/B TESTING ON A LANDING PAGE DIFFER, IF AT ALL, FROM A/B TESTING EMAILS, LIKE YOUR SUBJECT LINE?

Users interact differently with emails than they do with landing pages. For one, if a user has landed on your page, they have presumably clicked through out of interest from your email campaign. From there, it’s important to align your pre-click (email) and post-click (landing page) messaging (this is known as “message-match” or “scent”).

Another difference is that landing pages have more elements, and therefore more complicated interaction effects. Finally, landing pages are optimized for macro-conversions, whereas you’re probably optimizing an email (in isolation) by micro-conversions, like click through rate.

Of course, you should be analyzing the entire campaign, from email open to credit card confirmation, to ensure the holistic success of a campaign. That alone is a solid reason to invest in a centralized optimization team.

WHAT LANDING PAGE ELEMENTS TYPICALLY HAVE THE BIGGEST AND/OR QUICKEST IMPACT WHEN A/B TESTING?

Everyone’s audience is different, so to say that adding client testimonials or a sticky CTA will always work would be misleading. That said, if you follow a heuristic framework, there are some things that will usually lift conversions:

• Improve your message’s clarity: Clarity trumps persuasion.
• Increase user motivation: Why should they complete this action?
• Remove distraction: Does this element assist the conversion? If not, it’s irrelevant.
• Reduce friction: Do people trust your site? Is there sufficient proof?
• Maintain relevance: Do you maintain the same message throughout the process?

Of course, making your site more usable and more accessible will almost never hurt your conversions (and it’s the moral thing to do). These principles are embodied by different tactics at different points in time. Most of these still hold up if you want actionable “implement this right now” ideas.

DO YOU RECOMMEND A/B TESTING MORE THAN ONE ELEMENT AT A TIME? IF SO, WHAT ARE SOME WAYS TO MAKE SURE AN OUTCOME CAN BE ATTRIBUTED TO A SPECIFIC TEST?

Yes! There’s this common advice of “only change one element per test” because otherwise, how can you tell what affected the result? Well, this advice is misleading, especially so for people who don’t have Amazon-level traffic.

For one, if you change your headline, CTA, and hero image and you get a 40% lift, how much do you care that you didn’t “learn” which element specifically caused the win? Second, it depends on how you define your Smallest Meaningful Unit. That is to say, you may consider “one unit” to be your headline, or your CTA, or your hero image. But you could also argue that your headline is made up of several different words, all of which could be considered “one unit.” So it depends on the scope of the change you’re looking to make and the effect you want to reach. This is to be determined by your analyst. But I really wouldn’t limit myself to “just one change per test” simply because it limits how you think about optimization.

DO YOU EVER TEST THE ENTIRE LANDING PAGE AS AN A OR B VARIANT?

If you have a good reason to believe that a completely different variant has a good chance of winning, then sure. Otherwise, completely transforming the page may be a waste of resources and a roadblock to moving faster and making more agile changes. Instead of redesigning the whole thing, I would spend my time and money a) doing more customer research to try to learn what really matters or b) increasing the beta of ideas you’re testing (i.e. a/b/c/d/etc. test) if you have enough traffic for it to be valid.

HOW LONG SHOULD AN A/B TEST TYPICALLY RUN BEFORE YOU CHOOSE A WINNER?

It depends on your traffic, minimum detectable effect, validity factors, business cycles, and risk profile, but generally, you should:

• Test for full weeks.
• Test for two business cycles.
• Make sure your sample size is large enough (use a calculator before you start the test).
• Keep in mind confounding variables and external factors (holidays, etc.).
• Set a fixed horizon and sample size for your test before you run it.
• Wait until the test ends to call it.
• Read this post about what not to do.

HOW DO YOU DETERMINE WHAT PART OF YOUR AUDIENCE SHOULD SEE THE TEST? DOES THIS RELATE TO THE WAY YOU SEGMENT YOUR EMAIL CAMPAIGN?

This is a really good question with a complicated and a simple answer.

The complicated one? If you’re running email tests, especially with multiple variables, as well as landing page tests, then you’re essentially running multivariate tests (email experience A, B, or C gets bucketed to go to landing page D, E, or F and they all have interaction effects).

This is complicated, and accurate analysis requires lots of traffic. Here’s the thing though: when you’re optimizing a landing page, you’re always dealing with a multitude of interaction effects via different traffic channels and customer journeys.

The simple answer: avoid worrying about the interaction effects from different traffic sources and just worry about optimizing the bottom of the funnel metrics (the macro-conversions). If you’re running at a large scale and really want to know which combination of email subject line, copy, CTA, and landing page headline, copy, and CTA is the absolute optimal, you can UTM tag different email tests and post-test segment them later to dig into differences in conversion rate. This would be a very mature level of operations and would assume a solid amount of traffic and tests run per month. Give that work to your analysts and move on to running your next test.

WHAT KINDS OF METRICS CAN HELP DETERMINE A TEST’S WINNER? HOW CAN THESE RELATE TO THE METRICS FROM YOUR EMAIL CAMPAIGN?

This is different depending on what type of business you are and what your goals are, but I would highly recommend optimizing for macro-goals. Yes, track micro-conversions (opens, click through, engagement, bounce rate, etc.), but the number you should make decisions off of is the one that most affects your business. For lead gen, this is the quantity and quality of leads that fill out a form. For e-commerce, this could be average order value or revenue per visitor. It depends. But don’t make decisions off of clicks or something like that—tie it to the macro-conversion.

Once your test has a “winner,” what do you recommend as the next steps? How many times do you recommend running a test for the same element before it’s “done?”

If you ran the test correctly, implement the winner. Take your money and move on to testing new things.

WHAT ARE SOME EXAMPLES OF SUCCESSFUL LANDING PAGE TESTS THAT YOU’VE SEEN IN THE PAST?

There are the famous ones like the Moz landing page tests and we’ve got a few of our own case studies (this is a really nice one on iterative testing).
I’m more interested in talks about A/B testing strategy, optimizing the optimization process, if you will. (Claire Vo from Experiment Engine has an awesome course with that title in our ConversionXL Institute, if I may offer a quick shameless plug).

WHAT IF A LANDING PAGE TEST DOESN’T RESULT IN A CLEAR WINNER—WHAT NEXT?

Well, you have a few options:

• Dig into the segments and learn or implement targeting rules. Maybe your test was inconclusive overall, but worked really well for [X] email promotion traffic.
• Iterate on your hypothesis. There are a million different ways to implement something like “social proof”, so try a different way.
Maximize your beta of test options, and figure out whether it’s an execution problem or a lack of influence (whether the hypothesis actually matters or you’re totally off track).
• Try something new (next item on the testing backlog).
• Try something more radical (stop changing button colors).
• Track micro-conversions and if important correlative metrics increase in a given variation, implement it.
• Implement your favorite option (or your boss’s favorite) and call it a day. I think it’s almost time for happy hour, and you don’t want to miss that for worrying about inconclusive results. 
• 
  
• Source