behance github medium

The Idiot’s Guide to A/B Testing: How to Improve Every Email

This blog post was first published on my other website, A blog called, “Grow Your Online Store with Email.”

Do your customers like your emails?

You’re looking at your daily reports but has there been any improvements?

A 2% increase in open rate isn’t shit, boo.

There’s a process that allows you to compare emails to find out what your subscribers want.

Yep! And it’s not that hard too.

It’s called A/B testing.


What is A/B testing?
Remember your middle school science project? It’s coming back! The process of A/B testing is similar to the scientific method.

A/B testing is the act of comparing two versions of an email to determine which one performs better.

The first email, Version A, is your control. Your second email, Version B, is your dependent or variation.

You’ll send both email versions to a portion of your list at the same time. To find out which one has higher conversions.


Why Your Store Needs It
If you want more sales from your emails, then you need to create emails that they want to open.

A/B testing will help you find your subscribers’ preferences.

Most brands will send emails every other day without getting any useful insights from them. So their marketing never improves, and their conversions stay low.

When you routinely test your emails, you’ll learn to design content that will lead to faster and bigger conversions. Meaning money!


You Can’t Fail
The best part of testing is that you’ll always learn from the results.

Was your hypothesis proven wrong? That’s ok. Just record it and implement the results in your next campaign.

With every experiment you perform, you’ll learn to create a better email marketing experience.

I hope you got your science goggles ready! It’s time to breakdown the A/B testing process.


The Step by Step Guide to Email Testing

Step 1: Observe Past User Behavior

Before you test, collect any past data of customers interacting with your online store. This will give you a better idea of what your customers want from your brand and how they interact with your sales funnel.

Quantitative data:

  • Heatmaps
  • Visitor recordings
  • Google Analytics
  • Email marketing reports

Qualitative data:

  • Surveys
  • Feedback forms
  • Online chats
  • Customer emails

This might seem tedious, but you need to acknowledge the current relationships you have with your customers to track improvements later.


Step 2: Pick What You Want to Test

Next, select which email element that you want to test. You have so many things to choose from! Here’s the list:

“From” name and email address 2017-04-22 22-19-42
Don’t change your name too much. Your subscribers might get used to one and to switching it up can confuse them.

Subject Line 2017-04-22 22-20-01

  • Length: Test a short line vs. a long line
  • Personalization: Try adding their names to get more opens
  • Emojis: Add personality with cute emojis
  • Numbers: Get to the point with numbers like 30% off
  • Capital Letters: Maybe subscribers prefer lowercase headlines over uppercase

Pre-Header 2017-04-22 22-20-23
Experiment with edgy text to get more opens and clicks. Urge subscribers to read the rest of your email.

Body Copy 2017-04-22 22-21-28
You have to spend more time crafting a body copy that increases clicks to your store. Try testing these:

  • Text Email vs. Visual Email: Maybe a text based email will get more action than a visual email
  • Hard Sell vs. Soft Sell: Maybe subscribers want text explaining your offers than just SHOP NOW
  • Long vs. Short: Maybe longer promotions will lead to more sales than shorter ones
  • Tone of Voice: Experiment with your tone. Maybe fun and lighthearted text are better than a serious one. Also experiment with your audience’s data (preferences, demographics, behavioral patterns) to find out if personalization will urge them more to buy.

Images 2017-04-22 22-22-20

  • Number of pictures: Find out if having too many images is causing your conversions to lack.
  • Placement of pictures: Find out if where you place your images will lead to more clicks.

Call to Actions 2017-04-22 22-23-59

  • Copy: Instead SHOP NOW, try BUY NOW or SEE MORE
  • Color: Maybe red is more clickable than green
  • Placement: Maybe putting one CTA is stronger than five
  • Design: Maybe a simple box design is better than a fancy frame
  • Text vs. image: Maybe no button is the best option. SHOP NOW might be better.


Try to find the best time and frequency that your subscribers engage the most.

  • Day of a week: Mondays vs. Thursdays
  • Time of day: 9 a.m vs. 6 p.m
  • Season: National holidays vs. International holidays
  • Frequency: Once a week vs. Three times a week

Offer 2017-04-22 22-26-04

  • Discount codes: Maybe repeat your code more in the email
  • Text: Maybe describing the offer is better than just giving the code
  • Images: Maybe it’s better to have the offer explained with text, not just an image.

Footer 2017-04-22 22-23-09

  • Social media: Maybe subscribers want to consume your content somewhere else
  • Newsletter archive: Maybe linking to past emails will increase engagement

And so much more!

WARNING: Test one element at a time! Don’t get complicated. Test different lengths of your subject line, NOT different lengths AND emojis. And don’t test your subject line and email copy. This will confuse you when you read your results because you wouldn’t know which was more important.


Step 3: Write a Hypothesis

Do you remember what this means? Lol just kidding. It’s an important line that will keep you on track.

A hypothesis is a statement explaining what you assume will happen and why:
“If _____[Variable] _____, then _____[Result]_____ [Rationale].”


  • Putting emojis in the subject line will increase email open rates because it’s fun.
  • Adding a “Reply” CTA will help improve quick responses because it’s direct
  • Adding a summary of the lead magnet next to the download CTA will improve the click-rate and lead conversions because our subscribers like explanations.
  • The click-rate will increase on emails for prospects who have attended our previous past events because they have shown significant interest.

Step 4: Start your Test

Here’s the fun time! Now, you’re going to A/B test. Luckily, most email marketing software will allow you to A/B test. Because I love you so damn much, I added pics on how to test in Mailchimp.

1. Pick A/B testing from the drop-down menu 2017-04-22 21-01-49

2. On the variable page, here is the basics of your A/B campaign. 2017-04-22 21-08-59

3. You can pick up to 3 combinations of each testing elements (subject line, from name, content, or subject line). But I advise you to start with only 2. 2017-04-22 21-09-48

4. Here you will select the percentage or portion of your list to test. You will also be setting the goal metric (revenue, click, open rate) and testing time by hour or day. 2017-04-22 2017-04-22 21-14-20

If you have a small list of 1,000 subscribers, I recommend testing on 50% of your list. 500 people are the testing segment. And the rest 500 people will be given your winning email.

5. Because I picked the subject line to test, here is the setup page. Here I can name my test. I can write my testing elements: Version A (Email Subject 1) and Version B (Email Subject 2). 2017-04-22 21-20-18

Don’t forget to track your opens and clicks! 2017-04-22 21-21-55

6. After confirming your email’s content, you can then send or schedule your email campaign 2017-04-22 21-33-56

7. Here’s your A/B report page, this is where you can monitor your overall campaign activity. You can see your winning metric and results of each element version. 2017-04-22 21-34-41

If you want to know more about each version, click on the “view report” link. 2017-04-22 22-04-37

8. Don’t forget to see your other version by clicking the next arrow. 2017-04-22 21-38-12


Step 5: Analyze Your Data

Look at your A/B test results, and see which variation had scored the highest winning metric. If there is a clear winner among the changes, record it and try it in your next email. If the test is inconclusive, go back and do the test again.


Step 6: Draw your Conclusion

Now that your test is finished, record your results in a spreadsheet so you can track trends and write notes. I recommend you performing the same test a few more time to notice a pattern or if it’s statistically significant – when the results of your tests are determined to be most likely not due to chance alone. Another topic for another blog post…

5 Testing Tips to Improve Your Results

Make Testing a Habit

You always need to test! Prepare campaigns and schedule them for weeks. When they’re sent, record the results. You should A/B test at least once a week. As trends become more obvious, make changes to your emails.

Don’t End the Test Too Early

When deciding how long the test should be, give yourself many hours. Because depending on when you test, your engagement can be effected. You need to wait for results to peak. At least 8 hours.

If I send a subject line test at 10am, I want the test to end 24 hours later so I can give my sample time to engage with my email.

Acknowledge All Factors

When testing, understand how the differences can impact your results. According to GetRsponse, “in testing subject lines, different results may appear due to variations in length of the subject line, choice of wording, placement of numbers, use of capitalization, and use of personalization.”

Don’t Make Your Subscribers Mad

When testing, make sure your that your complaints and unsubscribe are low. High open rates shouldn’t also mean high unsubscribes.

Look at More Metrics

Your testing program needs to go beyond open and click rates. Did an email earn you a sale? Or did it increase traffic to your website? Look at your Google Analytics, see any big changes?

Hold Up! One More Thing…

Enjoyed this blog post? I hope so!

giphy (1)

To get more advice on how you can use email to grow your store, please sign up to my email newsletter!

Related Posts

Leave a Reply

Your email address will not be published. Required fields are marked *