Home | Email Marketing | Happy Holidays split test results

Happy Holidays split test results

As an email provider, we are very conscientious about sending too many emails to our subscribers. So, when we sent our December newsletter last week, we included our holiday greeting so as to avoid sending a separate holiday email that might annoy people. But, we wondered, would people be more likely to open our email if they perceived it as more of a newsletter or more of a holiday email? To answer our question, we performed a split test when we sent the email. We used the same exact email content, but we varied the subject lines between:

  • Subject line A: ClickDimensions December 2012 Newsletter: Happy Holidays!
    • and…
  • Subject line B: Happy Holidays! ClickDimensions December 2012 Newsletter

Obviously, the A version of the subject line was written to emphasize the newsletter aspect of the email while the B version lead with the holiday greeting.

Let’s walk through the test. We first created an Email Send record and gave it our A version subject line. Then we clicked the Split Test button to begin the test…

The first step of the split test wizard is to specify how our B version email should vary from our A version. For this test, we only changed the subject line. We could have modified the template, content or sender (‘from’ email address and/or name), but we just wanted to experiment with the subject line…

The final step of the split test wizard asked us to set our sample sizes and specify how and when the system should pick a winning version. We chose to send our A and B versions to sample groups of ~500 recipients each and pick a winner based on unique email opens after 3 hours of sending.

Three hours after the email was sent to the A and B test groups, the system automatically sent the A version to the remaining recipients (i.e. the remaining 84% or 4756 recipients from the screen shot above) because 17% more people opened it than opened the B version…

So, had we not run the split test and sent the email with the B version subject line, our newsletter results would have fared 17% worse.

Our conclusion is that more people were interested in getting our newsletter than another holiday email. Leading with the words ‘Happy Holidays’ just wasn’t as effective. Bah humbug and Happy Marketing!

About the Author:

The ClickDimensions marketing team is committed to helping marketers using Microsoft Dynamics 365 achieve their goals.

Leave A Comment