Email Split Testing: Check For Statistical Significance

A recent post on Listrak’s blog outlines an often overlooked aspect of email split testing. Split testing is a way to test different email copy, subject lines, and even from lines to see which version of an email performs the best.

Statistical significance is a way of ensuring that your test results actually have meaning. If the sample sizes are too small, there’s a good chance that any measurable difference in click-through rates between the two emails is simply random chance. Obviously, you don’t want to base any decision on results that can’t be trusted.

Here is Listrak’s procedure for ensuring that your test results are worth using:

For example, if I wanted to test two copies on an email looking for the better click-through rate, I’d do the following:

1) Estimate or use an average of previous click-through rates to determine what I would expect the click-through rate to be. This is the projected “success” rate of any version.
2) Determine how many versions you’re testing.
3) Determine what an acceptable variance from the success rate is, found in step 1. This is what would make you think something making a difference; i.e. with a 4% click-through rate normally, a variance of 1% would be note-worthy.

Once you have these figures, plug them in as follows:

The above equation may seem complicated but it’s actually quite simple. Give it a try the next time you’re split testing an email.