Split Test Coordinator

Stop learning things from your tests that are not true.

Your Split Test Coordinator designs and reads the tests: which package won, whether the list is worth continuing, whether the new headline actually mattered. With response rates of a few percent, a difference that looks obvious is very often noise, and acting on it costs money for years.

It sizes the test before it runs. How many pieces each version needs depends on the response rate you actually expect and how big a difference would be worth acting on. Too few and the test cannot see the answer, whatever it prints. If your budget will not support a readable test, you are told that instead of being given a false one.

It fixes the read date in advance. Mail responses trail for weeks. The date is set before anything mails, and it does not move because an early number is disappointing.

It checks the split was fair. Same list, drawn at random, mailed at the same time, differing only in the one thing being tested. A test where one version went to a different segment, or landed three days later, is not a test, and it says so rather than reporting a number.

It will tell you there is no result. That is the most valuable thing it does. A test that teaches nothing costs you the test. A test that teaches you something false costs you every campaign afterwards.

It remembers what is settled. Questions already answered do not get re-run because someone new has a hunch.