Your Experiments Results Are Not Significant Enough
A while back I posted this image on LinkedIn and asked if people thought I should end the experiment:
A few days later, I added an update and posted this image alongside it:
Based on the names, you probably can't guess what the differences between the two experiment variants were.
The answer is that there are none. They were exactly the same and all visitors were shown the exact same content on the page.
Run The "A/A Test"
Typically people refer to split testing with the phrase "A/B testing", meaning that you're either showing variant A or B to any given user.
What I want you to try is to create two experiment variants and don't change a thing. This will show you the power of statistical significance and why so many people make bad choices too early on in experiments.
Get To Know Statistical Significance
If you've taken statistics courses during your studies, you know the term statistical significance. But even if you have, I highly suggest you take 10 minutes to read this amazing blog post. I was sure I had a fundamental understanding of the concepts, but this post explains it really well.
The Silent 3rd Variant In Your Experiments
Now that you're aware of what statistical significance is and why it's important, it's time to see what we can do about it.
A way to ensure you're not forcibly generating a winning variant in your experiment is to add a 3rd experiment variant to your existing 2 variants.
This is simply done by duplicating one of the existing variants in your experiment. It doesn't matter if you're testing ad copy, button colors, or website redesigns. Including that extra variant will make sure you're not making conclusions too early.