or not to peek at your A/B tests.. 👀🙈
If you are like me you’ve peaked at least everyday for your first five experiments ever. Checking out Google Optimize, shout when it’s over performing, cry when it’s underperforming. What you then probably also remembered from those days is the variety of results you got in the first few days and essentially every day.
If you’re going for a statistical significant result of 95% and higher it is key that you fix the sample size and running time up front and stop the test when it’s done. However if you test on a bayesian scale you do not have this problem.
What are you using? Bayesian or frequentist P-values?
See you next week!
PS. Check out these three vacancies at the bottom. In case you apply, tell the world what newsletter brought them to you ;-).