View profile

CRO weekly #9 - Can you peek at A/B test results? / How small should the change in my A/B test variation be?

Goodmorning all, In this week newsletter the fourth statistical ghost: Peeking. When conducting AB te
CRO weekly #9 - Can you peek at A/B test results? / How small should the change in my A/B test variation be?
By Tom van den Berg • Issue #9 • View online
Goodmorning all,
In this week newsletter the fourth statistical ghost: Peeking.
When conducting AB tests, it is tempting to check your results whilst the test is running to see how it is performing, colloquially known as ‘peeking’. But in doing so you are giving this ghost more opportunities to fool you in to thinking an ineffective treatment has had an impact.
Simply looking at the data is not a problem, but taking any actions based on what you see can introduce bias. After all, it is far more common to stop an experiment halfway through the intended run time because it is significant than it is to stop it because it hasn’t reached significance yet.
What else in this newsletter:
  • How small should the change in my A/B test variation be?
  • Methods of workflow sanity with Google Tag Manager.
  • [DUTCH] Opzetten van een experiment gedreven cultuur
  • AB Tasty Launches Nudge Engagement
  • AI For Everyone | Coursera
Enjoy your week.
Cheers!
Tom

Reads
Tools
Courses
Did you enjoy this issue?
Tom van den Berg

CRO weekly: reads, events, talks, tools about conversion rate optimization

If you don't want these updates anymore, please unsubscribe here.
If you were forwarded this newsletter and you like it, you can subscribe here.
Powered by Revue