On medium I found a great read from Luca Albertalli who is reflecting on a year of experiments and what he has learned. I want to highlight 2 important learnings, but I recommend you to read the whole article:
- The power of a single metric. Multiple metrics can have conflicting objectives. And by looking at multiple experiments the chances of false positives increases. If you have ten metrics (and they are independent of each other), you have 10 independent experiments, so you have around 40% probability of detecting an effect even if it doesn’t exist.
- If you are not happy with being wrong, look for another job, there is no place for you. Statistics are showing that only between 20 and 30% of the proposed experiment actually show a significant and positive impact, all the other experiment are insignificant or cause damage.
Furthermore in this newsletter:
- Empowering data with design @ Booking
- Learnings from Measurecamp Amsterdam 2019
- Book with statistical methods in online A/B testing
- Video: you can’t spell measure without customization
Have a nice day!