Evaluating a New Recommender System with an A/B Test
An online store's A/B test of a new recommendation system was left unanalysed when its analyst departed. I audited the test and evaluated it with user-level funnel analysis, z-tests with a Bonferroni correction and power analysis. The new system significantly reduced product page views by 14% and improved nothing, but the test was seriously flawed, with a severe sample ratio mismatch and contamination from a concurrent experiment, so I recommended fixing the group assignment and rerunning it.
Want the full details? View the complete project on GitHub →