by Will Barker
A/B testing is a valuable and in-demand skills that data analysts, BI developers, and data scientists have in their analytical toolkits. This beginner-oriented talk will explain the basic intuitions and statistical theory behind A/B testing and showcase a simple implementation in Python.
This quick talk is aimed at beginners who are interested in learning about data analysis and are looking for project ideas to add to their portfolios.
What is A/B testing? Running experiments to take the guesswork out of decision making. Provide successful examples: click-through rate of different web page designs, drug-testing, etc.
The Bayesian Framework: Introduce our dueling web-page example. The difference between observed frequency and true frequency. Defining prior probabilities, likelihoods, Posterior probabilities. Establishing our prior - flat or informed? We'll try both!
A quick explanation of the beta distribution, its parameters, and how it will model the success rates for both sites. Visualize our priors.
Randomly generating our data for both our test sites and their defining their true success rates. Adding our data to our priors and visualizing our posteriors. What difference does having an informed prior have? Understanding the posteriors.
Examining the cumulative distributions of our posteriors. Bayesian error bars. Looking at the area under the curve. P-Values
Overview of steps. Project ideas for collecting and A/B testing real data (e.g. polling, habit tracking). Concluding thoughts and useful resources for more info.
About the Author