A/B Tests
This tutorial is verbatim from Source: https://towardsdatascience.com/the-math-behind-a-b-testing-with-example-code-part-1-of-2-7be752e1d06f.
1. Set up the experiment
The goal of running an A/B test is to evaluate if a change in a (say) website will lead to improved performance in a specific metric. You may decide to test very simple alternatives such as changing the look of a single button on a webpage or testing different layouts and headlines. You could also run an A/B test on multi-step processes which may have many differences. Examples of this include the steps required in signing up a new user or processing the sale on an online marketplace.
Baseline Conversion Rate and Lift
Before running the test, we will know the baseline conversion rate and the desired lift or increase in signups that we would like to test. The baseline conversion rate is the current rate at which we sign up new users under the existing design. For our example, we want to use our test to confirm that the changes we make to our signup process will result in at least a 2% increase in our sign up rate. We currently sign up 10 out of 100 users who are offered a premium account.
[15]:
bcr = 0.10 # baseline conversion rate
d_hat = 0.02 # difference between the groups
Control Group (A) and Test Group (B)
Typically, the total number of users participating in the A/B test make up a small percentage of the total amount of users. Users are randomly selected and assigned to either a control group or a test group. The sample size that you decide on will determine how long you might have to wait until you have collected enough. For example, websites with large audiences may be able to collect enough data very quickly, while other websites may have to wait a number of weeks. There are some events that happen rarely even for high-traffic websites, so determining the necessary sample size will inform how soon you can assess your experiment and move on to improving other metrics.
Initially, we will collect 1000 users for each group and serve the current signup page to the control group and a new signup page to the test group.
[16]:
N_A = 1000 # Number of users in control group
N_B = 1000 # Number of users in test group
[20]:
import scipy.stats as scs
import pandas as pd
import numpy as np
np.random.seed(2)
def generate_data(N_A, N_B, p_A, p_B, days=None, control_label='A',
test_label='B'):
"""Returns a pandas dataframe with fake CTR data
Example:
Parameters:
N_A (int): sample size for control group
N_B (int): sample size for test group
Note: final sample size may not match N_A provided because the
group at each row is chosen at random (50/50).
p_A (float): conversion rate; conversion rate of control group
p_B (float): conversion rate; conversion rate of test group
days (int): optional; if provided, a column for 'ts' will be included
to divide the data in chunks of time
Note: overflow data will be included in an extra day
control_label (str)
test_label (str)
Returns:
df (df)
"""
# initiate empty container
data = []
# total amount of rows in the data
N = N_A + N_B
# distribute events based on proportion of group size
group_bern = scs.bernoulli(N_A / (N_A + N_B))
# initiate bernoulli distributions from which to randomly sample
A_bern = scs.bernoulli(p_A)
B_bern = scs.bernoulli(p_B)
for idx in range(N):
# initite empty row
row = {}
# for 'ts' column
if days is not None:
if type(days) == int:
row['ts'] = idx // (N // days)
else:
raise ValueError("Provide an integer for the days parameter.")
# assign group based on 50/50 probability
row['group'] = group_bern.rvs()
if row['group'] == 0:
# assign conversion based on provided parameters
row['converted'] = A_bern.rvs()
else:
row['converted'] = B_bern.rvs()
# collect row into data container
data.append(row)
# convert data into pandas dataframe
df = pd.DataFrame(data)
# transform group labels of 0s and 1s to user-defined group labels
df['group'] = df['group'].apply(
lambda x: control_label if x == 0 else test_label)
return df
[21]:
ab_data = generate_data(N_A, N_B, bcr, d_hat)
ab_data.head()
[21]:
| group | converted | |
|---|---|---|
| 0 | A | 0 |
| 1 | B | 0 |
| 2 | A | 0 |
| 3 | A | 0 |
| 4 | A | 0 |
The generated data has two columns. The converted column indicates whether a user signed up for the premium service or not with a 1 or 0, respectively.
2. Run the test and record the success rate for each group.
[22]:
import numpy as np
ab_summary = ab_data.pivot_table(values='converted', index='group', aggfunc=np.sum)
# add additional columns to the pivot table
ab_summary['total'] = ab_data.pivot_table(values='converted', index='group', aggfunc=lambda x: len(x))
ab_summary['rate'] = ab_data.pivot_table(values='converted', index='group')
ab_summary
[22]:
| converted | total | rate | |
|---|---|---|---|
| group | |||
| A | 114 | 1070 | 0.106542 |
| B | 17 | 930 | 0.018280 |
It looks like the difference in conversion rates between the two groups is 0.028 which is greater than the lift we initially wanted of 0.02. This is a good sign but this is not enough evidence for us to confidently go with the new design. At this point we have not measured how confident we are in this result. This can be mitigated by looking at the distributions of the two groups.