A/B testing for emailing campaigns
Each time we change the design/text/timing of emails or notifications we need to measure the performance of the new settings.
The following steps are necessary to conduct a statistically well-founded A/B testing:
-
Splitting the user base
The user base (or the users that we need to perform the experiment on) needs to be randomly split into disjoint groups, Control and Treatment (several Treatment groups can be created if needed, each one reflects a particular variant in the settings).
- The Control group will be targeted using the the old original settings before applying the changes.
- The Treatment group will be targeted with the new settings
Certain level selection can happens before the randomisation to guarantee a more effective testing (e.g. select the users only in the US).
-
Deciding the performance measure(s)
For each experiment, we have to define the performance measure(s) we need to track. Example: in the welcome email, the performance measure could be the convergence rate (i.e. the ratio of users clicked on a link in the email to go to Minds).
-
Labeling and tracking actions
Based on the performance measure(s), we will implement tracking mechanisms that allow us to perform the calculations (e.g. the group of the user and if clicked on a button or not).
- Each setting and each user group needs to have a unique descriptor to allow identifying the experiment and measure the performance.
- Each UI element involved in the tracking the performance has to record wether the user interacted with it or not.
No need to know user guid, we just need to know the group of that user and his actions.
-
Performing the experiment
The more actions performed by the users in both groups the more the statistical significance will be accurate.
1,000 user participating from each group can be statistically sufficient to make a judgement.
changed milestone to %Kite Phase 1
added scoped labels
changed weight to 4
added scoped label and automatically removed label
- Developer
Rami what's your followup on this?
@ottman I have discussed with @markeharding about it yesterday. As far as I understood, we have the ability to record the engagements with the emails, and we can split the user base. I just need access to this data and I can perform the A/B testing.
Before that, the new welcome email need to be written and designed.
added scoped label and automatically removed label
- Developer
It seems like the current email system isn't suited for getting this done and started quickly. Please make sure to bring this up next call.
unassigned @Johnthetester