Steps to take before creating an A/B test for your app
by Joe · 34 things on Twos
- Research
- find out what is missing in your app store listing or what could be improved
- keep an eye on your competitors
- specifically the ones in your category that will be going head to head with your app
- Hypothesis
- What do you want to change in your store listing?
- What do you want to achieve with these changes?
- "I want to optimize my screenshots to increase conversion. My target audience watches reality shows like "Ex on the Beach,” so if we create screenshots for our game using this TV program as an inspiration, my conversion rate from the app store visitor to app download will increase."
- When talking about creatives, you can focus on two types of experiments:
- App Store Listing
- Conversion focused
- Search results
- turn impressions from the search results into product page visitors
- Remember!
- Traffic
- you need to get considerable traffic in every variation
- When you get enough traffic, you reach a realistic "Confidence Level."
- calculated by the a/b testing tool and shows you, based on the traffic sample, how accurate the result of the experiment is.
- For example, if your variation A result showed a 25 to 45% increase in conversions, and its confidence level is 95%, this means that this result will prove to be true 95 times out of 100.
- If your experiment results show you a slight difference between the variations, you need more traffic volume in order to make a decision.
- It normally takes at least 50 ‘Clicks to Install’ per variation to get statistically significant results. On average, you need 400-500 visitors per variation for a test on your app’s product page and 800-1000 users per variation if you run an experiment on the search or category page.
- Creating your Variation
- Running the Experiments
- never finish your experiment before 7 days
- Even if you already think you have enough data to make a decision, leave it on for at least 7 days.
- Analysis and Implementation
- Traffic Volume per Variation
- Conversion Number Per Variation
- Time to Click to Install
- Engagement Rate in the Store Listing
- how many visitors scrolled through your screenshots, watched your video, read your description, and tapped on the install button?
- Improvement Metric
- if the variation increased or decreased your conversion rate.
- Retention Rate