A/B Testing on Mobile Apps: Essential Facts and Practical Tips for Beginners

Should we recommend similar products on the main view or not?

Is it okay to send notifications about the new features, or is it useless? 

Will any of our ideas improve the conversion rate?

If you’re on the app development team, you probably ask yourselves similar questions every now and then. It can be challenging to make the right decision, and discussions can go on for hours.

Luckily, instead of guesswork, you can use A/B testing on mobile apps. Sometimes, it’s the best way to determine which changes are worth the effort. Get to know when and how such tests can help you in projects.

---

Key takeaways

• A/B testing is a method of comparing two or more versions of a mobile app element to determine which performs better. It involves showing different versions to users and measuring the impact on a chosen metric.
• A/B testing on mobile apps help you make better decisions, improve UX, increase conversion rate, and get to know users better.
• You can test, for example, UI design and layout, navigation, copy, forms, features.
• To conduct A/B tests, formulate a hypothesis about why a change is expected to improve performance.
• It’s crucial to define clear metrics aligned with your goals (e.g., conversion rate, retention).
• Users must represent the same segment. The sample size should be sufficient for statistically significant results.
• Tests must run long enough to capture representative user behavior.
• Key elements include a control version and one or more variant versions. Test one element at a time.
• In certain cases, A/B testing isn’t recommended. You may consider, for example, usability testing or UX audit instead.

---

What is A/B testing?

A/B testing (split testing or bucket testing) compares different versions of the same view in an app or website to see which one drives better results.

The foundation of every A/B test is a hypothesis. Based on it, you can decide what changes should be implemented and tested. Then, you choose metrics to monitor during the test.

Example
Problem: Your team noticed that users of your mCommerce app rarely join the newly launched loyalty club.

Hypothesis: You think people don’t join the club because the information about its benefits isn’t clearly stated in the app.

Solution: You decide to display an additional popup with more details on the home view, but you’re unsure if this new solution is better than the one you already have. Your team’s opinions are divided, so you test options A (an unchanged version) vs. B (with a popup) to determine which is best.

Success indicator: In this test, you will monitor the number of new loyalty club members.

Risk: Mobile users may be annoyed when a new element appears on the screen. Instead of joining the loyalty club, they may uninstall the app or stop opening it. That’s why you also check the number of uninstalls and churn rate.

How can AB tests support your business?

Split testing enables you to improve your product and channel your app development plan in the right direction. What benefits does AB testing on mobile apps offer?

• More chances for success – the test results tell you which version of the view gets you closer to achieving your goals.
• Data-driven decisions – instead of endless discussions based on guessing, you can form your decisions based on reliable data and avoid risk.
• Better position during business meetings – when speaking to stakeholders or your team, you have solid arguments that support your stand, and you can determine more precisely the predicted impact of your recommended changes.
• Improved user experience – you can make informed decisions that build positive UX and increase user engagement.
• More conversions – when you know what users find more intuitive or interesting, you can implement solutions that directly improve the conversion rate.
• Better knowledge about users – you get to learn about user behavior patterns. Such knowledge becomes valuable for app marketers and UX teams.

What can you test in the mobile app?

Teams often run A/B tests to check changes in:

• UI design and layout (e.g., button size, placement of various elements such as headlines, promotion info, or popups)
• Navigation (e.g., different checkout or onboarding steps)
• Copy (e.g., headlines, call to action, content in PUSH notifications)
• Forms (e.g., new fields or different types of responses available)
• Features

Users for A/B tests

To obtain valuable results from the AB test, divide users into two groups. All users must represent the same segment (which means they must have something in common, such as having created accounts at a similar time).

Each group will see a different view – A or B. At the end of the testing process, you should be able to determine which variant encourages the desired behavior more.

Example
In your mCommerce app, the information about seasonal promotions is placed at the top of the main view. The marketing team says it doesn’t catch enough attention and recommends a bigger banner. The UX designer doesn’t like this idea, so you run A/B tests. In this case, your target group is users who haven’t joined the loyalty club.

Sometimes, the user segments for tests don’t need to be so specific. You may want to check something universal so that all users can profit from it, such as placing the microphone button or a cart icon.

How many users do you need for AB testing?

The number of users must be big enough to obtain statistically significant results in a proper time. If it’s too small, the outcome may be accidental. To avoid that, first, you need to calculate the sample size.

The good news is that you don’t have to be an expert in mathematics to get this information. There are plenty of tools that will do the job for you. You just need to provide some data, such as:

1. The number of variations – you need at least two (control version and version with a change).
2. The current conversion rate you get on your control version. You can calculate it by dividing the number of conversions by the number of users.
3. The expected conversion rate you want to detect.
4. The statistical significance – it’s usually set at 95%, which means there’s a 95% chance that your results will be valuable and a 5% chance they will be random.

This set of data can look like this:

1. Number of variations: 2
2. The current conversion rate: 10%
3. The expected conversion rate: 15%
4. The statistical significance: 95%

Tools

Examples of online sample size calculators for A/B tests:

• Evan Miller
• AB Tasty
• Optimizely
• Speero

TIP: It’s best to prepare all this info with goals, user group description, and metrics in one document that you can easily hand over to UX designers. You can ask them about their opinion.

Duration of the test

Once you’ve determined the sample size, check how long the test should take. Again, you can use an online calculator.

First, choose the number of UI versions for a test. For example, in the most straightforward A/B test, you have two versions: the unchanged control version (A) and a new one (B).

Next, enter the average number of unique users that open your app daily.

Based on this data, the calculator will show you how many days it should take to get reliable outcomes from your test.

The result above was calculated using the AB Tasty tool.

When split testing is NOT a good idea?

If you think A/B testing can answer all your questions, hold your horses. Here are several cases in which you should consider a different approach.

Case no. 1: Feature implementation is time-consuming and requires much effort.

It sounds reasonable – you want to make a significant change, so it’s best to test it first. It’s true, but A/B testing requires feature implementation. Thus, app developers must write tons of code to run such tests. If the feature is a success, then there’s no harm, but when the results aren’t optimistic, it means you’ve wasted a lot of time.

A/B tests work better when you make changes that improve existing ideas or their implementation isn’t very time-consuming. Otherwise, they may not be the best research method.

Case no. 2: Changes can be analyzed for compliance with UI design standards.

There are some UI design standards you can follow without having to test them.

Example

You don’t like the buttons’ colors recommended by the UX designer. Tools for A/B tests allow you to check them, but it doesn’t mean you should. Instead, discuss propositions of changes with their author. Analyze if they meet your style guide requirements and apply principles of usability heuristics. Alternatively, you can ask UI designers from outside your company to prepare the UX audit for you.

Case no. 3: You don’t have enough users or time.

Sometimes, it’s easy to assume that the test is already giving you the final answers even long before its ending date, as suggested by the calculator. It wouldn’t be the right decision to finish it earlier, though.

Example
The calculator shows you that the test must take one month, given the number of average daily users. After one week, you see that version B clearly drives better results than version A. You may think your hypothesis was correct, but the results of this mobile app testing weren’t statistically significant, and you risk choosing the wrong option.

If you know that your app doesn’t get enough traffic to run statistically significant AB tests in a reasonable time, consider different research methods. Otherwise, you’ll only spend a lot of money on testing and still won’t know what works for your users.

Case no. 4: It’s hard for you to make a hypothesis.

As mentioned earlier, making a hypothesis is the first step in designing A/B tests. Hypotheses tell you what changes in the user interface may be helpful and indicate what metrics will be important in the testing process.

However, sometimes, it’s hard to make any hypothesis at all. If that’s your problem, wait with the test and focus on collecting more data about user interactions within the app, their preferences, etc.

Rely on data to make a more accurate hypothesis. For example, check Google Analytics reports, send surveys to your customers, ask users for feedback, or conduct interviews with them. You can also consider the UX audit to find out if your solutions apply to standards in app design and don’t cause confusion among users. It will help you understand them better and give you the foundations to create hypotheses.

Case no. 5: There are many factors you want to check at once.

When it comes to testing different variations of the same change, then it’s okay. You just need to have more groups of users, separately for versions A, B, C, and D. However, checking many various factors at once isn’t recommended because the test won’t tell you what aspect caused changes in user behavior.

You can make several tests and check changes gradually, although it can be costly. In such a case, consider other methods that will allow you to check how changes in the app affect user behavior and whether the UI is intuitive. Usability testing may be your way to go.

Usability testing or split testing

A/B tests are a quantitative research method, so they tell you what happens in your app after making changes. However, only qualitative methods, such as usability testing, can explain why something occurs.

Usability testing of mobile apps allows you to check if your product is intuitive from the user’s perspective. Such sessions can also bring some problematic elements to your attention and enable you to make changes in a project to improve the app’s user experience.

What does usability testing look like?

During the testing session, a participant performs specific tasks in the app. They are often observed by a researcher, who can ask questions to better understand the participant’s emotions and reasons for their behavior. You can conduct usability testing remotely or in person.

For example, if you plan to add new app features, you can run usability testing first to determine whether users understand how they work and see their value.

This UX research method saves time and money. Why? It helps you find elements that require improvements before the mobile app development process starts. Writing code is much more time-consuming than testing, so it’s best to ensure you’re developing the proper version of your designs.

AB testing on mobile apps vs web apps

Whether you’re new to AB tests or you’ve already run them on web apps, starting them on mobile apps would be a new kind of experience. What should you know?

1. Some things that work on the web aren’t good for mobile — people use certain types of apps depending on their location and purposes. As a result, they may have different needs. So even if you tested a UI change in a web app and it was successful, it doesn’t have to be the case with a mobile app.
2. Mobile apps offer different possibilities — using a smartphone, you can tap a notification to open the app, quickly take a photo to search by image, or even browse some elements of the mobile app offline. These are examples of features you don’t consider in web apps although they might be useful on mobile solutions.
3. Feature flagging — in web apps, you add changes, and users see them immediately. In mobile apps, you have to upload them to the app store and wait for the review before they become available to users. You can speed up this process by configuring certain elements remotely, thanks to feature flags. It comes in handy in A/B tests because it allows you to switch features on and off as you need, without waiting for the review every time. Specific user segments can see versions A and B.

As a product owner, your job is to prepare a document explaining how feature flags must work in specific scenarios. It tells software developers how to program the tests.

If you’re interested in the technical side of the AB testing process, read our article about feature toggles.

Final thoughts on AB testing with mobile users

In certain cases, A/B testing may effectively help you optimize mobile apps. Teams can check various elements such as UI design, navigation, forms, copy, and features.

This method allows for data-driven decisions, resulting in improved user experience, higher conversions, and deeper insights into user behavior. However, to conduct valuable testing, you first must determine proper goals, select the right metrics to monitor and run the experiment long enough.

Do you need help designing and implementing the AB tests in your app? Contact our team.