Simple Mobile App Split Testing with Facebook Analytics

This is part of a free course that teaches game design and the Swift programming language. It contains examples from a real game I created called ForeverMaze. The full course includes the source code for the whole game.

Fancy A/B testing suites are great, but sometimes you need something dead-simple. Facebook analytics has all the power you need for basic split testing in your mobile app. In this article, I’ll show you how I set up 3-way split test on the difficulty of the mobile game ForeverMaze.

Download the Source Code for Analytics.swift

Facebook Analytics: Treatment Assignment

To “assign a treatment” means to decide which of the scenarios a user sees. In my case, I was building an experiment around difficulty, so the treatments were easy, medium and hard. I wanted a function which would allow me to do this:

let treatment = Analytics.getTreatment(“difficulty”, treatments: [“easy”,”medium”,”hard”])

The first requirement is that the function should always return the same treatment for the same user. Since this is a dead-simple implementation, I decided to store the treatment in NSUserDefaults . This has the disadvantage of meaning that re-installing the app clears the treatment. A more sophisticated system might put the treatments assignments in a server K/V store based upon the user ID. This felt like over-engineering for my case, though.

Assuming that a treatment is not already assigned, the code simply picks a random treatment. Aside from saving this treatment to the NSUserDefaults , it also fires an event to Facebook Analytics about the treatment assignment. In the next section, we’ll see how this lets us do segmentation and analysis.

As a final note, the “cache key” used for storing treatments is the combination of the experiment name and available treatments. This has the intentional side-effect of making it so if I change the treatments in an experiment, it re-assigns all treatments and restarts the experiment.

Facebook Analytics: Determining the Results

Now that the events are being fired, it’s time to set up some “segments” in Facebook Analytics. A segment will let us look at data for just a subset of our users. So we’ll create a segment for each of the different treatments.

Difficulty Segments in Facebook Analytics

Facebook Analytics Segmentation

Now that we have segments, you can apply them to any Funnel or Cohort. For example, if I want to see how difficulty impacts the retention rate of the game, I can select the Install to Launch cohort. This cohort is built by Facebook and included in every app. I just choose a segment to see how that treatment group is doing:

Install to Launch in Facebook Analytics

You can define your own funnels and cohorts based upon any app events you want. Thus you can inspect the impact of your experiments on any imaginable stats in your app.

Limitations of This Approach

This isn’t a full-featured split testing platform. It can’t tell you when your experiment is statistically significant, for example. You also cannot compare performance in a side-by-side manner.

I find it most useful for fine-tuning a user-onboarding experiment. You can set up an onboarding funnel and create experiments for different tweaks to the onboarding and see how it impacts drop-off rate.

Limitations aside, this technique has served me well in two different games now. Since it’s so easy to implement, it makes the barrier to creating new experiments very low. This helps me iterate on development faster.

Do you have any suggestions for improvements? Let me know, below!

If you enjoyed this post, why not enroll in the free course? It contains hours of video content, source code downloads, examples and more. I cover many different intermediate and advanced topics for creating a real-time game in Swift with SpriteKit and Firebase. You can check out the complete game at ForeverMaze.com.

[author]

Leave A Comment

You must be logged in to post a comment.

Back to Top