Learning Through Real A/B Testing with Ad Studies

Ian Janes Apr 25 2016 8 AM | 3 min read

One of the great things about Facebook advertising is how effectively the platform optimises for your objectives right out of the box. But to get further ahead, Facebook advertisers need to understand their key success metrics, and this is much easier when you’re able to make statistically valid comparisons between various tactics and strategies. The Ad Studies feature is designed to make such comparisons easy by sorting members of your audience randomly into different study groups. In this post we’ll take a closer look at this feature and why it matters, before discussing best practices and some examples.

In recent years A/B testing has become an extremely popular tool for conversion optimisation, but it’s very easy for small mistakes to corrupt a study and lead to wrong conclusions being drawn. Before Facebook launched its Ad Studies feature, certain comparisons were almost impossible to make in a statistically valid way.

Let’s say you wanted to test different creatives against each other, for example. Facebook would start to give more delivery to individual ads within an ad set before there was sufficient data for you as the advertiser to reach any meaningful conclusions of your own. Splitting each ad into its own ad set wouldn’t solve the problem either, because then Facebook’s de-duping algorithm would intervene and favour the ad sets with the highest quality score, so again the delivery would not be even. You could force Facebook not to optimise at all by only running one ad at a time, but this doesn’t make for a valid study since any differences in performance could be due to seasonality.

Enter Facebook Ad Studies

This API-only feature allows advertisers to split the universe of Facebook users randomly into different groups. So in our example above, the same targeting rules would then be applied to each group, and each test audience would see a different creative during the same period. There’s no audience overlap, no seasonal distortion, and nothing to cause uneven delivery between different creatives (provided your budgets are proportional to the audience split sizes).

Your work isn’t done yet, though. For example, you still need to make sure you have enough data so that your results are statistically significant. Depending on your situation, this might require some patience as you wait for your ads to get enough conversions. You should also refrain from testing too many variations at once, as this increases your risk of false positives. And be very careful about testing more than one hypothesis at a time, since this can very easily skew your results.

Not just for testing content

There are many other things you can test with Ad Studies besides different content strategies. You can use the feature to test your approaches towards both bidding and budget allocation. For example, does bidding $50 per purchase get better results than bidding $10 per link click for the same campaign? Can your manual budget allocation strategy beat Smartly’s Predictive Budget Allocation feature?

You can even gain insights into how partially overlapping audiences differ from each other. Say you’ve selected different targeting options that partially overlap, for example people who like cats and people who like dogs. If you then create a 50-50 split, the overlap (people who like both) will be split perfectly, but half of each non-overlapping section is lost. In a way, you’re sacrificing reach in exchange for insight:



There are so many possible hypotheses which could be properly tested using Ad Studies. What questions do you have about your own tactics and strategies that you’d like to have answered? Let us know in the comments below, and remember: Always Be Testing!


   Retail Re-UP: Adjusting to the Next Normal of Social Advertising CHECK IT NOW
Ian Janes

Read Next