Follow this document here. Based on conversations that I have had with Leah, #1 and #2 are our recommended strategies to get the most consistent results. If the client has a substantial program (more than 30K per month) you can look at individual tactic level testing. Follow #4 and #5, the main takeaway is to exclude PMAX from test and control markets when testing non-PMAX campaigns.

See this slide here. There is no way to add geo exclusions to Advantage+ campaigns, meaning we cannot test Advantage+ campaigns in an MMT test. Additionally, running holdout tests on any non-Advantage+ Meta campaigns while running Advantage+ campaigns will interfere with results. We recommend only running growth tests on Meta while running Advantage+ campaigns.

See this slide here. TikTok Ads MMTs have historically run as Growth tests due to platform exclusion limitations; however, a recent update allows Holdout tests to be run on TikTok. The TikTok team has developed this resource for their team that leads to proper implementation.

You can run a conversion based TikTok campaign, but we do not recommend doing that. Historically, we have not seen conversion TiikTok campaigns come back as incremental.

Intuitively this makes sense, the more the platform’s algorithm tries to optimize towards people that have purchased in the past, the more it targets people that are going to purchase no matter what we do (the definition of non-incrementality, see this slide here).

Rather than getting in front of people that are already going to purchase, reach and frequency focused TikTok campaigns with rigid frequency and audience targeting often come back incremental.

Growth tests are a balancing act, we need to increase the budgets enough so that we can significantly detect the effect on revenue from increasing spend, but not too much where we run into diminishing returns. Budgets are recommended by the planning team based on the criteria here.

If you are worried that the platform is close to diminishing returns then you can try to adjust the set up of the test to be a holdout test (when possible). Growth tests are only recommended in two scenarios, when the channel is net new and when holdout testing is not possible (think Adv+). We can always have a discussion around the budgets if that is a concern before the test launches.

In general, as long as the change affects both test and control markets evenly (creative refresh, nationwide sale, month-over-month budget changes, etc.) then that should be fine. If the change will effect test and control differently (see these case studies) then we need to figure out how we can postpone the change or mitigate the negative impact on the test. When in doubt, reach out to the person who did the market selection to figure out if the optimization will effect the test.

See slide 1 and 3 here for a description of the three analyses.

MMT is used to validate channel/tactic incrementality in a rigorous geographic experiment. I.e. how incremental is PMAX?

CIA is used to look at how an individual decision impacted a variable. I.e. what was the impact of the iOS 14.5 update on revenue?

LRA is an analysis that can look at the relationship between multiple variables over a long period of time and is not bound by one start date. I.e. what happens to revenue as facebook spend changes?

See this slide here, the example is very helpful in understanding how the confidence is calculated.

If the iROAS estimate is close to 0 or the confidence interval around the estimate is too large, this will lead to low confidence. In that case as long as we are confident in the implementation of the test, it is an indication that the channel or tactic being tested is not incremental.

Not every test ends in incrementality, and because incrementality is a prerequisite for confidence, not every test ends in confidence.

The talk track for reaching statistical significance is this: as we adjust spend, if revenue responds similarly to our changes in spend, we are rewarded with confident predictions. The more that we change spend and revenue does not respond to our changes it indicates that there is no incrementality, leading to less confidence.

The MMT team are not ad strategists, and don’t have the background in Meta/Google/etc to check. The best QA’s I have seen in the past are when the person who added it to the platform has someone from their department that has done an MMT before check their work to ensure proper setup.

I think it would also be great for more teams to make an internal resource like the TikTok team, I am working with Sarah M. to set those up with other departments.

Another item that helps to ensure the proper set up is a test brief developed by the AD’s, this sets up clear expectations internally and with the client.

The last note I will make is just encourage any team members running the platform that do not have much MMT experience to reach out to the DI team. The more we can educate people and give them resources like the MMT guidebook and have them truly think about and understand what an MMT is the fewer mistakes will happen.

Like mentioned above, anything that affect the test and control markets unevenly or improper test set ups. Also see this slide here about outliers.

The lag in conversions is factored into the experiment based on the channel being tested. A Facebook ads test should only take 4 weeks to see results, while a programmatic test could take 6-8 weeks. As long as the test covers atleast 2 consideration cycles (which for most of our clients is 4-8 weeks) then the test is set up properly.

Measurement is started the day that spend is changed in the test market, there is not other day where measurement can start. The period to compare to is determined by the MMT team, we normally look at the past year of data to construct the counterfactual (control) estimate which we compare the real test market results to.

It was established by the TikTok team during the middle of the test that reach and frequency campaigns led to incrementality, causing the switch in order to see better results.

It was established by the TikTok team during the middle of the test that reach and frequency campaigns led to incrementality, causing the switch in order to see better results.

It was established by the TikTok team during the middle of the test that reach and frequency campaigns led to incrementality, causing the switch in order to see better results.

It was established by the TikTok team during the middle of the test that reach and frequency campaigns led to incrementality, causing the switch in order to see better results.

It was established by the TikTok team during the middle of the test that reach and frequency campaigns led to incrementality, causing the switch in order to see better results.

We used unique MMT measurement frameworks and were able to see very compelling results, so in my eyes this test was a success despite many hurdles, we took a longer test approach and were able to account for the seasonality of more deals coming in at the end of the month.