If this newsletter was forwarded to you, subscribe here to get these numbers in your inbox every week. 

🧀 In this issue: a hefty dose of realism about the new hot testing technique - it's neither as good nor as bad as you think.  

Holdout testing, so hot right now

We get it: incrementality testing is in vogue. 

Every other week someone’s launching a holdout test, yanking Meta spend, measuring lift. Wild claims like “we turned off our Meta spend entirely and our business actually GREW.” LinkedIn goes wild. 

Done right, incrementality testing is powerful, but there are downsides.

You isolate regions, channels, or audiences, hold out your spend, and measure true causal lift. It’s slow. Expensive. Laggy. But it can help refine your strategy, if you already have a solid data foundation.

(Ignoring the fact that most marketing efforts have a delayed “lingering” effect, meaning when you cut your spend, the marketing you were running prior to the cut can still impact conversions for months after.) 

Even after you’ve turned off your marketing, your ads still linger in the minds of your prospects. Google “ad decay rate.” This delay can make your spend look non-incremental. In reality you’ve turned off the tap of new customers – it just takes a while for the conversions to stop flowing.

Let’s talk about the real reason you’re testing:

You don’t trust your attribution.

And honestly? Fair.

If you’re relying on in-platform numbers, last-click attribution, or duplicate conversions across multiple tools… no wonder you’re skeptical.

But here’s the problem: incrementality testing is not a solution to bad data.

It’s a stress response.

It starts when your attribution system doesn’t inspire confidence. CAC spikes. Meta’s numbers don’t match Shopify. Last-click feels hollow. So what do you do?

You hold out regions. You pause channels. You spend weeks waiting for lift – or lack thereof.

In marketing, guesswork gets expensive fast.

Tossing dollars around without knowing what’s actually driving new revenue and profit? That’s how you end up scaling the wrong thing. You don’t need more data. You need to know how to slice the data you already have to get answers that move the business forward.

A technical definition of incrementality  

Incrementality aims to prove causation by comparing exposed vs control groups—but it comes with a cost. Tests are slow, expensive, disruptive, and often underpowered. If you're running them on small audiences, your results won’t reach statistical significance, and you’ll end up no smarter than before.

 
Once you have good data, incrementality becomes a powerful calibration tool. 

But if what you really lack is reliable performance data, the last thing you need is another control group. What you need is a better attribution foundation.

The smarter path: MTA first

MTA should be your first move.

MTA doesn’t need holdouts. It gives you granular, real-time insights across Meta, Google, TikTok, Shopify, far beyond last-click. And when built on clean, deduplicated first-party data, MTA isn’t just viable. It’s elite. 

Most businesses jump to incrementality because their MTA is weak. They lack accurate conversion deduping, reliable revenue mapping, creative experimentation plans. So they throw a holdout at the problem. The result: expensive, slow, and confidence-less tests.

Sometimes you have to run incrementality tests for six to eight weeks before getting insights. Not counting how long it takes to create the reporting. 

Fix the attribution stack first. Then let incrementality confirm what your MTA already proves. That’s how leading ecommerce teams operate Northbeam.

Step by step:

• Start with MTA. Build a clear, consistent view of cross-channel performance, no holdouts needed.
• Clean data. Merge clicks, views, conversions, revenue, deduplicated and tied to actual purchases.
• Run incrementality when you need validation, not because your data feels broken.
• Use test results to refine your models, including MMM and longer-term forecasting.

Northbeam’s engine was designed exactly for this. We unify multi-touch data, remove duplicate conversions, and tie everything back to revenue. Run clean MTA, then layer in incrementality tests only when they add value.

So book a demo already. 

🚫 Shopify blocked AI agents from making purchases. Without a final step of approval, these tools can't automatically purchase from Shopify stores. 

🔎 Instagram content now shows up in Google Searches. Here's how to optimize for that - get your organic teams in action! 

📌 Pinterest published a report on men's search trends. Find out what dudes are looking for on the app. 

Love this email? Please forward it to your friends, coworkers, or Cindi Rose. Subscribe here to get this data in your inbox every week.