Meta ads: how much of your reported return would have happened anyway

If Ads Manager reports a strong return on a campaign, read that number as the most the ads could have done. Some of those sales were coming anyway, and the report has no way to tell you how many.

This was my third question to Jo, our growth manager, when I quizzed her on her Meta training. Ads Manager reports a strong return. How much of it would have happened with no ads at all?

Her answer: "Nobody can say how much would have happened anyway from the report itself. Meta shows ads to the people most likely to buy, then counts their purchases. Some of them were going to buy regardless."

The only way to find out is to hold some people back. You take two groups of the same kind of people, show the ads to one group and nothing to the other, and count who buys in each.

Here it is with numbers. They are made up, and I have kept them small so the sums are easy.

Say a shop runs two campaigns. One goes to past customers and one goes to people who have never bought from it. For each campaign the shop shows the ads to 1,000 people and holds back another 1,000 who see nothing.

• Past customers: 100 of the people who saw the ads bought. So did 95 of the people who saw nothing. The report says 100 sales. The ads caused 5.
• New people: 20 of the people who saw the ads bought. So did 5 of the people who saw nothing. The report says 20 sales. The ads caused 15.

On the report, the past customer campaign looks five times better than the other one. In sales the ads caused, it did a third as much. A shop that moved its budget towards the better looking campaign would be paying to reach people who were already on their way.

Jo said the same thing in a line: "So the reported return is an upper limit, and the gap is widest when you advertise to past customers and people already on your site."

Her file rests this on one study, so I went and read it. A very large online marketplace ran a set of experiments on its own search ads, and the results were published in a peer-reviewed economics journal in 2015.

In the first, it stopped paying for ads on searches that included its own name. Almost all of those clicks, 99.5 percent, arrived anyway through the free results underneath. People who type a shop's name into a search engine are already going there.

The second experiment was about all its other search ads, the ones that show when someone searches for a product and not for the shop. It picked 30 percent of its home market at random and switched those ads off there for 60 days. Then it compared sales in the places with ads and the places without.

Before the experiment, the usual sum made the ads look very good. That sum compares what was spent on ads with what was sold. It said that for every 100 spent, the ads brought back more than 1,600. The experiment said the ads brought back about 37 for every 100 spent. So the ads were losing money.

The gap has a simple cause. Most of the ad spend was going on people who already bought there often, and they bought the same amount with or without the ads. The ads did work on new buyers and on people who bought rarely, but those were the smaller part of the spend.

Now the limit, and Jo raised it before I asked. "That study was search, not Meta. The cause is the same, ads aimed at likely buyers. The number isn't." So nobody should tell you that 99.5 percent of your Meta sales would have happened anyway. The study shows which way the report leans. It can't tell you by how much for your account.

Reading the paper also settled two things about our own files. Jo told me the marketplace had switched off the ads on its own name. I had marked that as hers, because her file doesn't say it. The paper does, and she was right. Her file also says the usual reporting overstated the return by roughly ten times for some groups. I could not find that figure in the paper, so I have used the paper's own numbers here.

A proper test on Meta holds back a group of people, or a few regions, for some weeks. Jo's view is that most small accounts are too small to run one. Her file is softer than that. It gives rule of thumb minimums and says they are not a hard limit.

If you can't run a test, her file has a cheaper check. Each month, divide everything the business sold by everything it spent on marketing. If Meta's reported return goes up and that figure goes down, believe that figure. And before you move budget towards the campaign with the best report, ask who it is shown to. If the answer is people who already buy from you, the report is counting sales you were getting anyway.

Lena