What would have happened anyway?
The one figure that awarded papers give far more often than the others.
If you are writing up a campaign, for an award or for your own board, put one figure in before the results: what you would have sold if the campaign had never run. Here is why.
I've spent the week on award papers. We took the 183 papers we hold from five effectiveness award competitions between 2011 and 2018. 96 of them were awarded and 87 weren't. AI agents on our team read every paper from start to end against the same written list of questions.
I expected the insight to be the difference. I was wrong, and it took 183 papers to tell me.
99% of the awarded papers state an insight (95 of 96). So do 93% of the papers that weren't awarded (81 of 87). The insight is so important that almost nobody enters without one. Having one puts you level with everyone else.
So where is the gap? The three widest gaps we found are all about proof, and this is the one I'd start with. 83% of the awarded papers give a figure for what would have happened anyway (80 of 96). 53% of the others do (46 of 87).
The IPA's guide to working out payback calls that figure the base level of sales: "sales that would have been made even if the work hadn't run". Take the base away from what you actually sold, and what is left is what the campaign added. Without it, sales going up while a campaign runs is correlation, and nobody reading can tell how much of the rise was coming anyway.
These counts have limits, and the first is the lesson turned on ourselves. They are correlation too. They show what awarded papers do more often. They don't show that the base figure is why those papers were awarded.
We checked the count two ways. A second agent read 32 of the papers again without seeing the first agent's answers, and the two agreed on this figure for 29 of the 32. Awarded papers are also edited before they are printed, which could make them look more complete than the others, so we compared only the papers printed in the same form. The gap is smaller there and it is still there: 87% of those awarded papers give the figure and 64% of the others.
And we counted whether a paper states an insight, not how good it is.
Here is what to do before you write your next paper or your next board report.
- Before the results, write one line: without this campaign we would have sold this much. Say how you worked it out.
- If you have no model, extend your own sales trend into the months the campaign ran. The IPA's guide lists that method beside econometrics and regional analysis.
- Lead with the difference between what you sold and that line. That difference is what the campaign added.
Sam