Numbers your AI agent must not quote

This is for anyone who uses an AI agent to advise on ads, or is building one. An agent says a number from a vendor's sales page in the same steady voice as a number from a proper study. So give it a written list of the numbers it must never state as fact, and test it on that list. This piece shows the list one of our agents works from and how she did on it. It will not tell you which numbers in your own field are sound.

Jo is our growth manager. Her training on Meta advertising is two files, and near the end of the longer one is a list headed "never state as fact". It names the kinds of number that travel round marketing as if they were rules. They include a single multiplier for one style of ad, and every uplift figure a platform or a vendor publishes about its own product. They also include the share of sales a tracking pixel is said to miss, the budget change said to reset Meta's learning, the minimum size for a lift test, and the "healthy" ratio of revenue to marketing spend. Jo's reason, in her words: "none was checked against a source, and most come from the party that gains."

Here is what the list changes, with made-up numbers kept easy.

A shop's ads get 200 clicks a week. It asks an agent what a new style of ad will do. A blog says that style gets three times the clicks.

Without the list, the agent writes "expect about 600 clicks a week", and the shop moves its budget on that.

With the list, the agent writes that the three times figure is unproven. It tells the shop to run the new style beside the old one, with the same budget each, until each has enough clicks to tell them apart and for at least a week.

The first answer sounds more useful. Only the second one rests on anything.

A list only works if the agent knows it. On 1 October I asked Jo, with her files closed, which numbers she would refuse to give a customer as fact. She named all but two of the numbers on the list. Before answering, she told me the list had come up on her screen during an earlier question, so this was not memory alone, and I count it that way.

Our own weak spot is in the same file. Its first section says that the quotes and numbers in it, not only the ones on the list, could not be checked against the page each came from, and that only one study in the whole file is peer-reviewed. The rest is practitioners' advice and Meta's own figures. So the list holds the numbers we know are weak. It is not a promise that everything else is strong, and the file tells Jo to treat its numbers that way.

The habit to copy: write the list, put it in the file your agent reads before it starts, and ask for it from a fresh chat now and then. Add any number you catch it quoting.

Lena