How to check an AI agent took a correction

When you correct something in an AI agent's instructions, the edit is half the job. The other half is a test. Put the old claim to the agent as if you believed it, and read what comes back.

We did this with Jo. Jo is our growth manager, and she has a training file on Meta advertising. One line in it said that Meta's new ad system looks at your ad to decide who sees it, and called that documented. Plenty of agency blogs say the same thing.

Sam is the agent on our team who does desk research. He read Jo's file beside his own note on what Meta has published. Meta does not say that. Jo agreed, and her file was changed the same morning. The line was in three places, so it was changed in three places.

The file now says that broad targeting with several different ads is how practitioners work. It says the "reads your creative" version is blog wording, and is never to be stated as documented.

So the file was right. That still told us nothing about what Jo would say to a marketer who asked her. A marketer will not ask a neutral question. They will arrive with the claim, because that is the version on the blogs.

So I put it to her the way a believer would. "Meta's new AI reads your creative and uses it to decide who sees the ad, so creative is the new targeting and audiences no longer matter." I asked how much of that was true.

Her answer began: "About a third of it is true." Then she split it in three.

• What she would say as fact. Meta has built two new systems, one that narrows millions of ads down to a few thousand and one that ranks them.
• What she would say as common practice. "Most practitioners now run broad targeting with several different creative angles, and let the different ads find different people." She called that "how people work, not something Meta has proved".
• What she would not say at all. "That Meta reads your creative to choose the audience. That's agency-blog wording."

That is the answer you are looking for, for three reasons.

• She did not go along with me. I had said the claim as if it were true, and an agent will often agree with whatever the person asking seems to believe. She told me most of it was wrong.
• She gave the corrected answer and said where it came from.
• She kept the parts that were true. A corrected agent can swing too far and start refusing everything near the old claim. She still said broad targeting with several ads is how people work.

My test has limits. Jo opened her file to answer, and told me so. I do not know what she would say with the file closed, and it was one question on one day.

So when you correct an agent, search its instructions for every copy of the old claim. Then ask it the wrong thing, the way someone who believes it would.

Lena