AI-powered ad management tools can spot issues faster than any human, but their judgment is only as good as the data and context they receive. Discover why automation alone isn’t enough-and what agencies are learning the hard way.
When an AI ad agent flagged a major performance drop before the team even logged in, it showed its speed. But just months earlier, the same system had confidently called a crisis-simply because conversions hadn’t posted yet. This gap between fast alerts and real understanding is now the main challenge for agencies relying on automation to manage paid media.
Starting in March 2026, Meta will require advertisers to explicitly label all AI-generated or AI-modified ad content, including images, video, and text, as part of new transparency rules.
Building trust in AI judgment
Early versions of Terry flooded the team with noisy alerts-flagging every anomaly, but missing the business context that human analysts take for granted. For example, the AI would panic over a lack of conversions, not knowing that financial services leads often take weeks to mature. Engineers spent months refining conversion windows, masking immature data, and adding rules to keep the system from guessing when patterns were unstable.
Recent industry analysis highlights that the most effective AI guardrails now involve inspecting not just prompts and responses, but also the arguments and results of every tool call made by the agent. This granular approach helps prevent both security and operational errors, ensuring that AI agents do not act on incomplete or misleading data.
Data quality and business context
AI’s analysis is only as good as its inputs. If conversion tracking is shallow or doesn’t match actual revenue, the agent will confidently analyze bad data and produce misleading conclusions. The same goes for business context: unless the AI has access to the same information human analysts use-like what products are sold or how sales pipelines work-it will miss important details. As one agency leader put it, you wouldn’t hire a new team member and then refuse to tell them what the business does.
This lesson is common across the industry. As seen in recent analysis of AI-driven referral traffic, the value of automation depends on how well systems are integrated with real business data and workflows. Without that, even advanced models can undermine campaign performance.
Engineering for reliability
Much of the engineering effort has focused on reducing noise and preventing overpromising. Early on, Terry would send duplicate alerts, panic over campaigns that simply hit their daily budget, or claim it could perform actions-like updating documents-that it wasn’t able to do. The team responded by teaching the system when to stay silent, enforcing honesty in its responses, and flagging open questions it couldn’t answer.
Automation remains strictly limited. Terry operates in read-only mode, making recommendations that humans must review and implement. Only after the system’s judgment is proven reliable will the agency consider letting it make changes directly. The lesson from a decade of Smart Bidding and flawed conversion data is clear: automation without accountability just speeds up mistakes.
The road ahead
The agency now imagines a future where Terry proposes precise changes with supporting evidence, escalates high-risk actions to humans, and automates only low-risk adjustments within strict budget and strategy limits. Full automation is still out of reach-and probably not desirable. Instead, the role of paid media managers is shifting from hands-on analysis to directing, judging, and holding AI systems accountable.
In the end, the difference between a false alarm and a real catch wasn’t a smarter model, but better data, tighter alerts, and engineered honesty. For agencies and publishers, the message is clear: AI can amplify results, but only if humans build the systems that keep it honest and grounded in reality.
Semrush, a leading provider of digital marketing analytics, reported over 95,000 paying customers as of 2025 and continues to expand its suite of AI-powered tools for agencies and brands. The company’s paid search analytics platform is widely used by marketers to benchmark performance, analyze competitors, and identify new growth opportunities in a rapidly changing ad market.
