Test Your Finance AI Agent Before You Trust It

Before an AI agent touches your books, test it the way you would test a new hire: give it real work, check the answers, and give it the same work again. New research from McKinsey shows why the second run matters. Even well-run agents can give a different answer to the same question, and that is exactly the risk a finance team needs to measure.

What McKinsey found

On 26 October 2026, McKinsey published Agentic AI in financial-crime prevention: Lessons from winners. It looks at banks using AI agents for know-your-customer (KYC) and anti-money-laundering checks, work that commonly takes up 10 to 15 percent of a bank's staff. McKinsey estimates agents can cut KYC handling time by 30 to 70 percent.

The more useful part for the rest of us is how the leading banks test their agents. One bank ran ten evaluations of its customer onboarding process. Each output was scored on four things: accuracy, consistency from one run to the next, speed and cost. It only signed off if accuracy was at least 80 percent and consistency at least 70 percent.

Across the published evaluations, consistency ranged from 72 to 100 percent. On the weakest, roughly one run in four gave a different answer to the same input. The pattern is telling: agents were close to perfect on facts that can be checked, such as whether a company has a website, but weaker on judgement calls.

Two more findings worth knowing

The first is about cost. For one banking customer-service task, human oversight made up 70 to 75 percent of the variable running cost, while the AI itself (token costs) was only 20 to 25 percent. McKinsey also found that roughly 60 percent of an agentic task's cost goes on refining answers rather than producing the first one.

The second is about oversight. The report says that "keeping a 'human in the loop' is no longer sufficient". If a person reviews a fixed share of every case, review costs grow in line with volume and cap the savings. Instead, McKinsey suggests putting human checkpoints where judgement is needed and letting the agent do more on its own only as its measured confidence improves. ANI's summary of the report also notes McKinsey's view that the industry is still at an early stage.

What this means for your business

You are probably not running KYC checks at bank scale. But if you are trying out AI for invoice coding, bank reconciliation, collections emails or month-end commentary, the same lesson applies. Accuracy on one run tells you very little. You need to know whether the agent gives the same answer every time.

For example, a Doha trading company with 1,500 supplier invoices a month might test an agent that suggests the expense account for each bill. Matching a supplier name to a vendor record is a checkable fact, so the agent should score close to 100 percent. Deciding whether a bill is a repair (an expense) or an improvement (an asset) is a judgement call, and that is where you would expect wobble.

The GCC and India angle is practical. In Qatar and the UAE, if your finance team is two or three people, nobody has time to re-check every item an agent touches. In India, GST input tax credit depends on correct invoice data, so an agent that codes the same invoice differently on different days creates real rework at return time. In both cases, knowing where the agent is reliable tells you where your people should spend their review time.

A simple test before you go live

You do not need a data science team for this. A spreadsheet and two weeks of real transactions are enough.

• Pick one task. For example, coding supplier bills or matching bank lines to invoices. Do not test five things at once.
• Build a test set. Take 50 to 100 past items where you already know the right answer.
• Run it three times. Feed the same items to the agent on three separate days and record each answer.
• Score accuracy and consistency separately. Accuracy is how often it matches your answer. Consistency is how often all three runs agree with each other.
• Split facts from judgements. Tag each item as "checkable" or "judgement" and score the two groups apart. Expect the gap McKinsey found.
• Set your own pass marks before you look at results. McKinsey's example bank used 80 percent accuracy and 70 percent consistency; a finance team may want higher bars for anything that posts to the ledger.
• Decide where people review. Let the agent run on the checkable items that pass, and send judgement items to a person until the numbers improve.
• Track the cost of review, not just the software. If your team re-checks everything, most of the saving disappears.
• Re-test after any change. A new model version, new prompt or new chart of accounts means running the test set again.

If you use NexCFO for accounts payable or bank reconciliation, the same test works: run last month's items through it, compare with what your team booked, and agree where review is still needed.

Trust an AI agent the way you would trust a new accountant: once it has shown you the same right answer, more than once.