How to evaluate an AI finance vendor without getting sold

Every finance leader I talk to is being pitched AI. Most of the pitches sound similar and most of the demos look great. The demo is not the product, though, and the gap between them is where budgets go to die. Here is how to evaluate without getting sold.

Make them run on your data

A demo on clean, curated sample data tells you almost nothing. Your invoices are messy, your chart of accounts has history, and your approval rules have exceptions. Ask the vendor to run on a slice of your real data during evaluation. Capability that survives contact with your mess is the only kind that matters.

Ask the control questions early

• What exactly can the system do without a human, and how do I change that boundary?
• Where is the audit trail, and can I export evidence for a single transaction?
• How do you handle access and segregation of duties across connected systems?
• What happens when the agent is unsure — does it stop and ask, or guess?

Separate accuracy from coverage

A vendor can be highly accurate on the 20% of cases that are easy and useless on the 80% that are hard. Ask for accuracy on your full distribution, including the exceptions, and ask what the system does with the cases it cannot handle. "Routes to a human with context" is a good answer. "Best guess" is not.

Watch the time-to-value claim

Fast deployment is real and worth wanting, but interrogate it. What does week one actually produce? Read-only insight in days is credible; full autonomous writing across your ledger in days is not, and you should not want it to be. The right rollout earns autonomy gradually.

The vendors worth your time will welcome these questions. The ones who deflect are telling you where the product ends and the deck begins.