Every vendor shows you a demo where their AI reads every document perfectly.
By Mike Cooper ·
I think that tells you very little, since they picked the documents themselves.
Your applicants send whatever they have on their phone, so I think you should always test on your own files before you buy.
For my MSc thesis I tested deepfake voice detectors on compressed audio (the kind WhatsApp and phone calls produce). That's what a detector actually gets in real life. Document AI has the same problem, since it gets tested on clean PDFs and then has to read phone photos of bank statements.
So I would test it like this.
Use your own files to test the quality, never their samples. Pick a small subset of real (perhaps anonymised) scenarios, like a blurry photo, a scan with a stamp over the numbers, or a statement in Dutch and English.
Score only the fields your team acts on, such as revenue or finding a name. A file can look right and still have the wrong revenue, and that's the number your team decides on.
Then run the same file k(=3) times to see if the output stays consistent. If the runs give different answers, your team can't trust any of them, meaning they end up checking every file by hand again.
Also always check what happens when they update the model, since the same file can be read in a new way after that (most of the time new models perform better, but exception are still seen).
How do you test a tool like this before you buy it?