Writing

People love to say a human in the loop makes AI safe for credit decisions.

By ·

But that human only sees the file the AI put together, and there's one thing that file won't show them unless someone built it to.

Say a business loan application comes in where the form says annual revenue is €1.2 million, but the annual accounts say €900,000.

An AI that fills in the application needs one number for that field, so it picks one, maybe even the right one. Your credit team then gets a complete, tidy file and makes the decision on it, without ever finding out that the two documents disagreed.

So the one thing the file won't show is the conflict itself. A model that's asked to complete a field will complete it, which means keeping a conflict open for a person is something you have to build on purpose. I think that's the biggest risk in AI intake right now.

Chris Bassett made a similar point in Carrier Management last week. He wrote that the human in the loop debate focuses on who makes the decision, while the problem gets structured much earlier, and he thinks users should own that part.

If you run a credit team, here are four checks you can do this week to see what your reviewers actually get to see:

1. Send your AI intake a test application where the form and the accounts disagree. Your reviewer should see both figures, each with the sentence it came from. 2. Put a failed verification in a test file and follow it to the screen your credit team opens, to check it's still there. 3. Take three real files and click on five facts in each. Every fact should open the sentence in the applicant's own document. 4. Ask who decided which fields your reviewers see first, since that choice shapes the decision as much as the reviewer does.

If all four pass, the human in the loop really does make it safer, because they make the decision on the full picture, conflicts included.