← back to the front page
Primer

AI in regulated lending: what actually has to be explainable

You can't make a large language model faithfully explain its own decisions. So don't put it on the decision.

“Here's my reasoning” from an LLM is post-hoc narration — not the actual cause of the output, not reproducible, not testable. In a domain where every action must be explained as a matter of law, an architecture that asks the model to justify itself and then trusts the answer is built on sand. The fix isn't to make the model explainable. It's to architect so the model is never on the decision that has to be defended.

What the rules actually require

Two anchors define the surface. First, model governance: the Federal Reserve and OCC's SR 11-7 (2011) requires disciplined development, independent validation, and governance across a model's whole lifecycle. It is technology-neutral — an LLM in a credit process is a model, and it is in scope.

Second, borrower-facing explainability: the Equal Credit Opportunity Act and Regulation B require a statement of specific reasons for any adverse action. And CFPB Circular 2022-03 is blunt about the AI question: creditors cannot hide behind a “black-box” model. The complexity of the technology is not a defense — if you can't produce specific, accurate reasons, you can't use the model to make the decision.

The leak nobody budgets for

Move the decision into deterministic, auditable code and you've solved the visible problem. But you've pushed the risk to a seam: the LLM→deterministic boundary. If a model extractsthe income figure your underwriting engine then uses, the “explainable” decision rests on an unexplainable extraction. Confidence thresholds, abstention, and human verification have to live at that handoff — not bolted on after the number is already in the engine.

What good looks like

  • The LLM drafts prose and normalizes documents. It never places the numbers or makes the call.
  • Pricing, eligibility, and adverse-action reason codes are deterministic, validated code a regulator can read.
  • Every action writes one append-only, policy-version-stamped row — a decision log that's replayable against the rules in force at the time.
  • Below a confidence threshold, the system abstains and escalates to a human underwriter rather than shipping a weak decision.

A working demonstration

This site runs that exact pattern on a low-stakes surface. The daily brief lets an LLM write prose while code places every number; a deterministic gate decides whether to publish or hold; and every action lands in an append-only, Bitcoin-anchored public ledger. It even keeps its own score. It's a proof of pattern, not a compliance product — but the seams are the same ones a lender needs.

Common questions

Can you use a large language model to make a lending decision?
Not as the decision-maker. Under ECOA and Regulation B, a creditor must give specific, accurate reasons for adverse action, and CFPB Circular 2022-03 says the complexity of a “black-box” model is not an excuse. The workable pattern is to keep the decision in deterministic, auditable code and use the LLM only for drafting and normalization around it.
What is SR 11-7 and does it apply to AI models?
SR 11-7 is the Federal Reserve and OCC's 2011 Guidance on Model Risk Management. It requires disciplined model development, independent validation, and governance across a model's lifecycle. It is technology-neutral: an LLM used in a credit process is a model and is in scope.
What is the LLM→deterministic boundary?
It's the handoff where a language model extracts or summarizes an input (say, an income figure) that a deterministic engine then uses to decide. If the extraction is unexplainable, the “explainable” decision rests on it. Confidence thresholds, abstention, and human verification have to sit at that boundary, not after it.
How do you keep AI-assisted lending explainable?
Demote the model to a draftsman: it writes prose and normalizes documents, never places the numbers or makes the call. Put pricing, eligibility, and adverse-action reason codes in deterministic validated code. Keep an append-only, policy-version-stamped decision log. Abstain and escalate to a human when confidence is below threshold.

Sources

Written by Ryan Colkitt, who builds AI for regulated lending. Want the full transfer — the seam-by-seam build and the model-risk artifacts? Get in touch.