Banks already have a method for checking AI works. Four steps to steal it from their model-risk teams
Jamie Watters
Operational resilience and AI delivery practitioner.
Who this is for. Anyone who has been asked to show that an AI pilot works. Bank model-risk teams have run this test for years, and you can steal it whatever business you're in. By the end you'll have four steps, each traced to a paragraph of the regulator's text, and one sentence to take to whoever checks your work. If you're in a bank, don't be surprised if the dots were never joined up: your AI programme and your model-risk team have probably never met, and the second section says why.
Skip it if your pilot already has a challenger, a parallel run against real outcomes, a monitoring pack and a named owner. Nothing here will be new to you.
On the length. Eleven minutes. The regulator's five expectations and the four steps are the first two thirds. The last section is what the rules don't give you.
Somebody has asked you to show that the AI pilot works. You've been looking for a method: a vendor's dashboard, a consultancy's framework, a blog post. If you work in a bank, or advise one, stop looking. The method is written down already, by the regulator, for every model a bank with internal model approval uses, capital models among them. It sits with the model-risk team two floors down: the second line, the people who check the work of those who build and run the models.
From 2023 until I left this autumn I sat on the committee at a bank I worked for that approved AI and cloud initiatives for development, implementation and deployment. A subject-matter expert from each discipline signed off their own area. I was a member of that committee, not the designer of its process. In the same role I did around 25 critical vendor reviews a year. What follows is the test that team already runs, taken from the regulator's own text, and the four steps to borrow it for a pilot that nobody would call a model.
What the regulator already expects of a model
The Prudential Regulation Authority's supervisory statement on model risk, SS1/23, was amended in April 2026 and the amended text is public. It defines model use as "using a model's output as a basis for informing business decisions", and a model's output as "estimates, forecasts, predictions, or projections" (PRA, SS1/23 as amended, paragraphs 2.1 and 2.3). An AI tool that triages tickets, drafts a decision or predicts which invoice is wrong is producing a prediction that informs a decision. It is a model by that definition. The PRA said as much when it published the statement: the definition "intends to ensure that recommendation systems in client services and other AI/ML that deliver qualitative output are within the scope" (PRA, PS6/23, May 2023, paragraph 2.23).
Here is what the statement expects a firm to do with one.
Compare it with a challenger. "Where practicable", performance tests "should also include comparisons of the model output with the output of available challenger models, which are alternative implementations of the same theory, or implementations of alternative theories and assumptions" (Principle 3.3(b)(iii)). The rule on keeping a model inventory adds, in a footnote, that a decommissioned model "may be used as a challenger model itself" (Principle 1.2, footnote 6).
Run old and new against real outcomes. Testing of a material change "should include a comparison of the model output prior to the change and the corresponding output following the change to actual observations and outcomes (ie parallel outcomes analysis)" (Principle 3.3(c)). That paragraph names "dynamic models (ie models able to adapt, recalibrate, or otherwise change autonomously in response to new inputs)", which is a fair description of a system that learns.
Keep monitoring after go-live. Monitoring "should be performed to assess model performance against thresholds for acceptable model performance based on the model testing criteria used during the model development stage" (Principle 4.4(a)). It runs "with a frequency determined by the model tier" (Principle 4.4(d)), the tier being the rating a firm gives each model for how much rides on it. It includes "analysis of overrides", which the statement defines as evaluating "the performance of model adjustments made" (Principle 4.4(c)(iii)). For a vendor's model, the firm should "validate their own use of vendor products and conduct ongoing monitoring and outcomes analysis of vendor model performance using their own outcomes" (Principle 2.6(b)(iii)).
Name two people. The model owner is "the individual accountable for a model's development, implementation and use, and ensures that a model's performance is within expectation" (paragraph 2.12), and "Model owners should be documented for all models" (Principle 2.4(c)). Above every owner sits one senior management function, which is a named person the regulator can hold to account. That person answers for the whole framework: "The relevant SMF(s) should be the most senior individual with the responsibility for the risks resulting from models operated by the firm" (Principle 2.2(a)).
Keep the checker independent. "The validation function should operate independently from the model development process and from model owners" (Principle 4.1(d)), with "sufficient organisational standing to provide effective challenge" (Principle 4.1(e)).
The method, in the regulator's words. Source: PRA SS1/23 as amended, April 2026.
Read those five against the AI pilot on your desk. Is there a challenger, even the old process? Were old and new run side by side against what actually happened? Is anyone measuring it now, against the number written down before it was built? Who owns it by name? Who checked it who did not build it?
Why nobody told you this applies
Because, on the letter of the guidance, it may not. SS1/23 applies to UK banks, building societies and designated investment firms "with internal model approval to calculate regulatory capital requirements". Its expectations "do not apply to firms which do not have permission to use internal models", though the PRA "considers that those firms may find the proposed principles useful" (paragraph 1.2), and a footnote adds that "vendors and external consultants may find this supervisory statement useful". The April 2026 amendment added that the expectations "are not conditions for internal model approval", and the PRA said the change was "clarifying the existing policy, and not changing it" (PRA, LIAF01/26, April 2026).
The US has moved the other way on AI. On 17 April 2026 the Federal Reserve, the OCC and the FDIC issued revised model risk guidance, SR 26-2. It "supersedes and replaces" SR 11-7 from 2011. It is "expected to be most relevant to banking organizations with over $30 billion in total assets" (Federal Reserve, SR 26-2, 17 April 2026). Its footnote 3: "Generative AI and agentic AI models are novel and rapidly evolving. As such, they are not within the scope of this guidance." The next sentence keeps them in view: a firm's own risk management and governance practices "should guide the determination of appropriate governance and controls for any tools, processes, or systems not covered in this document". The principles still "apply to traditional statistical and quantitative models and non-generative, non-agentic AI models" (the attached guidance). The same document keeps the idea worth stealing: "effective challenge", which it defines as "critical analysis conducted by objective experts who evaluate model risk and effect appropriate changes throughout the model lifecycle".
Who the guidance binds. Sources: PRA SS1/23 and LIAF01/26; Federal Reserve SR 26-2 and its attachment.
So a chatbot pilot in a UK bank without internal models, or a generative tool in a US bank, is outside the letter of both, which may be why your AI programme and your model-risk team have never met. It is not because the regulator has been silent. When the PRA published the statement it reported respondents "in broad agreement that the principles are sufficient to identify, manage, monitor, and control the risks associated with AI/ML models" (PS6/23, paragraph 2.19), and in October 2025 it put its own findings on AI models under the statement to a roundtable of firms. Scope is not a reason to leave the method on the shelf. The regulator wrote it because it is how you find out whether a model is doing what you think, and that question does not care which regime the model falls under.
Four steps to borrow it
Here is the method, cut down to what an AI pilot needs. Each step is a paragraph of the supervisory statement, translated to a pilot. The translation is mine; the references are there so your model-risk team can check it.
-
Write the design objective and the test before you build. The statement wants every model to have "a clear statement of purpose and design objective(s)" (Principle 3.1(a)). It wants "clear criteria (tests) as a basis to measure a model's quality" and the outline of the monitoring pack, the set of tests that will run after go-live, set out at development (Principle 3.3(a)). For material models it wants "a maximum tolerance on model performance exceptions" (Principle 5.3(a)(ii)). For a pilot: the one number it is meant to move, the baseline today, and the result that would make you stop. On paper, dated, before the vendor demo.
-
Name the challenger and run them side by side. The challenger is usually the old process, or the team doing it by hand. Run both on the same cases for a period and compare each against what actually happened, not against each other's opinion. That is the nearest a pilot gets to parallel outcomes analysis (Principle 3.3(c)), which in the statement compares a model before and after a change against real outcomes. For a tool that drafts text rather than predicts a number, the real outcome is a structured review of a sample of its drafts by someone who knows the work. If the pilot cannot be run alongside the old way, you have found your first risk.
-
Keep the monitoring pack running, overrides included. The tests you chose at the start keep running after go-live, against the thresholds you set (Principle 4.4(a)), at a frequency set by how much rides on the tool (Principle 4.4(d)). Record each time a person overrides the AI and check whether they were right; the statement calls that analysis of overrides (Principle 4.4(c)(iii)). A vendor's tool gets the same treatment, "using their own outcomes" (Principle 2.6(b)(iii)).
-
Name the owner, and keep the checker independent. One person owns the pilot by name and answers for its performance (paragraph 2.12). Whoever checks the results did not build the tool and does not report to whoever did (Principle 4.1(d)), and the monitoring reports are "independently reviewed" (Principle 4.4(e)). In a small firm that can be one person from another team who understands the process and can take a problem upwards. It cannot be the vendor.
Those four are the outcomes half of the statement, cut for a pilot. The whole discipline also keeps an inventory of every model (Principle 1.2), tiers them by how much rides on each (Principle 1.3), has the design independently reviewed before use (Principle 4.2) and documents all of it. Borrow the four first.
The four steps, each traced to a paragraph of SS1/23.
What the rules do not give you
Three things. The first is a measured answer to whether any of this works for AI. The practice is required for capital models and common in software engineering; the AI deployment studies I read do not measure it, so treat it as the control that is missing from most pilots, not as proof. The second is a test of the people doing the checking. Every paragraph above tests the model. None tests whether the reviewer still catches the model's mistakes once it is right most of the time, and that is where the evidence says the risk sits. I wrote up how to test the checkers separately, and the full method, from choosing the process to measuring after go-live, is in the implementation paper. The third is a warning from the firms themselves. When the Bank of England summarised its AI roundtables in February 2026, several participants said firms' traditional model-risk approach to validation "wouldn't be sustainable in its current form" as generative and agentic systems spread (Bank of England, summary of AI roundtables, 16 February 2026). The four steps are the part that still holds whatever replaces it: a stated objective, a comparison, a measurement and a named person.
If you take one sentence to your model-risk team, make it this: treat the pilot as a model, give it a tier, and run the monitoring pack. They will know what you mean, because the regulator wrote it down for them in May 2023 and clarified it in April 2026.