By 2030, many firms could be operating AI across several functions at an approver level or beyond, according to the Financial Conduct Authority’s (FCA) Mills Review. Host Sebastian Barling and his colleague Deborah Kirk, who leads Skadden’s intellectual property and technology transactions practice in London and hosts the SkadBytes podcast, examine what that shift means for banks and financial institutions. They unpack the review's five-level autonomy spectrum and explain why traditional oversight and point-in-time validations break down as AI becomes more autonomous. Sebastian and Deborah also outline four compliance priorities and explore systemic risks created by reliance on the same models and providers. Tune in to find out what questions institutions should ask to prepare for agentic finance.

Connect and Learn More

☑️ Deborah Kirk | LinkedIn

☑️ SkadBytes | Apple Podcasts | Spotify | YouTube

☑️ Sebastian Barling | LinkedIn

☑️ Skadden | LinkedIn | X | Facebook

☑️ Subscribe Apple Podcasts | Spotify | Amazon Music

“The Capital Ratio” is presented by Skadden, Arps, Slate, Meagher & Flom LLP and Affiliates. This podcast is provided for educational and informational purposes only and is not intended and should not be construed as legal advice. This podcast is considered advertising under applicable state laws.

Welcome to The Capital Ratio, where Skadden explores the dynamic world of financial institution regulation, offering insights for institutions navigating the regulatory environments in the UK, EU, and US.

Hi, and welcome to this episode of The Capital Ratio, a series in which we look at the key regulatory topics relevant to the UK banking and financial sector. I'm your host, Sebastian Barling, head of the UK and European Financial Regulatory Practice here at Skadden. I am delighted to be joined today by my colleague, Deborah Kirk, who leads the intellectual property and technology transactions practice for Skadden's London office. What are we going to be talking about today? Well, we're going to be discussing the Mills Review, which was published by the FCA in July this year.

The review goes into how the FCA thinks AI will transform retail financial services by 2030 and beyond, and it draws on a large evidence base. There have been over 140 written submissions, a nationally represented consumer survey of over 5,000 UK adults, and extensive engagement across industry, academia, and international regulators. We've also seen the FCA subsequently published in September, its findings from its multi-firm review on front AI and cyber resilience. We will also talk about these a bit later.

Hey Seb, thanks for the introduction and thanks for the guest spot on your podcast. I am going to plug our tech podcast, SkadBytes, which we'll put the link to in the show notes, but really great to be joining you today. And look, my initial thoughts on the Mills Review, it is deliberately, in my mind, correctly forward-looking. I think that's a preferable approach to assessing how AI is deployed today. So, the review really asks what is the financial system going to look like. And what will it demand for firms and the regulator in the future and what will the stakeholders need to do in order to protect consumers?

Thanks. And I agree this is deliberately future looking. And to bring some focus on how we're going to address it in this episode, we're going to be looking at a few key questions that we think will be of most importance to banks and financial institutions. And in particular, we're going to cover what the autonomy spectrum means for oversight, what autonomy spectrum actually means. We're going to talk about whether the senior manager regime and consumer duty will come under strain as this rolls out, what banks and financial institutions should be thinking about right now, as well as how the FSA plan to supervise an AI-enabled market.

Right. And it is worth saying that institutions shouldn't look at this purely through a regulatory lens, right? The review, rightly in my view, treats AI as a truly systemic shift in terms of how it changes firm architecture, consumer behavior, market structure and fraud risk all at once. So, this is more than just a compliance exercise for banks. It is also a flag of how the financial world is likely to change and how stakeholders need to prepare for that. And that's not just a regulatory compliance piece. I mean, existentially might be too strong a word, but culturally and holistically certainly aren't.

I completely agree that this is a substantial development and material development for all financial institutions, and it's going to be something that they will need to look at as they move up the autonomy spectrum. And that is a term that most review keeps coming back to as in the risks are going to evolve as autonomy increases. Debs, would you like to unpack a little bit as to what we mean by autonomy spectrum so everybody's clear?

Yeah, sure. So, the autonomy spectrum as set out in the review is a five level framework and it sits at the very heart of the review. So, it essentially describes how AI shifts from being a simple tool to what the review calls operator mode right through to observer mode. So, what does that mean? That is where AI executes continuously within preset boundaries and the human role is really just to monitor. In this new mode, the human becomes sort of less of a sole director, but more as a collaborator, a consultant, and an approver.

What matters here for bank is that the review finds, and this will not surprise everyone, that firms are already piloting applications at more autonomous end and that by 2030, many could be operating across several functions at an approver level or even beyond whatever that looks like. So, there is then I guess an obvious question on the regulatory side and Seb, I'll put this to you and putting my world of data protection and other regulation and the like aside for a moment. So, will the financial regulatory framework still hold up as AI becomes more autonomous in this universe?

The UK has long maintained that it doesn't need a separate piece of AI regulation and it can fold it within two existing regimes. And actually the Mills Review agrees with that. So, we have existing frameworks such as consumer duty and the senior managers regime and the operational resilience requirements, and they think that these are robust enough to be able to manage some of the risks as AI, even as we go up that autonomy spectrum. That being said, they do think that these regimes will need to develop as things become more complex.

So, for example, under the senior manager's regime, this works well where humans are directing or collaborating with AI, but once you reach observer or approve a status, it's going to become genuinely difficult to evidence meaningful human controls and demonstrate the reasonable steps that the regime demand. So, there's going to have to be some investment and some iteration around how that works. In addition, the FCA has also highlighted its multi-firm review findings on frontier AI. The existing frameworks firm have will need to evolve to meet the risks of in particular frontier AI, and that's going to be acute in the areas of cyber and operational resilience.

Governance frameworks around AI, including around escalation points and human judgment will need to evolve to ensure that they keep pace with AI development and to make sure that issues AI is flagging are triaged correctly and you have the right escalation points to deal with these efficiently and appropriately. So, Debs, from a technology perspective, why is it so hard to maintain traditional oversight over these more autonomous systems?

Yeah, well, it really comes down to the nature of the tech itself. These models update continuously. They act with fluidity rather than deterministically on fixed release cycles. So, they are probabilistic, meaning they can produce different answers to the same question. They can also draw on third party inputs that the firm themselves didn't build. And as AI shifts from being tools that generate static text to becoming independent agents that execute multi-step tasks across live tech ecosystems, the traditional governance structures break down and your point in time validations frankly just ain't going to cut it.

And the review is emphatic that banks need live monitoring, drift detection, controls that trigger when systems move outside the expected parameters in terms of the new look safety measures. And in a practical sense, this will mean that model risk management needs to extend well past the deployment stage into ongoing operations. So, back to you, what should banks be doing right now, do you think from a compliance point of view?

So, I think we've put our heads together and we've come up with four key priorities that banks and financial institutions need to be looking at when looking at how they roll out AI and manage the risks appropriately. And the first is, there'll be no surprises, mapping your autonomy footprint correctly and understanding where you are on that autonomy spectrum. That includes looking at what your plan A deployments will sit function by function and where humans are going to sit within that to make sure that there is sufficient checks and balances around that. And that's going to be really acute when we start moving from observer to an approver and beyond autonomy level.

The second area to prioritize is going to be oversight, asserting that a person is in the loop won't satisfy the FCA or PRA. Firms are going to need to document precisely what that person's expected to do, what information they receive, when they can intervene and what escalation looks like. Thirdly, firms will need to extend model risk management into live operations. Firms should be aware of the need to ensure compliance with consumer duty obligations, particularly as AI's autonomy increases when it moves towards approval or observer models. And lastly, firms will need to review their third party AI contracts. These could be vendor agreements, data access terms, and trigger conditions for workflows.

Firms should remember that they remain responsible for outcomes irrespective of whose models produce them and they need to make sure they have the right control rights baked into the documentation. So, Debs, focusing on third party AI contracts, what are some of the systemic technology risks that banks should be worried about?

Yeah, so actually before I come onto that, just an observation, and it's interesting how the sort of regulation that sits more in my world around specifically AI and data protection have similar themes to those that you've just outlined. So, sounds like the regulators and the legislature in all areas are kind of moving in a similar direction, just an observation. But to your question, this is where the review introduces the concept of ecosystem level risks. It is the shared reliance on the same models, the same cloud infrastructure, the same handful of providers that gives rise to this sort of higher level, more general risk.

So, if a major model provider has an outage or a security breach, multiple banks could be affected simultaneously. And that is not a firm level resilience problem. It is a systemic or at least a potentially systemic issue. And the review goes further on this. It warns that correlated behaviors could emerge where banks are all using similar models for credit underwriting or pricing, potentially amplifying market movements or creating pro-cyclical effect. So, those are some of the risks, but Seb, the review to its credit doesn't just diagnose the problem. It makes several recommendations, seven in fact to the FCA. Which of those do you think banks should be paying the closest attention to at least now?

Before jumping in on those, I think it's worth an observation that it certainly feels likely that these AI providers are the kind of institutions that may fall squarely within the critical third party service provider remit that people are already looking at under operational resilience frameworks. So, it'd be great to see how that evolves. Going back to the FCA's recommendations, we've pulled out three that we think banks and financial institutions should look at specifically. The first is the regulatory perimeter. General purpose AI tools such as large language models that consumers use are currently outside the perimeter, but are increasingly influencing outcomes.

This is really acute when you look at things like investment advice with consumers increasingly turning to LLMs to provide that rather than a registered FCA investment advisor. So, I think the FCA is going to be looking very carefully about this and this could reshape what is in and out of scope in terms of a regulated activity. Secondly, banks will need to be able to ensure that they can adapt to changes in the regulatory framework such as the consumer duty quickly to respond to AI-driven risks. As AI journeys become dynamic, personalized, and continuous, evidencing consumer understanding and fair value is going to get materially harder.

One-off consent will not necessarily be sufficient for an agent making ongoing micro-decisions on behalf of somebody. Banks may need continuous iterative consent mechanisms to make sure that people really understand what is being undertaken on their behalf. And in respect to pricing, the FCA is closely watching for personalization around individual pricing that may not necessarily be providing fair value to consumers. So, there's going to be a need to drill down that all those consumer duty goals get baked into whatever AI deployment firms have. Lastly, the FCA itself is looking to build AI enabled supervisory capability.

Whilst that sounds slightly dystopian, it does mean there's going to be greater capacity within the FCA to monitor quickly, potentially to start taking action quickly. The review in particular calls for an agentic supervisory model, which would facilitate near real-time cross-firm monitoring. This means the banks are going to have to get used to regulators being able to spot patterns and respond to them as quickly as they can move. Debs, on the technology infrastructure side, what should banks be preparing for?

Right. So, the big one I guess is agentic finance and banks need to act on this now rather than wait for the standards to be finalized. The review envisions AI agents transacting on behalf of consumers across multiple firms, switching savings, rebalancing portfolios, comparing insurance, and that means that your systems will receive inbound requests from external AI agents that you didn't build and you don't control. So, practically speaking, banks should be stress-testing their ADIs and data sharing architecture to ensure that they can deal with those scenarios.

And so, I suppose some questions that the banks should be asking themselves are, can your infrastructure verify an agent's identity, the scope of its mandate before it initiates a transaction? Second, can your payment rails handle delegated execution without human approval at each step? And then thirdly and critically, have you mapped the liability chain when an external agent triggers a workflow on your platform and something goes wrong?

So, if you had three takeaways that you would have from this review?

So, I guess bouncing off those sort of three points, mine would be one, prepare your infrastructure for agentic finance and do it now. Number two, extend your AI governance across the full life cycle, not just with respect to deployments. And thirdly, assume that the FCA will match your AI capability with its own. Yeah, I think they're my three. So, Seb, tell me yours.

Well, I'll try not to just copy yours, which I entirely agree with. I think I've probably got two to add. So, one is that AI deployment needs to be controlled and scaled with the appropriate development of governance and monitoring frameworks, particularly at high autonomy levels. And the second is that this is a review focused on retail financial services. So, actually firms that are operating in the wholesale market, we need to think about this a bit more carefully as whether this is something they want to follow or whether they wish to try and push other approaches beyond what the Mills Review is suggesting that is more appropriate for their markets.

Yeah, that's a good one.

But if there is nothing else, that is the end of today's episode of The Capital Ratio. Debs, thank you for joining me today and to share your insights on the tech aspects of the Mills Review. And listeners, thank you for joining us.

You're welcome. Thanks for having me, Seb.

Thank you for joining us for today's episode of The Capital Ratio. If you like what you're hearing, be sure to subscribe in your favorite podcast app so you don't miss any future conversations. Additional information about Skadden can be found at skadden.com.