PS6/23 – Model risk management principles for banks

1. Overview

1.1 This Prudential Regulation Authority (PRA) policy statement (PS) provides feedback to the responses to consultation paper (CP) 6/22 – Model risk management principles for banks. It also contains the PRA’s final policy, as follows:

• supervisory statement (SS) 1/23 – Model risk management principles for banks (see Appendix 1).

1.2 The feedback to the responses to CP6/22 in this PS is relevant to all regulated UK-incorporated banks, building societies, and PRA-designated investment firms (hereinafter ‘firms’). For the reasons explained in the Change in scope section below, the final policy in Appendix 1 (SS1/23) only applies to firms with internal model (IM) approval to calculate regulatory capital requirements^{footnote [1]} (hereinafter ‘IM firms’).

Background

1.3 In CP6/22, the PRA proposed firms should adopt five principles which it considers to be key in establishing an effective model risk management (MRM) framework. The principles were intended to complement existing requirements and supervisory expectations in force on MRM, and included proposals for:

• a proportionate implementation within firms and across firms;
• the identification and allocation of responsibility for the overall MRM framework to the most appropriate Senior Management Function (SMF) holder;
• reporting on the effectiveness of MRM for financial reporting to the audit committee; and
• identifying and managing the risks associated with the use of artificial intelligence (AI) in modelling techniques such as machine learning (ML) to the extent that it applies to the use of models more generally.^{footnote [2]}

1.4 In CP6/22, the PRA also invited responses to the following question:

In your view, are there any components of the MRM framework where the proposed principles are not sufficient to identify, manage, monitor, and control the risks associated with AI or ML models?

Summary of responses

1.5 The PRA received 26 responses to the CP, 11 of which included a response to the question on MRM for AI/ML models. Overall, respondents supported the PRA’s proposals to raise the standard of MRM practices and recognised the need to manage risks posed by models that have a material impact on business decisions, while providing comments or requests for clarification on individual proposals that are set out in chapter 2.

Changes to draft policy

Change in scope

1.6 CP6/22 proposed that all firms should adopt the principles proportionately when implementing across firms. It also proposed additional ways to ensure a proportionate implementation of the principles for ‘Simpler-regime Firms’ as defined in CP5/22, noting that this is a working term that might be revised in due course.^{footnote [3]} Most respondents welcomed the proposal to create an explicit link to the proposed ‘Simpler-regime Firm’ definition for the purposes of the proportionate application of the principles consulted upon.

1.7 At the time of publishing this policy statement, the PRA has not published its final definition of ‘Simpler-regime Firms’ under the Strong & Simple Framework. As a result, at this stage the PRA has narrowed the scope of the expectations in SS1/23 to apply only to IM firms.

1.8 The PRA will provide an update on the approach for all other firms, including ‘Simpler-regime Firms’, at a future date, once the definition of a ‘Simpler-regime Firm’ has been finalised. Irrespective of scope of application of SS1/23, all firms regardless of size are already expected to manage the risks associated with models, as they would with any risk they are exposed to.^{footnote [4]} All firms should continue to apply the existing supervisory expectations relevant to them and their particular models, including attestations and self-assessments where applicable (see paragraph 1.3 of Appendix 1).

1.9 The final SS has also been amended to add implementation guidelines for any firms that are granted permission to use internal models to calculate regulatory capital for the first time after the publication of the SS. These arrangements are intended to be transitional and will be reviewed once our approach for all other firms is updated.

Changes following the consideration of industry responses

1.10 Following consideration of the respondents’ comments, the PRA has made changes to the final SS. A summary of the key changes is set out below:

• Senior Management Function (SMF) accountability – the PRA has modified the wording of Principle 2.2 to remove potential ambiguity in responsibilities of the SMF and clarify that more than one SMF may be appointed;
• financial reporting – the PRA has replaced a reference to accounting with financial reporting and clarified the intent is to ensure MRM reporting is available to the audit committee;
• model tiering – the PRA has modified the wording in Principle 1.3 (c) to clarify that firms can select the relevant factors to determine model complexity;
• subsidiaries – the PRA has clarified that subsidiaries using models developed by their parent-group may leverage the outcome of the group’s validation of the model if the conditions in Principle 2.6 (c) are satisfied;
• dynamic recalibration – the PRA has combined the expectation for models that recalibrate dynamically in (former) Principle 3.3 (d) with the (former) clause on model changes in Principle 3.3 (e);
• model documentation for vendor models – the PRA has clarified the expectation for model documentation of vendor models in Principle 3.5;
• post model adjustments – the PRA has modified the principle on model adjustments (Principle 3.4) to acknowledge that model adjustments are an important risk management tool, and changes have been made to Principle 5.1 to recognise the need for proportionality; and
• escalation processes – the PRA has modified Principle 5.3 to be less prescriptive and more principles-based in line with CP6/22’s intent.

1.11 The PRA has also made minor changes to improve the readability of the SS and to ensure the use of consistent MRM terminology. In addition, the PRA has amended the wording of the main principles and included the word 'should' to make it clear that the principles are describing the PRA's expectations with regards to firms' management of model risk.

1.12 The PRA considers that these amendments address the feedback received in light of the intended policy. This will not increase the burden on firms (including mutuals) or have a differential impact on mutuals compared to other firms, and will in certain areas reduce the likely costs of meeting the expectations.

1.13 In carrying out its policymaking functions, the PRA is required to have regard to several matters, as set out in Appendix 2 – 'The PRA's statutory obligations' of CP6/22. The PRA considers the factors set out in CP6/22 on how it had had regard in developing the policy to remain the most relevant matters in relation to the proposed policy. However, the PRA considers the regulatory principle of exercising its functions as transparently as possible particularly influenced the changes in this policy. The PRA considers the feedback provided in this PS brings greater clarity and transparency to what it considers effective MRM.

Implementation and next steps

Implementation date

1.14 The policy will take effect 12 months after publication of this PS, ie Friday 17 May 2024. Firms that first receive permission to use an internal model to calculate regulatory capital after the publication of this policy will have 12 months from the grant of that permission to comply with the expectations in SS1/23.

PRA reviews

1.15 The PRA considers that the assessment of firms’ model development, independent validation, and risk mitigation practices will continue to underpin the PRA’s review of firms’ internal regulatory capital models. The PRA intends to seek opportunities to embed the assessment and review of firms’ overall MRM frameworks into the business as usual supervision of firms and risk assessments.

1.16 In the first year following policy implementation, the PRA will assess the overarching MRM frameworks and MRM practices for a sample of firms with permission to use IMs to calculate regulatory capital.

2. Feedback to responses

2.1 The PRA must consider representations that are made to it in accordance with its duty to consult on its general policies and practices and must publish, in such manner as it thinks fit, responses to the representations.

2.2 The PRA has considered the responses received to the CP. This chapter sets out the PRA’s feedback to those responses, and its final decisions.

2.3 Respondents provided thematic feedback on the proposals as well as feedback on the individual principles and sub-principles. Respondents requested clarification of certain aspects of the principles which are summarised in this chapter. The sections of this chapter have been structured as follows:

• feedback where changes have been made to draft policy
• feedback on MRM for AI/ML models
• additional feedback received

Feedback where changes have been made to draft policy

Senior Management Function (SMF) accountability

2.4 Respondents welcomed the proposed introduction of an accountable individual for the MRM framework and agreed that the Chief Risk Function (SMF4) is likely to be the most appropriate individual to fulfil this proposed expectation. Respondents raised concerns, however, that the proposed responsibilities were too prescriptive for an SMF holder. Respondents also sought clarity over the ambiguity in the references to the SMF’s accountability that appear to encompass both the second and first line of defence responsibilities.

2.5 The PRA does not consider the appointment of an accountable SMF to prejudice the respective responsibilities of business, risk, and control functions. In line with SS28/15 –Strengthening individual accountability in banking, the SMF responsibility for MRM is additional and complementary to the responsibilities of SMF holders for business, risk, and control functions. This has been made clearer in Principle 2.2 of the final SS. To remove any potential ambiguity, the PRA updated the wording in Principle 2.2 (a) to reflect that the SMF is expected to assume overall responsibility for the MRM framework, its implementation, and the execution and maintenance of the framework.

2.6 In practice, certain duties of an SMF may be delegated, in line with existing policy, and firms have the flexibility to appoint more than one SMF. The PRA has updated the final SS to acknowledge this with modifications to the wording in Principle 2.2 (a), (b) and (c), Principle 2.3 (d) and paragraphs 1.8 and 3.6. Notwithstanding the delegation of responsibilities, the SMF(s) will remain accountable for the overall MRM framework.

Financial reporting – effect on audit

2.7 One respondent asked if the PRA intended to create new expectations on audit committees. Paragraph 3.8 of the final SS has been amended to make clear the intent is for a report to be made available to the audit committee on a regular basis to support the audit committee in carrying out its role. This is to make clear that no changes have been made to the PRA rules or expectations relating to audit committees.

2.8 One respondent said that references to accounting and financial reporting could be clarified, and that MRM principles might be equally relevant to other areas of focus such as climate related matters whether included in the annual report or elsewhere. The PRA has therefore amended paragraph 3.7 of the final SS to replace one reference to ‘accounting’ with ‘financial reporting’.

Model tiering

2.9 Seven respondents considered Principle 1.3 on model tiering overly prescriptive. They suggested that Principle 1.3 considers too many factors, particularly to define complexity, which could result in a tiering methodology that is overly complex and hard to understand.

2.10 The PRA considers that assessments of model complexity are standard practice in industry, and are consistent with the notion that more complex models are prone to greater levels of uncertainty. While the PRA expects firms’ model tiering approaches to at least consider a metric to characterise model complexity, the relevant factors to determine model complexity will vary across firms and models. The PRA has amended Principle 1.3 (c) of the final SS accordingly.

Subsidiaries

2.11 Respondents requested clarity on whether subsidiaries can leverage validation reviews performed at the group or parent company level. The PRA has amended the SS to clarify that subsidiaries using models developed by their parent-group may leverage the outcome of the group’s validation of those models if the conditions in Principle 2.6 (c) are satisfied.

2.12 Respondents also requested clarity around the application of the principles across a UK group, ie overseas branches and subsidiaries of UK headquartered groups. The expectations in this SS are intended to apply group-wide, which includes overseas subsidiaries and branches of UK headquartered firms. All models are expected to be considered within scope of these expectations regardless of their coverage, ie application across single or multiple business units, balance sheets, or legal entities. Groups should consider the PRA’s expectations alongside any local regulatory or corporate governance requirements which apply to model governance, policies, or processes in its branches or subsidiaries.

Dynamic recalibration

2.13 Some of the respondents considered that the automatic recalculation of performance test results for models with dynamic calibration would be computationally intensive and may not be practical across all model types, eg algorithmic trading and other pricing models. The PRA expects firms to manage the risk that a series of small (immaterial) changes due to recalibrations could accumulate, when uncontrolled or unchecked, into a material change in the model output over time without it being tested. The draft SS already covered this risk when referring to model change in general (former Principle 3.3 (e)). Consequently, in the final SS, the PRA has combined (former) Principle 3.3 (d) on the risks associated with models that recalibrate dynamically with (former) Principle 3.3 (e) that covers model changes.

Model documentation

2.14 Respondents requested that the PRA clarify expectations on model documentation for vendor models. In particular, respondents highlighted that full replication of model documentation would not always be possible for vendor models.

2.15 The PRA recognises that the documentation provided by vendors is unlikely to be as extensive and detailed as for internally developed models, and that there is no obligation on vendors to disclose proprietary information on their products. The PRA expects firms to ensure the level of detail in the documentation of third party vendor models is sufficient to validate their use of the model. This is in line with current requirements, eg Internal Ratings Based (IRB) approaches for credit risk (SS11/13) and current expectations, eg MRM principles for stress testing models (SS3/18). The PRA has updated Principle 3.5 (a) to this effect.

Post-model adjustments (PMAs)

2.16 Respondents generally welcomed the PRA setting expectations for the use of PMAs. Some respondents noted that PMAs are an important risk management tool and proportionality was key to ensure their use is not disincentivised. The PRA recognises the need for PMAs that are subject to robust governance to capture risks and uncertainties not adequately reflected in models. The PRA has modified Principle 3.4 (f) of the final SS to reflect this view and changes have been made to Principle 5.1 of the final SS to recognise the need for proportionality.

Exceptions and escalations

2.17 A respondent noted that all exceptions, once they occur, are escalated and the appropriate management level decides on their approval or rejection on a case-by-case basis. Respondents were concerned that Principle 5.3 was too prescriptive and noted that restrictions on model use may not always be appropriate.

2.18 The PRA has considered the responses received on escalation processes, and acknowledges that Principle 5.3 could be too prescriptive in some cases. The PRA has therefore sought to make this principle more proportionate and removed clauses (i), (ii) and (iii) from Principle 5.3 (b) on escalation processes. These processes should be determined by firms.

Feedback on the specific question on MRM for AI/ML models

2.19 The PRA received 11 written responses to the question on AI/ML models. Respondents were in broad agreement that the principles are sufficient to identify, manage, monitor, and control the risks associated with AI/ML models. In addition, respondents pointed to specific areas where AI/ML models may amplify risks and present additional challenges, thereby necessitating greater care and attention when implementing and applying the principles. The main areas highlighted by respondents were as follows:

• AI/ML systems often span multiple functional areas including data, models, and technology, therefore, a firm-wide approach with greater collaboration across relevant areas would be beneficial;
• explaining how an AI/ML model works and how it produces its outputs can be challenging – one respondent noted that while the principles refer to the topic as part of model tiering, firms may benefit from the PRA giving practical examples;
• some AI/ML models are dynamic by design ie they can change and/or recalibrate frequently – this may present additional challenges including ensuring adequate oversight and review;
• monitoring of model performance becomes increasingly important as AI/ML model complexity increases; and
• several respondents pointed out that the use of AI/ML models can raise ethical challenges including fairness and bias – such ethical challenges could increase conduct and reputational risks, and better management and oversight of such risks may be needed.

2.20 In October 2022, the Bank, the PRA, and the FCA published a discussion paper (DP) on AI (DP5/22). An initial analysis of the responses suggests that there is alignment with responses to CP6/22. In particular, respondents to DP5/22 generally agreed that the MRM principles are broadly sufficient to cover AI/ML models. However, as with CP6/22, respondents highlighted areas where additional clarification and guidance on best practice may be useful.

2.21 The PRA, the Bank and the FCA are in the process of analysing the responses to DP5/22. The PRA will consider the outcome of the analysis, together with the results of the 2022 machine learning survey and the responses to the MRM CP, to inform any decisions on further policy actions.

Additional feedback received

Model definition

2.22 Several respondents noted the proposed model definition in Principle 1 has a broader scope than that in SR11-7^{footnote [5]} in the United States, as it includes qualitative model outputs. The respondents asked the PRA to provide further clarity around the intention to widen the definition to include qualitative model outputs.

2.23 The PRA’s model definition intends to ensure that recommendation systems in client services and other AI/ML that deliver qualitative output are within the scope of the MRM policy. For example, machine learning models that uses data mining to seek to predict, narrow down, and find relevant content for users or recommend additional products to consumers.

2.24 The PRA considers that the proposed model definition reflects the increasing complexity and use of models at firms following the rapidly changing digital landscapes, and evolution of more sophisticated modelling techniques. Since the publication of SR11-7 and the definition of a model introduced by the Federal Reserve and Office of the Comptroller of the Currency, a number of international regulators^{footnote [6]} have broadened the definition to incorporate quantitative methods delivering qualitative output to be classified as a model.

2.25 Principle 1.1 (b) includes the possibility of applying the relevant aspects of the MRM framework to material deterministic quantitative methods such as decision-based rules or algorithms that are not classified as a model, have a material bearing on business decisions, and are complex in nature. Several respondents expressed concern that a broad reading of Principle 1.1 (b) could lead to quantitative approaches or mechanisms typically not classified as models being brought into the scope of the MRM policy. Further clarity was sought around the type of ‘material deterministic quantitative methods' the PRA expects could potentially be placed within model risk governance and which aspects of the framework firms should consider to apply (in Principle 1.1 (b)). The PRA considers that examples of highly complex quantitative calculation systems that could have a material bearing on a firm's financial position include Electronic Trading Systems that are made up of a complex interdependent network of components, and which may constitute a model, as well as financial crime and/or anti-money laundering systems.^{footnote [7]}

Board of directors’ responsibilities

2.26 Respondents generally welcomed that Principle 2 on governance would likely strengthen the board of directors’ involvement in establishing MRM frameworks. However, respondents were concerned that Principle 2.1 appeared to impose unrealistic obligations on the board of directors, is overly prescriptive, and would result in model risk being managed at a higher level compared to other risk types.

2.27 Respondents requested clarity on the expectations for the board of directors’ involvement, in particular whether the board can delegate responsibilities to an appropriate sub-committee such as the board risk committee or model risk committee.

2.28 The PRA has set the expectations in Principle 2.1, in particular the role of the board of directors, in the context of existing PRA rules and expectations for boards and senior management. The risk committee plays an active role in advising on risk appetite and overseeing the implementation of that strategy,^{footnote [8]} and delegation of a wide range of duties and responsibilities of the board to executive management is permitted (SS5/16 – Corporate governance: Board responsibilities).^{footnote [9]}

2.29 The board of directors and senior management should establish a firm-wide approach to MRM that fits into the organisation’s risk management framework. The board of directors is ultimately responsible and should ensure that the level of model risk is within tolerance. In practice, this may be through appropriate delegation to a relevant sub-committee.

2.30 The board of directors may delegate the responsibility for executing and maintaining an effective MRM framework to senior management. Senior management are then responsible for regularly reporting to the board of directors on significant model risk and associated policy. This may be directly or via appropriate sub-committee(s).

2.31 The PRA expects to see evidence that the board of directors and its relevant sub-committees exercise effective oversight of risk management and controls. The board of directors should therefore possess a general understanding of model risk and be able to appropriately challenge senior management on the firm’s compliance with the framework that has been put in place.

2.32 These expectations are in line with expectations on boards in current PRA policies (eg SS2/21 – Outsourcing and third party risk management), the PRA's approach to banking supervision, and with international expectations on boards – see for example the regulatory expectations of the Board of Governors of the Federal Reserve System and Office of the Comptroller of the Currency as set out in SR11-7^{footnote [10]} and SR15-18^{footnote [11]} with regards to Board understanding and challenge.

Financial reporting – effect on audit

2.33 The PRA received seven written responses on the effect on financial reporting and external auditors, and has discussed its proposals with the Financial Reporting Council (FRC).

2.34 Respondents generally agreed that the proposals should help enhance auditors’ ability to carry out high quality bank audits by strengthening the control environment and governance around models that impact the financial statements, and are likely to be relevant to the auditor’s work.

2.35 Several respondents suggested that, while typically and increasingly models are important to preparing financial statements, audit procedures may not necessarily involve a review of the MRM framework. Some respondents raised concerns that the proposals might create new expectations of auditors that go beyond the requirements of auditing standards and could lead to additional costs for firms and asked for further clarity.

2.36 The PRA does not intend to change the responsibilities of the auditor or to increase the level of audit work on MRM beyond the requirements of auditing standards. The PRA considers that the proposals should help the auditor to obtain a better understanding of the risks and controls around models that will be helpful to them in carrying out their role, and that this will improve the quality and focus of the auditor-supervisor dialogue. It is for auditors to judge as to whether and how they decide to engage with the MRM framework in order to meet their responsibilities. It is also possible that auditors may become aware of matters through their audit work that could indicate that the MRM framework is not effective, and the auditor will determine how to react to that.

Proportionality

2.37 Seven respondents supported the expectations on proportionality in paragraph 1.23 of the draft SS (paragraph 3.4 of the final SS) that the MRM framework should be applied proportionately within each firm with the rigour, intensity, prioritisation, and frequency of model validation and risk controls to be commensurate with model tiers.

2.38 However, respondents considered the expectations on proportionality was undermined by expectations (sub-principles) which appear to apply equally to all models within scope. Respondents requested that the concept of model tiering should be applied throughout the SS. Respondents also asked the PRA to:

1. clarify the proportionality rules for:
   • model development documentation;
   • the independent review;
   • model performance monitoring; and
   • the periodicity of re-validation, and
2. set a minimum expectation for:
   • the frequency of validation of the lowest tier models; and
   • the performance testing range and depth of validation of the lowest tier models.

2.39 The PRA considers paragraph 3.4 of the final SS to allow firms to determine their own proportionate application of the principles to the models they use based on their size, business activities, use and complexity of their models, and how firms manage their own risk decisions. The PRA considers the model validation approach applied within firms to be firm and model or risk specific and should be determined through their model tiering approach. The PRA does not consider setting standard minimum expectations for validation activities as appropriate and would not be equally applicable to all firms for all risk or model types. The intention of linking paragraph 3.4 of the final SS with the principles on model tiering is to provide firms with adequate flexibility to develop and implement the MRM framework that works for them. Setting specific expectations for validation activities would be unduly prescriptive and contrary to the intent of the principles.

Post-model adjustments (PMAs)

2.40 Respondents requested further clarification on the distinction between the model adjustments identified during the development process that are addressed in Principle 3, and post-model adjustments (PMAs) that are addressed in Principle 5. Respondents also noted this distinction matters as the expectations for validation and documentation of certain PMAs is reduced, compared to models and model adjustments.

2.41 The definition of model adjustments used in Principle 3 is intended to ensure that where the need to adjust model outputs on an ongoing basis is identified in advance, such adjustments are documented as part of the model development process and independently validated. The definition of PMAs used in Principle 5 is broader and recognises that it is not feasible to predict all adjustments that will be needed to model outputs in advance, nor is it always feasible to independently validate adjustments that require the use of significant judgement, or adjustment due to new information becoming available.

2.42 In practice, the distinction between model adjustments and PMAs will depend on the specific circumstances of each firm, and the degree to which it is possible to predict the need for adjustments in advance.

Independent review of PMAs

2.43 Respondents requested clarification on who should conduct the independent review of PMAs in Principle 5.1 (e). Respondents also suggested that the references to ‘senior management' and levels of approval authority in Principle 5.1 (c) should be replaced with an appropriate authority and approver.

2.44 The PRA considers who will have the appropriate level of skill, experience and independence to conduct the independent review of PMAs will vary depending on the specific facts and circumstances relevant to the PMA. The intention of Principle 5.1 (e) is that each firm is responsible to determine whether the PMAs need to be reviewed by a model governance team or another independent party with an appropriate level of skill and experience. This extends to the choice of an appropriate level of authority for the approval of material models in Principle 5.1 (c).