Transforming data collection from the UK financial sector: a plan for 2021 and beyond

We worked with industry to develop a vision for data collection and a plan on how to get there.
Published on 23 February 2021

Executive summary

Background and aims

Data collection has always been an integral part of what the Bank does. Without the data we collect, we cannot identify risks, design good policy, and take action in a timely and targeted fashion.

But we, and the systems we oversee, are changing. Technological advances and automation mean that more data than ever before is being created and captured. Simultaneously participants across the financial system, including authorities like ourselves, expect more high quality, timely data to guide them in their decision making.

These changes have put strains on the current data collection process, and on the suppliers of data within the financial sector. They have increased costs, and meant systems and processes are being used in ways their designers never envisioned. But digitisation may also offer solutions to these problems. As we explored in our recent joint innovation project ‘Digital Regulatory Reporting’,footnote [1] technology could transform how the data collection process works, with benefits for the Bank and industry.

In 2019, in order to decide how to respond to these changes, the Bank committed to a review of data collection. The review was launched with the publication of the discussion paper ‘Transforming data collection from the UK financial sector’, in January 2020. The review’s aim was to shape the evolution of reporting over the next 5-10 years. So its goals were long-term, and its scope was wide. It covered structured data used by our regulatory, statistical, and markets and banking teams. At the heart of its approach was open and collaborative dialogue between all parties to the data collection process. This dialogue has, so far, taken place via more than 260 internal and external events, with over 130 organisations, and through receipt and review of over 60 written responses to the discussion paper.

This paper sets out the findings from that discussion process, and the next steps for the review. It documents what we have learnt about the problems with data collection – and the possible remedies for those problems – before setting out our aspirational vision for data collection and how we think we will get there.

Key findings

We think the issues the industry and the Bank face can be summarised in three key questions that need to be addressed:

  • How can we ensure our data collections are worthwhile and valuable exercises for regulators and industry to invest in?
  • How can industry best understand and interpret our reporting instructions so that high quality data is provided?
  • How can we remove legacy data, process and technology siloes and streamline the reporting process?

By successfully addressing these three questions, we believe we can achieve our vision for data collection that: ‘The Bank gets the data it needs to fulfil its mission, at the lowest possible cost to industry’. Achieving that vision means that our data collections deliver rich and rapid insights into emerging risks, and the Bank can act confidently and in a targeted way to mitigate those risks. It means that our data collection processes are lean, so we and industry can make the most of scarce resources. It means that the process and purpose of data collection is clear. And it means that data collection continues to be safe, so that data is collected and used securely, legally and ethically.

In order to realise the vision, the Bank and industry will need to transform how we manage and collect data. In particular, the review identified three key reforms that need to take place:

  • Defining and adopting common data standards that identify and describe data in a consistent way throughout the financial sector. These common standards should be open and accessible for use by all who need them. Their adoption will bring benefits well beyond reporting.
  • Modernising reporting instructions to improve how our reporting instructions are written, interpreted and implemented. There are a range of steps we think this will involve, from setting up better Q&A processes to potentially rewriting our instructions as code.
  • Integrating reporting to move to a more streamlined, efficient approach to data collection. This reform includes making data collection more consistent across domains, sectors and jurisdictions, and designing each step in the data collection process with the end-to-end process in mind.

We don’t think the reforms can be achieved quickly or easily. We are aware of the difficulties we and industry will face moving away from legacy solutions. And many of the changes required will be cultural as well as technical, with sustained investment required to make the range of improvements we have identified.

Next steps and transformation programme

To help deliver these reforms we, alongside the FCA, want to set up a multi-year, and multi-phased, transformation programme. We want this programme to be delivered with an iterative and pragmatic approach. During each phase we will aim to deliver a series of ‘use cases’ focusing on particular collections or types of collections. Each use case delivered will add value in its own right, as well as delivering improvements that can then be applied to other collections over time. We expect the first phase, taking place over the next 24 months, to deliver improvements and lessons for the future, but to only impact a small number of selected use casesfootnote [2]. The second phase, taking place over roughly the subsequent three years, will focus on expanding the transformation into new areas with an increased focus on integration. Subsequent phases will scale the transformation to maximise value.

And while we are aware that the global pandemic poses unique challenges to us all, we do want to make progress over the coming year. We hope by the end of this year to have the structures in place that will govern this programme of work, with delivery teams designing and testing the future of data collection.

In particular, we are looking to build a core team comprising staff from the Bank, the FCA and from the firms from whom we collect data, as well as solution providers who wish to participate. We currently estimate the team’s size to be between 10-15 people, of which 12 might come from firms. We expect the core team to be supplemented by part-time input from others, both from industry and from the Bank and FCA. We will need participants who have expertise and vision to contribute to designing solutions. In addition, to ensure the new changes work for everyone, we would like individuals who are directly involved in the current data collection process to provide feedback on issues and solutions. We invite interested parties who wish to contribute resource to contact us directly by email:

As we did during the review, we want to continue to work with industry in an open, collaborative and transparent manner. We think having the widest possible set of views will help us get to the best solutions to existing problems. And we are acutely aware we can’t think about ‘integrating’ the reporting process without having all the parties in the reporting process involved.

Openness is also important because we do not think we can deliver these reforms via the UK transformation programme alone, and nor do we want to. We know there are ongoing and completed data collection projects taking place around the globe, some of which we are already involved in, and many of which we talked to during the review process. And, as became clear during the review, our reforms tackle fundamental issues that industry are already trying to address: in particular inconsistencies in how financial data are described and identified. So we will need to align and coordinate with these and other initiatives.

Being open and transparent is the first step in doing so, but we will also need to take an active role to align with, and in some cases possibly coordinate, broader initiatives. That is likely to be particularly important regarding international engagement. For instance, we will continue to work with other financial authorities to push the development and adoption of key financial identifiers. We will deepen our relationship with relevant private sector initiatives. And we will engage authorities around the world to develop and share best practice, and push reporting and data standards up the global policy agenda. This international work will take time. As an initial step, we are working on a joint projectfootnote [3] run by the Singapore BIS-Innovation Hub to create common data standards and machine executable reporting rules for mortgage reporting.

Together we think these programmes, and the reforms they can bring about, will build some of the soft infrastructure for the digital age. That digital infrastructure will deliver value throughout the financial system.

Review on a page

1: Structure of the paper

The paper has two aims. Firstly, the paper aims to share the findings of our year-long data collection review work. That involved an intensive engagement process with internal and external stakeholders, where we sought to understand the underlying issues and assess possible solutions for improving data collection. This is the focus of the first half of the paper: chapters two, three and four. Secondly, the paper aims to set out a path towards transforming data collection, responsive both to legacy constraints and the opportunities made possible by technological innovation. We describe our vision for data collection, and our incremental, use-case based approach for delivering tangible change. We intend to start the first phase of this transformation, in close collaboration with industry, during 2021. This is the focus of the second half of the paper: chapters five to seven.

The second chapter sets out the background to the review, which was launched with the publication of a discussion paper at the beginning of 2020. The review was conducted in two phases, the first exploring the options with industry, and the second contributing to the design of the plan itself.

The third chapter discusses the current problems with data collection, weaving in the comments submitted in formal responses to the discussion as well as feedback we received during bilateral discussions with firms. It highlights that many issues start from the complexity of reporting systems, both within the Bank and within firms.

The fourth chapter summarises industry preferences for the various solution areas presented in the discussion paper: common data inputs, modernising reporting instructions, and architecture and governance. There was strong support for the first two solution areas, with respondents suggesting that the third area can only be useful once the first two are achieved.

The fifth chapter discusses the Bank’s vision for transforming data collection. Our vision centres around three key reforms: common data standards, modernised reporting instructions and an integrated approach to data collection. The first two link closely with ideas previously presented in the discussion paper; the last reform outlines the need for breaking down existing data and solution siloes within firms and the Bank.

The sixth chapter describes how we plan to tackle the reforms and reach our vision. It lays out the principles we plan to use to approach this transformation, our primary vehicle for carrying making change: our joint transformation programme, and how we plan to work with other relevant initiatives and bodies to deliver the reforms.

The annexes contain additional information. Annex 1 gives more detail on how the review was carried out. Annex 2 lists the external working group members. Annex 3 discusses a stylised view of the data collection process. Annex 4 contains an overview of some of the use cases discussed within working groups.

2: Review background and activities

The review took place in response to recommendations in the Future of Finance report, published in June 2019. Specifically the Bank committed to ‘Launch a review in consultation with banks, insurers and financial market infrastructures to explore a transformation of the hosting and use of regulatory data over the next decade.’ The review investigated ways to decrease the burden on industry and to increase the timeliness and effectiveness of data used by the Bank.

The Bankfootnote [4] launched its review by publishing the discussion paper on ‘Transforming data collection from the UK financial sector’ (the DP) in January 2020. The DP clarified the scope of the review to include statistical data and data collected for our markets and banking operations in addition to regulatory data. The DP also asked for feedback on the current reporting process, potential solutions for issues identified, and possible delivery approaches. The potential solutions it discussed drew on ideas explored in the Future of Finance report and the joint Bank/FCA pilot on Digital Regulatory Reporting (DRR).

The review took place throughout 2020 (see Annex 1). The external review participants were a mix of firms (who are responsible for submitting data to us for regulatory or other purposes), trade bodies (representing subsets of the financial industry), solution providers (supplying software or services to firms) and various public authorities who have an interest in our work. Collectively, we refer to all these organisations as industry. Alongside external conversations, the review included discussions and meetings with individuals from across the Bank and PRA – hereafter referred to as internal participants.

The review started with a four-month discussion period. During this phase we explored the DP questions through a process of direct stakeholder engagement, including bilateral meetings and roundtable events. We also held similar discussions internally with colleagues across the Bank. By the close of the discussion period, the Bank received over 60 written responses from industry. The evidence gathered from this period informs the first half of this report, setting out problems with the current reporting process, and summarising the feedback on possible solutions to those problems.

Figure 1: Organisations involved in review process

Chart showing numbers of different types of firms that submitted discussion paper response and those engaged at events.

During the second half of the review we shaped our discussion feedback into our transformation plan for data collection. During this phase we discussed the overall vision, approach, and use cases that fed directly into the plan presented within this document. The focus of this phase was a series of internal and external working group meetings. External working group members represented a range of reporting firms (retail and commercial banks; wholesale banks; financial market infrastructure; and insurance – see Annex 2). Each external working group meeting in the series of three was prefaced with an internal one, with representatives across the Bank’s business areas. For those not part of the working group process, such as technology solution providers and other standards bodies, we discussed many of the same topics in bilateral meetings and specific roundtables.

Figure 2: External working group members by organisational typefootnote [5]

Boxes represent the proportions of different types of firms in each of the external working groups.

3: Problems with the current reporting process

The review deepened our understanding of the current problems with data collection, and where specifically they occur. This enhanced awareness of the issues helped us frame potential solutions. In the future, it will also help us evaluate the extent to which the solutions that we implement have effectively tackled the problems identified. In addition to providing the areas of focus for this review, we hope this chapter proves useful to other parties interested in improving data collection.

Review participants confirmed many of the costs and issues we laid out in our original discussion paper, before fleshing out these problems with useful examples and ancillary points. These issues are grouped into themes and summarised in this chapter.

To frame our feedback on issues – and the solutions presented in the next chapter – we developed a stylised view of the data collection process. This process contains a number of phases which, although they should not be read as sequential, do have dependencies on each other. The reporting process outlined here could apply to either setting up a new data collection, or making a significant change to an existing one. The process for one-off collections is similar, but truncated.footnote [6] We see the reporting process as having seven key phases:

  • Analytical question and collection initiation: an authorityfootnote [7] identifies a new policy question and determines what data to collect
  • Rule-making governance, approval and publication: authority drafts proposals, conducts internal review, consults with firms and publishes final, approved version
  • Impact analysis and interpretation: firms interpret the rules and review their impact on internal processes and resources
  • Solution development and implementation: firms establish how to source, authorise and submit data to the authority
  • Execution, maintenance and submission management: firms execute the designed processes, and produce the data required
  • Assurance and use: authority quality assures the data submitted, and firms provide explanations or corrections as needed
  • Change and review: authority reviews data collection to assess its continued relevance, and ceases collections that are no longer required

The graphic below depicts the phases, with a fuller description included in Annex 3.

Figure 3: Rule-making, governance approval and publication