Transforming data collection – request for input to the solution design for our use cases

The data collection joint transformation programme are reaching out to suppliers to support our design phase for our use cases to find potential solutions to the issues we face.
Published on 01 December 2021


Following work with the Financial Conduct Authority (FCA) and Industry, and as a response to the Future of Finance report, the Bank of England (BoE) carried out a review of Data Collection in 2020. The review aimed to understand what issues industry face in supplying the data, what issues the BoE faced with receiving and using it, and what collectively is needed to do to tackle those issues. At the end of the review, the BoE published ‘Transforming Data Collection from the UK Financial Sector: A Plan for 2021 and beyond’ (the Transformation Plan) on 23 February 2021.

Alongside the publication of the Transformation Plan, the BoE and FCA sent a Dear CEO letter to give an update on progress and plans for 2021, and to ask for engagement from firms. A Town Hall was held in Spring 2021 to communicate and discuss these messages with relevant stakeholders, and explain our plans for a joint transformation programme with industry. This joint transformation programme sets out a vision and approach to delivering improvements in data collection over the next decade.

The vision of the joint transformation programme is that: ‘The Bank of England, Financial Conduct Authority and other users get the data they need to fulfil their mission, at the lowest possible cost to industry.’

In order to realise the vision, the BoE, FCA and financial services industry (including third party suppliers to financial firms) will need to transform how we manage and collect data. In particular, the review identified three key reforms that need to take place:

  • Defining and adopting common data standards that identify and describe data in a consistent way throughout the financial sector. These common standards should be open and accessible for use by all who need them. Their adoption will bring benefits well beyond reporting.
  • Modernising reporting instructions to improve how our reporting instructions are written, interpreted and implemented. There are a range of steps we think this will involve, from setting up better Q&A processes to potentially rewriting our instructions as code.
  • Integrating reporting to move to a more streamlined, efficient approach to data collection. This reform includes making data collection more consistent across domains, sectors and jurisdictions, and designing each step in the data collection process with the end-to-end process in mind.

We don’t think the reforms can be achieved quickly or easily. We are aware of the difficulties we and industry will face moving away from legacy solutions. And many of the changes required will be cultural as well as technical, with sustained investment required to make the range of improvements we have identified.

Joint transformation programme

Central to our plan to deliver the reforms is our joint transformation programme which commenced in July 2021. Alongside BoE and FCA staff, the programme comprises industry participants with a wide range of knowledge and experience, drawn from across the financial sector. Following a nominations process, participants were selected in line with our selection criteria. There are currently around one hundred participants working across the programme from over forty regulated firms.

The joint transformation programme thinks the issues the industry and the Regulators face can be summarised in three key questions:

  • How can we ensure our data collections are worthwhile and valuable exercises for regulators and industry to invest in?
  • How can industry best understand and interpret our reporting instructions so that high quality data is provided?
  • How can we remove legacy data, process and technology siloes and streamline the reporting process?

The joint transformation programme is taking a use case approach in this work to research, design and test solutions to address the issues the regulators and industry face. The three use cases for 2021/22 are:

  • The Quarterly Derivatives Return – a Bank of England return
  • Commercial Real Estate reporting – a Bank of England return
  • The Covid-19 Impact survey focussing upon the Financial Resilience elements – an FCA return

We wish to engage all stakeholders with relevant knowledge and expertise in areas that relate to the Quarterly Derivatives Return and Commercial Real Estate reporting use cases. This request for input is part of our engagement process with third party suppliers to the financial services industry.

The use cases

Quarterly Derivatives Return


The Financial Derivatives return (Form DQ) is a high-level view of the value of the derivatives business done by UK banks. It is submitted quarterly to the Bank of England. The data also feeds into the UK ‘financial account’ - the ONS’ estimate of the UK economy’s balance sheet and financial flows, which are used by economies and businesses to understand changes and risks in country markets and sectors. They also help the ONS to sense check their calculation of financial services output in GDP. The data are defined and prepared in accordance with international rules on the preparation of economics statistics, known as the System of National Accounts (SNA 2008) and the Balance of Payments Manual (BPM6, maintained by the IMF). Form DQ and the instructions for how to complete it are communicated on the Bank website. Please see the appendix for links and further details to the requirements mentioned above.

Key challenges with the Quarterly derivatives return

Whilst the existing return has been operating for a number of years, there are a number of opportunities and drivers for change:

  • Firms are likely to process and reconcile the data in different ways which means that the data is not necessarily comparable
  • Firms have to maintain a range of mapping tables associated with products and counterparties which are expensive to maintain and likely to be implemented in different ways
  • The Bank undertake a range of plausibility checks with firms as quality assurance but these can be unnecessarily time consuming for firms
  • The existing way the requirements are communicated via the Bank’s website require further interpretation and is not user friendly
  • There are proposed new changes to the international statistical standards which will call for further detail of required derivatives data but cannot be delivered through existing data collection
  • There are a wide range of parallel data-related initiatives in the derivatives space which may provide solution opportunities

Commercial Real Estate (CRE) reporting


A report by the Investment Property Forum and Real Estate Finance Group, sub titled “Recommendations for reducing the risk of damage to the financial system from the next commercial real estate market crash” was published in May 2014. This outlined the importance of the role played by commercial real estate (CRE) in the wider economy and in particular its contribution to economic growth and prosperity, as well as its potential to cause or exacerbate financial crises. Among the recommendations was the creation of a comprehensive UK CRE loan database to help the regulator understand the market better, and participants make more informed decisions around their lending. There are a number of existing voluntary collections in the market.

Key challenges with the CRE

  • The existing collections do not capture all participants (eg it excludes non bank entities)
  • The current underwriting collection is adhoc. It requires authorisation to collect the data each year which has implications for both the Bank and firms who are uncertain whether it will happen every year and have to set resource aside to manage the process
  • There are some specific data standards issues which need to be tackled - for example in how valuation data is defined and created
  • There are a number of existing voluntary collections in the market

How can you help?

We are reaching out to suppliers to support our design phase for our use cases in order to find potential solutions to the issues we face. Any potential solutions will need to address elements or all of the ‘How can we’ statements which are outlined in the design brief below. Potential solutions will also need to cover how they could deliver benefit in the short, medium and longer term. The ‘How can we’ statements have been identified as the priority areas to solve to deliver value for stakeholders and have been defined after our discovery phase. We will be holding demo days where interested parties can describe:

  • The solution idea(s)
  • The key hypotheses for change/ benefits
  • Key milestones on a theoretical implementation road-map (covering initial steps, and well as longer term milestones)
  • Key implementation risks

All submissions will be reviewed by the joint transformation programme. Some vendors will be invited to attend one of our demo days to present their proposal to a small panel of representatives from the FCA, BoE and reporting firms involved in the joint transformation programme. We may also follow up separately with other vendors depending on the basis of the proposal.

If you would like to submit a potential solution for consideration, please outline your proposed solution demonstration covering the key areas described above in this template and send to Please use the attached template for your submission. Please note submissions should be no more than two pages. We will then let you know if we intend to include your proposal in this set of demo days.

Please note, this is not part of a procurement process. Any procurement process will be a separate process and may include aspects of information gained from this demo process.

The joint transformation programme will be carrying out a similar process for further use cases. These will be communicated at a later date.

The Design Brief and “How Can We” statements

Quarterly Derivatives Return

We have prioritised two main focus areas. One in an area related to data standards, and one related to reporting instructions. These are the areas we are seeking ideas for solutions for. The characteristics of the solutions are that they need to:

  • Focus on the precise data requirements of this return today and possibly in the future
  • Delivers a range of benefits including more consistent/ trusted data, flexibility, scalability and a positive trade-off between costs and management benefits for firms/Bank over time
  • Deliver end to end impact/ value to stakeholders involved in the production and use of the data, including Firms, the Bank of England, the ONS and international bodies such as the IMF and Bank for International Settlements
  • Be easily understandable for a range of audiences
  • Define critical no regret features/solutions which could deliver value in the short term as well as have longer term impact

We will be assessing the proposed solutions against these characteristics.

The ‘How can we’ statements – Quarterly Derivatives Return use case

Statement One

How can we meet the quarterly financial derivatives market reporting needs of the current and future anticipated international standards (SNA, BoP) requirements in the most consistent, effective and efficient way

Delivery of the DQ report by firms involves calculations and mappings which are costly to deliver, and are reconciled in different ways between businesses. Whilst current reporting meets the needs of the ONS, moving forward proposals for future additions to these reporting standards have been published by IMF that will require extended and more detailed data submissions. For DQ, this is part of a complex derivatives reporting environment which raises challenging questions concerning how counterparties, product, services, currencies, costs and income are understood, defined and recognised. At the same time in the derivatives area there are a wide range of initiatives associated with data standards which could potentially inform the way forward by offering efficiencies. We would like to understand what sorts of solutions could be implementable to deliver benefits for the ONS, Bank and Firms to increase trust in the market and reduce costs for form DQ.

Statement Two

How can we communicate and embed DQ-related reporting requirements in the most effective, consistent and flexible way?

The current BoE website describes the requirements for form DQ (see links in the appendix) but requires considerable interpretation leaving scope for inconsistency, and is not user friendly. We would like to understand how DQ - related reporting requirements could be communicated in a way which minimises ambiguity and makes it easier to deliver high quality consistent data.

The ‘How can we’ statements – Commercial Real Estate (CRE) reporting use case

Statement One

How can the Bank efficiently get the CRE data it needs to proactively monitor and intervene if necessary?

Whilst some progress has been made in creating a voluntary CRE database over the last few years, we are still some distance away from industry committing to and delivering the scope and range of data the Bank needs to perform its role in the market. It is known that some jurisdictions have made good progress in this area, so there are potentially some transferrable ideas and solutions. We would like to explore what ideas the community has to accelerate progress in this area.

Statement Two

How can we communicate and embed (CRE-related) reporting requirements in the most effective, consistent and flexible way between the Bank and firms including expectations, feedback and other useful information for firms

The current approach to communicating expectation and feedback does not facilitate shared understanding. How could this improve?

Key dates

  • Publication of the Request for Input – 30 November
  • Submission of proposed solution sent to – 10 December
  • Response from the joint transformation programme to selected solution providers with time and date of demo by 21 December
  • Presentation at demo day
    • Quarterly Derivatives Return use case – 12/13 January 2022
    • Commercial Real Estate (CRE) database use case – 19/20 January 2022


Quarterly Derivatives Return use case

Bank of England – forms, definitions and validations
Bank of England – Transforming data collection
Form DQ box codes
Form DQ definitions
Future of Finance report
IMF Balance of Payments Manual (BPM6)
IMF FITT Draft Guidance Note, 'F.4 Financial Derivatives by Type' (2021)
System of National Accounts 2008 (SNA 2008)

CRE use case

BPF - Commercial Property Forum - London
CRE Underwriting Standards Review Template
The UK Commercial Property Lending Report (Bayes Survey)
Financial Stability Reports | Bank of England Commercial property has featured as a focus in several of the last few financial stability reports
Investment Property Forum and Real Estate Finance Group (Recommendations for reducing the risk of damage to the financial system from the next commercial real estate market crash)
Stress Test Data Framework Manual (
The Federal Reserve - reporting firms for CRE

Transforming Data Collection and the Joint Transformation Programme

Bank of England – Transforming data collection
Transforming data collection from the UK financial sector - the Transformation Plan