Non-standard errors

Staff working papers set out research in progress by our staff, with the aim of encouraging comments and debate.
Published on 17 December 2021

Staff Working Paper No. 955

By Albert J Menkveld et al. Bank of England co-authors: Gerardo Ferrara and Simon Jurkatis

In statistics, samples are drawn from a population in a data-generating process (DGP). Standard errors measure the uncertainty in sample estimates of population parameters. In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). We claim that EGP variation across researchers adds uncertainty: non-standard errors. To study them, we let 164 teams test six hypotheses on the same sample. We find that non-standard errors are sizeable, on par with standard errors. Their size (i) co-varies only weakly with team merits, reproducibility, or peer rating, (ii) declines significantly after peer-feedback, and (iii) is underestimated by participants.

Non-standard errors