Post by icemandios on Apr 23, 2021 13:38:52 GMT
Niels Bohr, the Nobel laureate in Physics and father of the atomic model, is quoted as saying, “Prediction is very difficult, especially if it’s about the future!” This quote serves as a warning of the importance of testing a forecasting model out-of-sample.
Tyler Durden's Photo
BY TYLER DURDEN
FRIDAY, APR 23, 2021 - 03:30 AM
Authored by Phillip Magness via The American Institute for Economic Research,
A fascinating exchange played out in the UK’s House of Lords on June 2, 2020. Neil Ferguson, the physicist at Imperial College London who created the main epidemiology model behind the lockdowns, faced his first serious questioning about the predictive performance of his work.
Ferguson predicted catastrophic death tolls back on March 16, 2020 unless governments around the world adopted his preferred suite of nonpharmaceutical interventions (NPIs) to ward off the pandemic. Most countries followed his advice, particularly after the United Kingdom and United States governments explicitly invoked his report as a justification for lockdowns.
Ferguson’s team at Imperial would soon claim credit for saving millions of lives through these policies – a figure they arrived at through a ludicrously unscientific exercise where they purported to validate their model by using its own hypothetical projections as a counterfactual of what would happen without lockdowns. But the June hearing in Parliament drew attention to another real-world test of the Imperial team’s modeling, this one based on actual evidence.
As Europe descended into the first round of its now year-long experiment with shelter-in-place restrictions, Sweden famously shirked the strategy recommended by Ferguson. In doing so, they also created the conditions of a natural experiment to see how their coronavirus numbers performed against the epidemiology models. Although Ferguson originally limited his scope to the US and UK, a team of researchers at Uppsala University in Sweden borrowed his model and adapted it to their country with similarly catastrophic projections. If Sweden did not lock down by mid-April, the Uppsala team projected, the country would soon experience 96,000 coronavirus deaths.
I was one of the first people to call attention to the Uppsala adaptation of Ferguson’s model back on April 30, 2020. Even at that early date, the model showed clear signs of faltering. Although Sweden was hit hard by the virus, its death toll stood at only a few thousand at a point where the adaptation from Ferguson’s model already expected tens of thousands. At the one year mark, Sweden had a little over 13,000 fatalities from Covid-19 – a serious toll, but smaller on a per-capita basis than many European lockdown states and a far cry from the 96,000 deaths projected by the Uppsala adaptation.
The implication for Ferguson’s work remains clear: the primary model used to justify lockdowns failed its first real-world test.
In the House of Lords hearing from last year, Conservative member Viscount Ridley grilled Ferguson over the Swedish adaptation of his model: “Uppsala University took the Imperial College model – or one of them – and adapted it to Sweden and forecasted deaths in Sweden of over 90,000 by the end of May if there was no lockdown and 40,000 if a full lockdown was inforced.” With such extreme disparities between the projections and reality, how could the Imperial team continue to guide policy through their modeling?
Ferguson snapped back, disavowing any connection to the Swedish results: “First of all, they did not use our model. They developed a model of their own. We had no role in parameterising it. Generally, the key aspect of modelling is how well you parameterise it against the available data. But to be absolutely clear they did not use our model, they didn’t adapt our model.”
The Imperial College modeler offered no evidence that the Uppsala team had erred in their application of his approach. The since-published version from the Uppsala team makes it absolutely clear that they constructed the Swedish adaptation directly from Imperial’s UK model. “We used an individual agent-based model based on the framework published by Ferguson and coworkers that we have reimplemented” for Sweden, the authors explain. They also acknowledged that their modeled projections far exceeded observed outcomes, although they attribute the differences somewhat questionably to voluntary behavioral changes rather than a fault in the model design.
Ferguson’s team has nonetheless aggressively attempted to dissociate itself from the Uppsala adaptation of their work. After the UK Spectator called attention to the Swedish results last spring, Imperial College tweeted out that “Professor Ferguson and the Imperial COVID-19 response team never estimated 40,000 or 100,000 Swedish deaths. Imperial’s work is being conflated with that of an entirely separate group of researchers.” It’s a deflection that Ferguson and his defenders have repeated many times since.
As it turns out though, Ferguson and the Imperial College team were being less than truthful in their attempts to dissociate themselves from Sweden’s observed outcomes. In the weeks following the release of their well-known US and UK projections, Ferguson and his team did in fact produce a trimmed-down version of their own modeling exercise for the rest of the world, including Sweden. They did not widely publicize the country-level projections, but the full list may be found buried in a Microsoft Excel appendix file to Imperial College’s Report #12, released on March 26, 2020.
Imperial’s own projected results for Sweden are nearly identical to the Uppsala adaptation of their UK model. Ferguson’s team forecast up to 90,157 deaths under “unmitigated” spread (compared to Uppsala’s 96,000). Under the “population-level social distancing” scenario meant to approximate NPI mitigation measures such as lockdowns, the Imperial modelers predicted Sweden would incur up to 42,473 deaths (compared to 40,000 from the Uppsala adaptation).
The Imperial team did not specify the exact timing of when they expected Sweden to reach the peak of its outbreak. We may reasonably infer it though from their earlier US and UK model, which anticipated the “peak in mortality (daily deaths) to occur after approximately 3 months” following the initial outbreak. That would place Sweden’s peak daily death toll around mid-June, or almost the exact same time period as the Uppsala team’s adaptation.
Read more