The Models Were Wildly Wrong about Reopening Too

By Phillip W. Magness.

"The follow-up ICL paper from May attempted to model the effects of reopening in 5 US states: New York, Massachusetts, California, Washington, and Florida. In all five cases, the Imperial College team predicted an aggressive rebound of COVID-19 fatalities under even the most modest relaxation of stay-at-home policies and practices.

To illustrate this pattern, the ICL team presented three scenarios based on the expected change in human mobility in each state after the lifting of lockdown restrictions. The first scenario kept the lockdowns in place, assuming that mobility would remain constant at its severely reduced post-lockdown rate. Under the other two scenarios, the ICL team assumed a 20% and 40% increase of mobility corresponding with the reopening process.

In both of these reopening scenarios, the model depicted a catastrophic rebound of COVID-19 fatalities. As the ICL team itself put it, their model “illustrate[s] the potential consequences of increasing mobility across the general population: in almost all cases, after 8 weeks, a 40% return to baseline [mobility] leads to an epidemic larger than the current wave.” Media reports at the time touted the study’s dire warnings as reasons to stall the reopening process – even at its sluggish pace of recurring 2-week delays and extensions.

More than 8 weeks have passed since the publication of the ICL team’s warnings against reopening, meaning we can now see how their model performed.

As with other examples of ICL COVID modeling, their attempt to predict the effects of a US reopening can only be described as an embarrassing scientific failure.

The image below shows the three modeled scenarios from May, as depicted in the ICL report for the five states under consideration. Note that even under the “constant mobility” scenario of remaining under lockdown, their model predicted an increase in COVID deaths for every state except New York, which had already peaked. Under the reopening scenarios where mobility increased 20% and 40% respectively from its lockdown state, all five states were predicted to surge into apocalyptic territory by the middle of July. Under the 40% scenario, this even entailed upper boundaries of more than 4,000 deaths per day (the bands represent the 95% confidence interval). Massachusetts and New York, two of the hardest-hit states from the first wave back in March and April, would easily match or exceed their previous COVID-19 daily death records.

To see how these predictions held up, I indicated the daily death totals for each state for July 20^th with a small red dot on the graphs above. As you can see, the actual totals are below the ICL model’s predictions in every scenario. In Massachusetts, the current daily death totals are even falling below the lower boundary of the ICL model’s projections for both its 20% and 40% mobility increase scenarios.

Coronavirus cases and deaths have spiked in two of the modeled states, Florida and California. As of the week of July 20^th, both are averaging between roughly 100 and 150 deaths per day. Yet even with this “second wave” spike, Florida and California are only showing about one-tenth of the projected deaths that the Imperial College modelers predicted for this time back in May.

In New York, Washington, and Massachusetts, daily death counts have dropped to the low double-digits and remain a tiny fraction of the ICL predictions for mid-July.

Although all five states remain under COVID-19 restrictions of varying degrees, even partial reopening has increased mobility at levels that match or exceed the ICL’s modeled scenarios. The main Google mobility indicators for Massachusetts are depicted below for reference, and show a clear upward trend since the time of the ICL predictions in mid-May.

These patterns confirm that US mobility trends are increasing as lockdown restrictions are slowly lifted, and as society moves toward reopening. They therefore show that the ICL model correctly anticipated one effect of relaxing the lockdowns.

At the same time though, the ICL model severely overstated the projected mortality associated with reopening in all five states. Actual data do not map onto any of their scenarios, including the broadest of the three predictions for reopening. States that peaked back in March and April show no signs of a resurgence, let alone the predicted resurgence that would surpass the first wave. And states that are undergoing later surges are still well below the ICL team’s predictions – so far below that they barely even register on the graphs.

As with other predictions from the ICL team, the May paper likely faltered due to a fundamental error in its underlying code. These flawed ICL models begin with an unproven assumption, namely that lockdowns are effective at combating the coronavirus. The models are therefore automatically calibrated to produce a sharp spike in deaths after the removal of lockdowns or any move toward reopening.

As we’re now seeing in actual data however, that assumption is grossly exaggerated."