""Is [the U.S. Agency for
International Development] a good use of resources?" James Macinko, a
health policy researcher at UCLA, asked in an NPR interview this
month. "We found that the average taxpayer has contributed about 18
cents per day to USAID." That "small amount," Macinko estimated, had
prevented "up to 90 million deaths around the world."
Macinko was referring to a study he coauthored, which was published in the prestigious medical journal The Lancet.
In addition to estimating that USAID programs had saved 90 million
lives from 2001 to 2021, Macinko and his colleagues project that if the
Trump administration's USAID cuts continue through 2030, an estimated 14
million people could die who otherwise would have lived.
In the same NPR story, Brooke Nichols,
an infectious disease mathematical modeler and health economist at
Boston University, gushed about the study. "Putting numbers to the lives
that could be lost if funding isn't restored does something very
important," she said. "I like [the study's] statistical approach; it was
really well done and robust."
To
the contrary, the study's statistical approach was poorly done and not
at all robust. But you need not be an economist or a mathematician to
recognize that its results are absurd. The authors failed to apply
common sense to the numbers, and so did reporters at NPR, the BBC, the Associated Press, NBC, and other news outlets that amplified the study's findings.
Is
it plausible that USAID programs saved more than 90 million lives from
2001 to 2021? Let's compare that to the total decline in worldwide
mortality during the same period.
In 2001, the United Nations reports,
there were 52.43 million deaths, or 8.4 per 1,000 people. In 2002, the
death rate fell slightly; if it hadn't, 370,000 additional people would
have died. In 2003, if the 2001 death rate had stayed constant, 750,000
more would have died, and so on. If you add up all the people who would
have died over 21 years if the 2001 death rate had remained constant,
you get 79 million lives saved.
The Lancet
study claims USAID programs saved more than 90 million lives during
this period. In other words, USAID was allegedly responsible for the
entire global improvement in mortality, plus another 11 million lives.
If
you drill down into the numbers, the claim gets even more absurd. Most
of the mortality decline (47 million of the 79 million) occurred in
China, which received only 73 cents per capita annually from USAID. The
least developed countries—the ones with the highest per capita USAID
spending—actually saw an increase in mortality of 8 million during the study period, thanks to higher average death rates from 2001 to 2021.
Even if you believe that foreign aid is the primary cause of the declining death rate, these numbers make no sense. USAID comprises about 60 percent of U.S. foreign aid, and the U.S. contributes about a third of all government aid worldwide. Private cross-border charity dwarfs
the USAID budget. Was all that money wasted while USAID funds were
well-spent? Did advances in medicine, agriculture, public health, and
economic growth have no role?
Anyone claiming that USAID was the sole source of declining global mortality during the two decades covered by The Lancet study bears a tremendous burden of proof. Yet the study does not provide convincing evidence of any mortality reduction attributable to USAID.
Studies of this sort often involve
problematic data, plagued by missing values, inaccurate numbers, and
changes in definitions. Such data require complex processing. To avoid
reporting risible results, researchers must do constant reality checks.
If simple analysis reveals an effect, more complex methods can refine
it, making the result more certain and precise. But when simple methods
show no effect, as in this case, it is dangerous to rely entirely on
complex methods whose results defy common sense.
The
authors of the study began with a common academic approach called
"regression analysis." They compared average mortality declines in
countries with little or no per capita USAID spending to declines in
countries with high per capita USAID spending.
Regression
analysis shows only correlation, which does not demonstrate causation.
Most researchers are careful to note this distinction. But Macinko et
al. claimed their analysis showed USAID caused mortality declines, which
is statistically impossible.
That
assumption is particularly problematic in this case because a lot of
USAID spending went to the same countries that also get aid from other
sources. Effectively, the authors gave USAID credit for all the aid that
countries received, no matter the source.
In any case, simple regression
analysis did not demonstrate any correlation between per capita USAID
spending and mortality declines.
For
four of the 21 years in their study, Macinko et al. used "dummy
variables" to ignore data showing USAID was associated with mortality increases.
Their excuses for excluding two of the four years were the 2008 global
economic shock and the 2020 COVID-19 pandemic. So why exclude the
remaining two? "To adjust for major economic and health shocks," the
authors say, with no clue about what those shocks were and why they were
more important than shocks in other years.
It's
not clear why shocks should be ignored in the first place. Does it not
matter that USAID recipient countries were hit harder by COVID and the
global financial crisis than other countries? Does USAID only save lives
in calm times and cost lives in turbulent ones? If so, don't we care
about the net lives saved or lost at all times?
Next,
the researchers tested 48 "control variables." The purpose of these
adjustments is to account for factors unrelated to USAID spending that
affect mortality rates. Unfortunately, in this case, there are no good
controls. For example, the study included education spending and the
availability of piped water; however, since USAID funds education and
water infrastructure, these factors are related to the amount of USAID
received. All the control variables tested were things affected by USAID
spending.
Including controls that are causally
linked to your main variable of interest—USAID spending in this case—can
cause illusory effects to appear statistically significant. This is
particularly problematic when you select a large number of control
variables relative to the number of data points and choose those control
variables from a long list of candidates.
Macinko
et al. did not preregister their controls, meaning they did not publish
a research plan that committed them to a particular set. That safeguard
aims to prevent researchers from falling prey to the temptation of
selecting control variables based on whether the results align with
their preconceptions.
What
about the estimate that the Trump administration's USAID cuts, if
continued, will result in 14 million preventable deaths by 2030?
Programs like USAID have numerous consequences, both positive and
negative, and it is impossible to accurately calculate or project their
impact with any confidence. Pretending to have scientific confidence in a
quantitatively dubious measure, such as lives saved, is irresponsible
and leads to a loss of trust in science.
The Lancet study
seems designed to generate a partisan talking point, suggesting that
anyone who supports the Trump administration's actions values 18 cents
over 90 million lives. Note the trick of making the cost look small by
dividing it among 150 million U.S. taxpayers and 365 days per year while
making the benefit look large by totaling it over the entire globe for
21 years.
This study has nothing to do with science. It waves the bloody shirt while feigning scientific detachment."