Standard methods of measuring welfare overstate cost of living increases by ignoring new products and demand shifts

See What big data tells us about real income growth by Stephen Redding and David Weinstein. Redding is Professor of Economics at Princeton. Weinstein is Professor of Economics at Columbia. Excerpt:

"Our index enables us to identify a novel form of bias that arises from the assumption of time-invariant demand for each good in existing price indexes. ‘Consumer valuation bias’ arises whenever expenditure shares respond to demand shifts. Since conventional indexes assume that expenditure shares are only affected by price changes, they will be biased whenever expenditure share changes are correlated with demand shifts. For example, if higher consumer demand causes prices to rise, a conventional index will overstate cost-of-living changes because it will not adjust for the fact that some of the price increase is offset by the higher utility per unit associated with the demand shift.

Our second main insight is to develop a novel way of estimating the elasticity of substitution between goods. Extant approaches focus on identification from supply and demand systems. However, we show that one can also identify this parameter by combining information from the demand system and unit expenditure function. One of the desirable properties of this ‘reverse-weighting’ estimator is that it minimises departures from money-metric utility given the observed data on prices and expenditure and the assumed constant elasticity of substitution utility function.

Finally, we use barcode data to examine the properties of our unified price index and reverse-weighting estimator. We find that we obtain reasonable elasticity estimates in the sense that they are similar to those identified using other methodologies on the same data. Moreover, the consumer valuation biases in existing indexes appear to be quite substantial, suggesting that allowing for demand shifts is an economically important force in understanding price and real income changes.  

Figure 2. Changes in cost of living for various indexes

We can see these differences at the aggregate level in Figure 2, which plots the expenditure-share-weighted average of the changes in the cost of living across product groups for each of the different index numbers over time, again using the initial period expenditure share weights. Not surprisingly, the Fisher, Törnqvist and Sato-Vartia result in almost identical changes in the cost of living that are bounded by the Paasche and Laspeyres indexes. This similarity is driven by the fact that they all assume no demand shifts for any good. The distance between the Sato-Vartia index and the Common-Goods Unified Price Index tells us the importance of the consumer valuation bias and the distance between the Sato-Vartia and the Feenstra-CPI indicates the value of the adjustment for changes in variety. In other words, big data suggests that standard methods of measuring welfare overstate cost of living increases by several percentage points per year because they ignore new goods and demand shifts."