As Society Becomes More Complex, Central Planning Becomes Even More Difficult

See Modernity and Liberalism -- A Match That Defies the Progressive's Wisdom by Peter Boettke.

"In one of the most important books in the history of American political life -- Herbert Croly's The Promise of American Life (1909) -- a contrast is drawn between Jefferson and Hamilton.  Hamilton in this work is taken to symbolize professional bureaucracy and a large and effective national government.  The argument was that as America shifted from predominately an agrarian economy to an industrial one, the Jeffersonian vision was no longer a viable model of government for America.  Jeffersonianism was a political philosophy of extreme individualism, and despite the great faith this vision held for the people it was only applicable in a world of pioneers and frontiersmen.  But in the industrial age,  government could no longer be content with protecting negative rights; it needed to actively promote the general welfare.  To achieve this, we could not rely on the "invisible hand" of the market, but instead the very "visible hand" of the modern public administration and a three-pronged program that entailed the nationalization of large corporations, the strengthening of labor unions, and a strong central government.

This progressive vision of government has been in the public imagination of the elite ever since.  It is what guided Keynes's turn to the administrative state to manage investment and maintain full employment levels of output; it is what guided Samuelson's neoclassical synthesis that would enlist the administrative state to correct for macroeconomic imbalances and microeconomic inefficiencies; it was the vision behind John Kenneth Galbraith's demand that the administrative state be structured such to reinforce the countervailing force to manage the relationship between business, labor, and the public sector; and it is the vision that governs the contemporary recipes for the activist administrative state that one can read in Krugman, Stiglitz and Summers.

The vision of bureaucratic public administration as synonymous with modernity has been an article of faith for the establishment elite since Weber and Wilson. But consider the following from Hayek's The Road to Serfdom (2007[1944]) 94-96:

The assertion that modern technological progress makes planning inevitable can also be interpreted in a different manner.  It may mean that the complexity of our modern industrial civilization creates new problems with which we cannot hope to deal effectively except by central planning.  In a sense this is true -- yet not in the wide sense in which it is claimed.  It is, for example, a commonplace that many of the problems created by the modern town, like many other problems caused by close contiguity in space, are not adequately solved by competition.  But it is not these problems, like those of the "public utilities," etc., which are uppermost in the minds of those who invoke the complexity of modern civilization as an argument for central planning.  What they generally suggest is that the increasing difficulty of obtaining a coherent picture of the complete economic process makes it indispensable that things should be coordinated by some central agency if social life is not to dissolve in chaos.
This argument is based on a complete misapprehension of the working of competition.  Far from being appropriate only to comparatively simple conditions, it is the very complexity of the division of labor under modern conditions which makes competition the only method by which such coordination can be adequately brought about.  There would be no difficulty about efficient control or planning were conditions so simple that a single person or board could effectively survey all the relevant facts.  It is only as the factors which have to be taken into account become so numerous that it is impossible to gain a synoptic view of them that decentralization becomes imperative.  But, once decentralization is necessary, the problem of coordination arises -- a coordination which leaves the separate agencies free to adjust their activities to the fact which only they can know and yet brings about a mutual adjustment of their respective plans.  As decentralization has become necessary because nobody can consciously balance all the considerations bearing on the decisions of so many individuals, the coordination can clearly e effected not by "conscious control" but only by arrangements which convey to each agent the information he must possess in order effectively to adjust his decisions to those of others.  And because all the details of the changes constantly affecting the conditions of demand and supply of the different commodities can never be fully known, or quickly enough be collected and disseminated, by any one center, which is required is some apparatus of registration which automatically records all the relevant effects of individual actions and whose indications are at the same time the result of, and, the guide for, all the individual decisions.
This is precisely what the price system does under competition, and which no other system even promises to accomplish.  It enables entrepreneurs, by watching the movement of comparatively few prices, as an engineer watches the hands of a few dials, to adjust their activities to those of their fellows.  The important point here is that the price system will fulfill this function only if competition prevails, that is, if the individual producer has to adapt himself to price changes and cannot control them.  The more complicated the whole, the more dependent we become on that division of knowledge between individuals whose separate efforts are coordinated by the impersonal mechanism for the transmitting the relevant information known by us as the price system.
It is no exaggeration to say that if we had to rely on conscious central planning for the growth of our industrial system, it would never have reached the degree of differentiation, complexity, and flexibility it has attained.  Compared with this method of solving the economic problem by means of decentralization plus automatic coordination, the more obvious method of central direction is incredibly clumsy, primitive, and limited in scope.  That the division of labor has reached the extent which makes modern civilization possible we owe to the fact that it did not have to be consciously created but that man tumbled on a method by which the division of labor could be extended far beyond the limits within which is could have been planned.  Any further growth of its complexity, therefore, far from making central direction more necessary, makes it more important than ever that we should use a technique which does not depend on conscious control. (emphasis added)

Lets rethink the entire intellectual and practical agenda of progressivism -- it just isn't the appropriate philosophical vision of a modern vibrant, diverse and cosmopolitan liberal social order."