I was pleased to speak this morning at a conference organized by the Brookings Institution about the uncertainty of budget estimates. For a session on “Communicating Uncertainty to Policymakers,” I offered CBO's perspective and Robert Chote, who is the Chairman of the U.K. Office for Budget Responsibility (which is similar in some ways to CBO), offered the perspective of his organization.
My colleagues and I at CBO are acutely aware of the uncertainty of the budgetary and economic estimates we provide to Congress. We view our estimates as representing the middle of the distribution of possible outcomes. We frequently explain the estimates that way to Members of Congress and their staffs, and we regularly discuss risks to our estimates. For example, our reports on the budget and economic outlook over the next 10 years and over the long term have sections or chapters discussing key sources of uncertainty.
We have also worked hard in the past few years to quantify the uncertainty of more of our analyses. However, there are important limitations currently on our ability to quantify uncertainty and to help legislators make effective use of such quantification. I began by discussing the ways in which we quantify uncertainty and then turned to the limitations we see.
In What Situations Does CBO Quantify the Uncertainty of its Estimates?
We quantify the uncertainty of our estimates in a number of contexts, and I gave four examples. First, when we estimate the macroeconomic effects of changes in fiscal policies, we regularly provide both a “central estimate” and a “range.” The ranges allow for uncertainty about the response of labor supply to changes in tax rates, the effect of changes in budget deficits on national saving and international capital flows, and other factors, and they are intended to cover roughly two-thirds of the distribution of possible outcomes. One example of such analyses is our estimates of the short-term and long-term economic effects of alternative paths for the federal debt (see chapter 6 of this year’s Long-Term Budget Outlook.
Second, the Long-Term Budget Outlook regularly includes alternative projections. In July 2014, we showed what would happen to the budget if four key underlying factors—the decline in mortality, the growth of productivity, interest rates, and the growth of health care costs—differed from the values that are used in most of the report (see chapter 7 of the report). The chapter also discusses other sources of uncertainty that we do not quantify. Similarly, we publish additional information on our long-term projections for Social Security that regularly includes a range of possible outcomes for the Social Security trust funds based on the historical year-to-year variation in key demographic and economic factors, including fertility and mortality rates, interest rates, and the growth of productivity.
A third form of quantification of uncertainty occurs in some analyses of specific federal policies. Examples include our estimates of the effects on employment of raising the minimum wage and our estimates of the effects of adopting a more competitive system for Medicare.
A fourth and slightly different way in which we quantify uncertainty is to analyze how actual outcomes have differed from our estimates. One example is a recurring evaluation of our economic forecasting record and another is a forthcoming evaluation of our revenue forecasts. We also publish rules of thumb for how much budget outcomes vary for a given variation in economic variables.
Why Are Most of CBO’s Estimates Presented Only as Point Values?
Still, the vast majority of CBO’s estimates are provided as point values without ranges. There are three principal reasons.
First, the Congressional budget process requires point estimates of the budgetary effects of proposed legislation. Budget resolutions provide committees with allocations of funds expressed as point values, and the House and Senate Budget Committees track the estimated budgetary effect of approved legislation using point values. The statutory pay-as-you-go law enacted in 2010 and various parliamentary rules of the House and Senate all are enforced using point values. A range that encompassed some values that complied with budgetary rules and others that did not would not be useful in following those procedures or enforcing those rules, so CBO needs to produce point estimates. One might argue that the Congress should change its approach to explicitly reflect uncertainty, but developing comprehensible procedures and rules that used ranges of figures rather than point values seems quite daunting.
A second reason that CBO usually does not provide ranges for its estimates is that we often lack a strong analytical basis for constructing such ranges. One obstacle is that most of the models and estimating techniques we use are not formal probability models, so they do not readily yield measures of uncertainty. For example, our projections of interest rates draw on futures markets, from which it is more straightforward to elicit expected values than uncertainty about those values.
Another obstacle is that the underlying research on which we draw generally provides only limited information on uncertainty. For example, our analysis of the effects of raising the minimum wage drew on dozens of studies, so we could form a sense of uncertainty related to time, place, and modeling approach as well as sampling uncertainty. But our analysis of the effects of prescription drug use on other medical spending was based on just the handful of studies that have been done in that area, so forming a sense of uncertainty would have been more difficult.
And a further, related challenge is a lack of time. We are often rushing to finish analyses before Congressional action on an issue is expected to occur, and doing the additional modeling or gathering the additional information needed to quantify uncertainty can take considerable time.
The third main reason that CBO usually does not provide ranges of estimates is that we are still developing ways to help legislators make effective use of our quantification of uncertainty. Part of the challenge is that providing ranges for estimates sometimes muddies, rather than enhances, general understanding of our analysis. For example, when we report ranges, people who would prefer that our estimate be smaller tend to cite the bottom of the range, and people who would prefer that our estimate be larger tend to cite the top of the range. That can make the public discussion of our analysis quite confusing, and we have a limited ability to clear up that uncertainty.
Another part of the challenge is that it is often unclear how legislators might respond to the quantification of uncertainty, and we, and the analytic community more generally, have developed only limited guidance in this area. Surely it is useful for legislators to be aware of the uncertainty of budgetary and economic estimates, and that is the main reason we quantify it. But what else might they do with such quantification?
Legislators might want to adopt only policies that were very likely to increase or decrease a variable of interest. For example, we were asked a few years ago whether a certain change in health care benefits that had been implemented had resulted in budgetary savings for the Department of Defense, and we concluded that it probably had. Conducting that analysis when the change in policy was being considered might have been helpful. However, it is unclear how often policymakers might apply criteria of that sort.
As another example, in some situations, legislators might want to adopt policies with a smaller variance of budgetary effects in order to reduce the risk of large fiscal problems. For example, understanding the extent of uncertainty about future federal spending that arises from uncertainty about lifespans might affect whether policymakers want to index eligibility ages for certain programs to lifespans. In other situations, though, legislators might want to adopt policies with a larger variance of future budgetary effects as a means of experimenting to identify the best policies. I think analysts can help to sort out those different situations, but it’s a complicated problem.
As a final example, maybe legislators might want to respond to greater uncertainty about long-term budget projections by taking larger deficit-reducing actions in order to reduce the unfavorable tail of the distribution. On the other hand, legislators might want to respond to greater uncertainty by focusing less on projected long-term outcomes altogether. I think that more research—like that presented at this conference—can help to guide policymakers on that issue as well.
But the profession is still in the fairly early days of this sort of analysis. As a result, policymakers’ direct use of our estimates of uncertainty is limited at this point, which in turn leads us to devote only a limited portion of our time to producing such estimates.
I concluded by emphasizing that we are working hard to quantify the uncertainty of more of our analyses. However, there are important limitations on our current ability to do that and to help legislators make effective use of the information.