Commercial Launch (Forecast) Fever
This will be a difficult entry to pen without being accused of being “hostile” to the emergent U.S. commercial space industry (which I am not). It is – as with most posts in this blog – a discussion of data, more specifically, data from forecasts. Very optimistic forecasts. If it makes it any easier to swallow, very optimistic government forecasts.
First, some background. Each year since 1995, the FAA, which has responsibilities for regulation of commercial space launches, has issued 10-year forecasts of future launch vehicle demand for both geostationary/geosynchronous (GSO, typically communications) and non-geostationary (NGSO, typically low earth orbit) payloads. These forecasts form the backbone of this analysis.
Second, some context. The commercial space bubble of the 1990s, fueled by high expectations in the communications satellite arena, is well-known. Those responsible for launch forecasts were also well aware of systematic optimism in their integration and run-out techniques and have worked systematically to reduce them. My goal here is not to savage the hard-working folks at FAA, but to highlight what happens when models quietly diverge from reality, and to help create better educated “consumers” of the official forecasts.
This graphic comes from the 2010 FAA/COMSTAC forecast, and demonstrates FAA’s good faith in providing both forecast and actual data against which to validate model skill. Included in this chart are total (GSO+NGSO) launch demand forecasts from 2000 forward only. An appendix in the 2010 report also includes NGSO forecast vs actual data for 2004 forward only. The systematic positive bias is plain to see, and has typically been remarked upon within the reports themselves.
However, this chart has some serious defects which very much get in the way of communicating the wide disparity between available forecasts in any given year, and “truth”. They are almost certainly unintentional defects, but the net effect is significant:
- Forecasts prior to 2000 have been eliminated from this chart, and prior to 2004 from the NGSO-only chart. This has the effect of removing some of the most egregiously bad bubble-era forecasts from the composite (see below for the effects of the actual bubble-era data). In fairness, the forecast model has not been stationary over that time frame, undergoing tweaking and evolution. Also in fairness, the caption represents this graphic as “Commercial Space Transportation Forecasts” – i.e., the performance of the FAA office – not, “Forecast model version 3.x forecasts”. The implication is that this is a graphic of COMSTAC’s historical performance, and as such, it should include all data.
- In compiling this graph, FAA has excluded all of the “robust market” alternative scenarios included in old forecasts (a practice stopped in the mid-2000′s when it became obvious that there wasn’t a robust market in the immediate future). By instead showing only the conservative forecast, it obscures the fact that consumers of the forecasts in the Y2K time frame had a very wide range (one could argue, uselessly wide) of forecasts to “choose from”. Again, see below for the forecast data.
- Visual sleight of hand with the 2010 forecast gives the illusion of greater skill than actually exists. Up until 2009, all forecasts are represented on the chart by lines, and “actuals” by the blue sand chart wedge. However, the 2010 forecast is shown by seamlessly extending the actuals sand chart, not by including an additional line. The effect is that all of the old 2010-2020 forecast run-outs now appear to live “within” an actuals wedge, the grey wedge (which of course, is nothing of the sort, it’s just another forecast). This hides the fact that prior to 2009, the forecasts almost never intersected with reality, instead, systematically overestimating demand. This is either intentionally deceptive (unlikely) or really unfortunately bad chartsmanship (likely).
- By visualizing the joint NGSO and GSO forecast, the graphic downplays the severe errors in the very badly modeled NGSO forecast, since almost all of the “skill” in the model is from the comparatively stable GSO market.
Again, given FAA’s transparency and ongoing efforts to improve their models, this is almost certainly due more to poor chartsmanship and the difficulty of tracking and integrating heterogeneous data over multiple decades, than any intent to deceive. To get a clearer picture, I’ve created the graphic at right, which shows instead the probability distribution of FAA forecasts for each year, compared against the actuals. The intent is to show the full range of forecasts “available to choose from” for potential consumers of the data. In compiling the forecasts, I have treated the “baseline” and “robust” market historical forecasts as equally valid, since in reality, consumers of the historical forecasts had little basis to select between them.
In this graphic, the dashed line shows the median of all forecasts, the innermost dark grey band contains the “most central 33%” of forecasts (33%-67% percentiles), the next lightest grey band contains the “most central 50%” of forecasts (25%-75% percentiles), and the outer very light bands contain 90% and 100% of all forecasts, respectively. (In short, the intensity of grey shading is intended to cue viewers towards the most frequently occurring forecast values). This view provides a clearer view of what a “bubble” looks like in practice. The compilation is troubling not only for revealing the excessive optimism, but also for calling into question the model’s skill in predicting any trends whatsoever. Other than the initial rise from 1996-1997 (forecasted two years too early), there is little evidence that the forecasts as a whole actually capture any signal in the actual time series. A simple flat runout of the previous year’s actual launch rate (known as a “persistence” forecast in weather circles) would likely have provided better overall skill as a model (I need to verify this, but eyeballing the graphic, I’m 99% confident of that outcome).
The situation gets worse if we split the forecast into its non-geostationary and geostationary components, below. The bulk of the forecast “skill” is coming from the stable GSO market (at bottom), although even there the forecasts are optimistically biased, lying atop an essentially flat trend. The NGSO forecast performance contains even less “skill” than the aggregate above.
So – why share this bad news? The point most certainly isn’t to be cute and condemn the folks at FAA who work hard to forecast an extremely fragile and unstable emergent market – a thankless and “no-win” task, almost by definition. Nor is it to make life any more difficult for the struggling commercial space sector.
The point is to help create educated consumers of the official data, and to heighten awareness of how critically dependent we are on the industry’s own self-reported future manifests, which are almost certainly optimistically skewed (since they do not allow for unexpected contingencies). Indeed, in the aggregate, the 10-year forecasts actually have performed worse in the first three years of each forecast, than in the last seven years – one hypothesis could be that this is bias by overly optimistic near-term industry manifests.
FAA continues to work to improve its model and has been fully open in communicating its performance. Until, however, the forecasts are consistently able to eliminate positive bias, and to demonstrate an ability to actually detect trend signals, at minimum a deep “discount factor” should be applied before using them for policy or commercial purposes.