Why We’re So Bad at Predicting the Future of Energy

I’ve long been struck by how hard it is to predict the evolution of our energy system even a few years in advance, never mind 25 or 30 years. I still remember the “peak oil” craze in the mid-2000s, when people were telling me the end of oil was nigh. It sounded convincing right up until it turned out to be wrong.

Let me show you how bad previous predictions have been for the electricity sector.

The Evolution of Our Energy System in Five Charts

Each plot below shows predictions of how a particular source of electricity will evolve, as well as what actually happened. The data comes from the Energy Information Administration and covers the U.S. electricity sector.

We’ll start with coal. In the first plot, the black line shows actual U.S. coal-fired electricity generation. The blue lines are predictions made each year since 2008.

In 2008, coal was expected to produce increasing amounts of electricity into the future. Instead, it immediately started to decline. It took until 2023 for the EIA to begin predicting a long-term decline in coal, despite the fact that coal had been declining for 15 years.

Natural gas, by contrast, has generated an increasing share of U.S. electricity. This is largely due to the tidal wave of cheap natural gas from hydraulic fracturing. The predictions, on the other hand, did not anticipate this.

The takeaway here is that predicting the evolution of our energy system is not just difficult in the long run, e.g., 30 years from now, but also that it’s difficult even in the short run.

If we combine coal and gas, the forecasts look better. This reflects the fact that natural gas has largely replaced coal over the years, so that the underestimate for gas helps cancel out the overestimate for coal.

But even for the combined category, the forecasts vary widely.

Moving on to renewables, here’s solar, including both utility and residential solar:

And here’s wind:

For both energy sources, predictions before 2015 were really bad. What changed after that I can’t say — my guess is they got sick of being so wrong.

Across all energy sources, the 2023 and 2025 forecasts differ sharply from the 2026 forecast. The predictions made for those years assume the persistence of Biden’s Inflation Reduction Act, while 2026 predictions assume the reversal of those policies.

The difference between 2025 and 2026 is an estimate of the role that politics plays in the future evolution of our electricity sector. That we cannot confidently predict who will win future elections or what their policies will be is another very good reason why it’s so hard to predict the future of our energy system.

The Cost of Energy

Why is it so hard to predict the energy mix in our electricity system? One big reason is that it is hard to predict the future rate of innovation. We can see this in a plot of the cost of energy:

I’m using levelized cost of energy as my measure of the cost to produce power from each source. I understand the limitations of LCOE, but for an energy developer, LCOE is the number that counts. Yes, wind and solar are intermittent, but that’s a grid problem. All that matters to the developer is which low-LCOE energy source they can build.

You can see that the price of wind and solar plummeted in the early 2010s, reflecting enormous innovation in the production of renewable energy. That was not predicted by most mainstream forecasts, as confirmed by predictions of wind and solar above.

There has also been a lot of innovation in fossil fuel production, most importantly fracking and horizontal drilling. These technologies drove down the cost of natural gas in the late 2000s and changed the economics of electricity generation almost overnight. Coal plants that had looked like safe long-term investments suddenly faced a cheaper competitor.

Yet this, too, was largely missed. In the late 2000s, many utilities were still trying to build coal plants, unable to see that coal was entering a precipitous decline. TXU Corp., for instance, tried to build 11 new coal plants in Texas in the mid-aughts. Though it was the state’s largest utility at the time, it ultimately got bought out by private equity, who compromised with environmental groups and agreed to build just three of the original 11 proposed plants, two of which are still in operation.

Meanwhile, the restructured TXU declared bankruptcy in 2014, after natural gas prices collapsed.

All of this goes to show that coal was not beaten by a single technology. It was beaten by a sequence of technologies that forecasters failed to anticipate.

Based on economics, coal is now a stone-cold loser. Its remaining advantage is not cost, nor is it speed of construction or flexibility. It is politics. The Trump Administration is forcing coal-fired plants to stay open, and recent reporting suggests these interventions are raising costs for consumers.

In the competition between solar, wind, and natural gas, solar and wind are the cheapest. The combination of low costs and short construction times with the price volatility of natural gas gives wind and solar a huge market advantage, explaining their exponential growth.

Yes, solar and wind are coming for natural gas.

The LCOE plot also shows the profound disadvantage nuclear faces. Nuclear energy costs nearly $200 per megawatt-hour, around four times the cost of wind and solar. And it takes a decade or two to get it online. Without government mandates or heavy policy support, I would say there is little likelihood we will see a nuclear renaissance.

So What Should We Take From This?

Much of the debate in climate policy centers on the cost, difficulty, and timeline for phasing out fossil fuels in order to achieve net zero. You constantly hear pundits and analysts throwing around eye-popping numbers, confidently claiming, e.g., that “it will cost XXX trillions of dollars to reach net zero in our economy by 2050.”

Image courtesy of McKinsey

But if the forecasting failures of the past 20 years have taught us anything, it’s this: We simply have no idea how much decarbonization will cost.

You should treat numbers like McKinsey’s estimate above as guesses. They could be right, but historically speaking, they probably aren’t.

To summarize, here are the reasons why the true cost of reaching net zero remains so uncertain:

• We can’t predict the foundational energy mix. As the charts above show, our ability to forecast the trajectory of the electricity sector even a few years out is abysmal. If forecasters cannot accurately predict the baseline scenario (how much wind, solar, or natural gas will be on the grid), it seems unlikely they will be able to make accurate predictions of how much additional solar and wind will be needed in 2050 to reach net zero.
• Innovation shatters financial models. Long-term cost forecasts rely heavily on estimates of how fast innovation will occur. Such predictions are incredibly hard to make. Almost no one foresaw the exponential drop in the price of solar energy that happened in the late 2000s, nor did experts predict the current plummeting cost of battery storage. Falling battery costs could reshape the electricity system, just as falling costs for natural gas and solar did.
• Geopolitics rewrites the math. External shocks can alter energy economics overnight. Few energy forecasts anticipated wars in Ukraine and Iran, both of which are going to have an enormous impact on our energy mix going forward.

Overall, the uncertainty in these long-term forecasts is enormous. And if history is any guide, the errors are not random. They usually point in the same direction — they overestimate the cost of the energy transition.

One reason is that traditional forecasting models tend to assume slow, steady technological progress. But energy technologies do not always improve that way. Solar, wind, batteries, and fracking all show that costs can change fast when conditions line up. Most models, which assume gradual change, will miss these breaks.

Another problem is that fossil fuels are often treated as stable, low-risk alternatives. They are not. Their prices can swing wildly, and their supply chains are exposed to wars, political instability, and global market shocks. Those costs are real and hard to predict, so they are left out of these estimates.

That is the central point: Estimates of the cost of the energy transition should be treated as conditional guesses built on assumptions about technology, fuel prices, politics, and geopolitics, all of which have repeatedly surprised us.

The lesson of the past 20 years is not that the energy transition will be easy or hard — we really don’t know. Anyone claiming to know the cost decades in advance should be treated with skepticism.

Editor’s note: A version of this article originally appeared in the author’s newsletter, The Climate Brink, and has been repurposed for Heatmap.