LONDON: Commodity indices have proved disastrous investments for pension funds and other institutions in the last decade and disappointed again in 2015.

Back in 2005, Gary Gorton and Geert Rouwenhorst published a paper on "Facts and Fantasies about Commodity Futures" which helped popularise commodities as a new "asset class" for institutional investors.

Gorton and Rouwenhorst claimed the risk-adjusted returns from an index of commodity futures were similar to equities while also offering useful portfolio diversification and protection from inflation.

"Facts and Fantasies," coupled with the bull market in oil, metals and agricultural products between 2004 and 2008, convinced investors to allocate several hundred billion dollars to commodity index products.

Index investments were especially attractive for conservative investors because they promised inherent long-term returns independent of market timing or the skill of the fund manager.

Commodity indices were attractive in the same way an equity index fund is attractive and were meant to be a different sort of investment from an actively managed portfolio or a hedge fund.

But since 2008, returns on commodity indices have far under-performed equities, creating a crisis of confidence in the idea of commodity indexing and commodities as a distinct asset class.

Frustrated commodity investors have watched as their own investments have fallen in value even as equity investments have rebounded from the financial crisis and continued to perform well.

Many pension funds have finally given up and scaled back or exited from their commodity programs altogether.

Gorton and Rouwenhorst based their paper on an examination of returns from a fully collateralized equally-weighted index of commodity futures between 1959 and 2004.

In practice, it has never been possible to invest in an equally-weighted commodity futures index on any sort of scale because some of the futures contracts are simply too small and illiquid.

Most investors allocated money to an exchange-traded fund or an over-the-counter swap that tracked one of two families of commodity indices, the Standard and Poor's Goldman Sachs Commodity Index (S&P GSCI) or the Bloomberg Commodity Index (BCI).

The S&P GSCI has always had a much higher weighting towards crude oil than the Bloomberg Commodity Index which has more exposure to metals and agricultural contracts. The BCI is closer to the idealised Gorton and Rouwenhorst index though it is still far from being equally weighted.

But the S&P GSCI also offers a family of modified indices with names like Reduced Energy, Light Energy and Ultra Light Energy, with trimmed weightings toward oil and more exposure to other commodities.

None of these indices has performed well in the post-2008 period.

Spot prices for most commodities have fallen, especially since 2014, in many cases reaching their lowest level in over a decade.

Low interest rates have cut the returns on the bonds posted as collateral to back the commodity futures position.

And the contango structure in most commodity futures markets means that long futures positions are being rolled forward at a loss.

The combination of falling spot prices, low interest rates and contango markets has had a disastrous effect on risk-adjusted returns for commodity indices compared with equities.

Under-performance

In 2005, when Gorton and Rouwenhorst were publishing their paper, the long-term returns on a real world commodity index like the S&P GSCI appeared similar to an equity index like the S&P 500.

The equivalent between commodity and equity returns held up until commodity prices peaked and the financial crisis erupted in 2008. Arguably, the equivalence persisted in the early stages of the recovery in 2009 and 2010.

Since 2011, however, equity markets have continued to offer consistent positive returns, while the returns on commodity indices stalled and then turned negative.

The poor performance of commodity indices in the long term has not been significantly affected by the choice of weighting scheme.

The Reduced Energy, Light Energy and Ultra Light Energy variants of the S&P GSCI actually show lower returns since 1989.

In 2015, Gorton and Rouwenhorst together with Geetesh Bhardwaj published an update of the earlier paper "Facts and Fantasies about Commodity Futures Ten Years Later".

— John Kemp is a Reuters market analyst. The views expressed are his own.