The Failure of Networked Systems: The Repercussions of Systematic Risk Revisited

This is an updated story that originally ran in January 2008. David Clarke's warnings about the risks of failure in highly connected systems have proved to be prescient in light of recent events - Big Gav. (David Clarke writes under the pen name aeldric.)

There are those among the Peak Oil community who suspect that we could be facing a failure of our interdependent society that may be sudden, profound, and complete. I have repeatedly said that I am not numbered among them. My opinion is that our way of life will have to change significantly, but slowly. I don’t expect to be clubbing anybody with a femur in any foreseeable future. This opinion is on record in both print and electronic media, and I don’t expect to be issuing a retraction any time soon--but a recent event forced me to admit that I may have to hedge a little.




Network


Our internal network here has been having problems. My email (and more importantly my access to TOD) has been very unreliable over the last two days. The network regularly flicked from "working" to "failed" in the blink of an eye. I was reminded that the speed of collapse in a network is often a function of the natural frequency (speed) of the network, while the breadth of failure depends on a number of factors, including load and the degree of interdependence within the network.

The problem was eventually traced to a problem with one piece of software on one machine on our intranet. The software drivers for the network interface card on one machine were corrupt.

This raised a question in my mind: The Internet Protocol was originally designed to be a robust, reliable, redundant system. How does one piece of software on one machine bring down a network with thousands of nodes?

The answer is easy: Cost efficiencies.

Our Intranet network could have been built to be reliable, but instead it was built to be "efficient". Far from being a network of fail-safe systems, our network is a network of interdependencies. When the system was loaded, a single failure brought the whole system down. "Business Efficiency" has brought our network to its knees for two consecutive days.

I have seen this pattern a lot recently. Last year the power went out in my city. The power transmission system was heavily loaded one afternoon, when a single failure brought the whole system down.

Academics have studied failures of complex systems with interesting results. One of the experiments they did will be familiar to anyone who has ever played with sand-castles as a child. Build a sand pile by gradually adding grains of sand. After a while, avalanches start to run down your pile. Sometimes they are minor, while other times they affect the whole pile. There is seemingly no way to reliably predict the outcome.

However Per Bak, in his book “How Nature Works,” shows that there is an instructive way to look at this question.

There is a critical angle for piles of sand--a level of steepness that the slope cannot go beyond without sand starting to roll down the slope. Imagine that, as you add sand, you colour red all of the areas of the pile that achieve this critical angle (and are thus on the verge of an avalanche). You will notice that the red patches appear as tendrils running down the side of the pile. As you add sand to the pile it gets higher and wider – the pile gets steeper and more little tendrils of red appear. Eventually you will see the tendrils of red start to interconnect.

If you drop a grain of sand on a red area then you will precipitate an avalanche. If the red area is interconnected with other red areas then all these areas will be drawn into the avalanche. If the red area is isolated, then the avalanche will be confined to one red tendril running down the side of the pile.

This basic principal can be applied to my network problem. If one route on the network gets loaded to capacity (i.e. turns red), the system detects that it has reached maximum capacity, and it delays traffic (piles it higher) or switches traffic to other routes (spreads wider).

If the other routes were new, unloaded and redundant parts of the network, then this would not be a problem. But they are not. The other routes are simply other parts of the old, heavily loaded network. Pretty soon all routes are red, and they are all interconnected. So when one part of the network fails, it passes the traffic to another part of the network, which fails and your avalanche starts. With all networks connected, all of them are vulnerable and all fail.

Our network operates at electronic speeds, and it failed with the same rapidity.

Understanding how this happened is critically important. There are four parts to creating the complete meltdown of a network:

1. Create a network by building connections between systems.
2. When a particular part of the network approaches overload (goes red), recognise that this is happening and use the connections you have created to allow you to switch load to another part of the network.
3. Continue doing this until all areas are red.
4. Now add more load.

When we poured sand on our sand pile we allowed the sand to fall randomly, and thus the avalanches seemed random. But once we had the ability to monitor (see our potential “avalanche” areas coloured red), we were able to carefully divert the sand into other areas. This delays the avalanche, but in the long run the avalanche is going to be much worse, because it will occur when all areas are red.

In summary: The ability to measure and monitor the system gives us the capacity to avoid small avalanches in individual areas. However, if we keep adding load without adding capacity we overload the entire network and thus make an all-encompassing avalanche inevitable.

If we can’t add capacity, then it would have been better to allow a series of small avalanches.

A look at the financial markets at the moment might illustrate the same point. When we look at the “sub-prime” issues that are emerging, we see that the market created a series of “Investment Vehicles” that allowed risk to be shared. A complex network of interdependencies was created to share this risk, but capacity was not added to deal with the possibility of default. The various institutions that bought these “Investment Vehicles” thought they were buying assets, not debts. The institutions failed to recognise that they needed to add “capacity” in the form of liquidity equal to the possible value of defaults on this debt. As a result, now that load is being applied (in the form of defaults) it threatens to bring down the entire network, rather than just the single “node” that originated the debt.

[Update, October 2008: In view of what has happened since I wrote this piece in January I should probably mention that, in my view, the natural frequency for the cascading failure of the economic system is quite variable. We have electronically linked systems in some areas, while other areas rely on lawyers and accountants laboriously unwinding CDS and other derivatives by hand. The variability means that contrary to what I was hearing yesterday, this cascade is far from finished.

It is also worth noting that a crash can happen in a fast system, but you may not feel it until it has propagated through a slow system, if these systems exist as part of a chain. For example, credit systems can lock up quite quickly, but you may not feel it until the effects have propagated through transport systems. When credit is unavailable, resellers cannot buy items (such as grain), so it does not go on ships, and does not get delivered--but it will be weeks before you notice the delivery failure. (Baltic Dry is an indicator of shipping rates. As I write this, the Baltic Dry Index is down about 80%. http://www.bloomberg.com/apps/quote?ticker=bdiy&exch=IND&x=15&y=11 --and we are starting to feel the effects of this downturn.) The speed of impact of a cascading failure is often limited by the natural frequency of the slowest link.]

The critical concept is that monitoring and networking the system allows us to go right up to the edge of disaster, and then move load to another part of the network until it, too, is on the edge of disaster.

Now that the networking effects have been discussed, I would like to push the analogy a bit further and look at how this plays out from a Peak Oil perspective.

Several years ago, sweet light crude oil started getting a bit more difficult to obtain. In response, we stopped talking about “oil” and started talking about “liquids”. The word “liquids” covers Liquefied Natural Gas (LNG), ethanol, heavy oils, tar sands, and an increasing number of other oil-substitutes.

Essentially the part of the network called “Sweet Light Crude” turned red, so we started connecting the "Oil Network" to other networks.

We connected oil to the “food” network by turning food into ethanol. Actually food was already connected because you need oil to make food in the modern world, but now the circle is complete-–previously we used oil to create food, and now we use food (corn, sugar, palm oil, etc) to create oil (or oil-substitutes).

Adding LNG and CTL (Coal-To-Liquid) to the network connects oil to other energy sources. As this connection strengthens and load starts to be applied, a shortage of any of these sources would have an impact in each of the other sources. To some extent, this has already started to occur.

Adding tar sands and various other oil substitutes to the network has made a surprising connection between the environment and oil. This connection takes many forms, but the most interesting lies in the fact that oil substitutes are less efficient than light sweet crude-–much more CO2 is produced for any given amount of work done. This connection is emerging, and could have interesting repercussions. The problem applies to virtually all the oil-substitutes, so the widespread adoption of substitutes (particularly CTL and tar sands) might cause an environmental disaster which in turn would suppress ethanol production and create knock-on effects in other parts of the network.

The financial system has an important role to play in this network. If energy, food and the environment can be considered three portions of the network, then our financial system can be considered to be both a form of network monitoring, and the communication medium that the network uses to pass signals around. Consider the financial system to be similar to the blue cable running out the back of your computer. Your computer’s blue cable isn’t likely to run hot, but our finance system is a network of networks, and it is glowing red. In addition to monitoring and communication, the financial system provides support for maintenance and upgrades of the energy systems, so capacity in the financial system is critical.

When one part of the network develops a problem (say production of LNG suddenly drops), then messages get sent via the financial system (in the form of increased prices), and the other parts of the system accept the load, if they can, by increasing production. When compared to an Internet Protocol network there are many faults in this system. High latency leads to slow responses. Poor monitoring leads to conflicting signals or a failure to detect faults. Bad messages are often not corrected, leading to incorrect responses, and so on.

The speed of a crash

The interesting point to note is that increasing demand past capacity will not immediately “crash” this system. Oil facilities that are working at capacity will not “crash” if demand exceeds the capacity, they will simply continue working at capacity. The crash may come, but it will come because demand heats up the financial system and crashes other systems that depend on finances. Since the oil production system is dependent on other systems, this could conceivably cause an eventual crash. Eventually lack of maintenance will degrade the capacity, but this is a process that occurs over a period of months or years.

Likewise, the process of adding capacity is exceptionally slow. Building CTL or NGL plants takes the best part of a decade.

The oil production system can certainly crash, but it would be a crash in slow motion.

The only part of the system that can crash quickly is the financial system. The financial system provides monitoring, communication, maintenance and upgrades. So a profound, complete crash in this area could conceivably bring down the whole network.

However, could such a financial crash occur? An immediate halt to oil production would require a crash far more profound than the Great Depression. The response speed of our financial system has been improved by linking many of the sub-systems electronically, but there are still a number of choke points, circuit breakers, and sanity checks. The Great Depression emerged over a period of months or years. Even with the electronic linkages in place today, a complete breakdown of our financial institutions is unlikely to happen overnight.

If this system crashes overnight, it will be because the plug got pulled-–a breakdown of society external to the system.

The natural frequency for events in the oil and oil-substitute network is in the range of months at least, or more likely years. Internal stresses cannot cause it to crash overnight.

The Breadth of a crash

The breadth of the crash depends on the degree of linkage and the degree to which each part of the network is loaded. This is where I start to worry.

Oil appears to be at or near peak capacity--exports are dropping. As for the food network--world grain reserves are at historic lows, and expected to drop a little more next year. And the environment? Climate change is clearly with us, indicating that the environment has already gone past its capacity.

When looked at in these terms it appears that the network is already in decline. Each of these three parts of the network is at or past capacity. If a span of years is the natural time-frame for a crash in this system, then it seems quite plausible that we are watching a very broad-based crash of our energy systems--right now.

Our actions in increasing the connections to the food and environment networks will not help, and may simply speed the crash.

The signals indicating the start of a crash would be seen in the monitoring and communication system–-the financial systems. Prices for oil would go up. Which we have seen.... Prices for food would go up. Which we have seen.... We might expect perturbations, volatility, and attempts to “price” the environment.... Hmmmm.

Conclusion

I am forced to concede that a broad-based collapse is a possibility. I still maintain that a sudden collapse is unlikely, but if it is already happening, then it could certainly look sudden when we eventually notice it.

[Update, October 2008: I am still hoping to avoid a sudden, broad-based collapse. Some factors look like they will contribute, while others will mitigate. In many cases, the pace cannot proceed faster than the slowest system in the dependancy chain. In monitoring this situation, look for dependancy connections between systems, and then ask yourself what the natural frequency of the slowest system in the chain is.]

aeldric.

Thank you for this article. What about external effects such as the posted Atlantic article on an Iran facility strike by Israel? A crash could then be sudden and cascading due to the failure of one oil transport choke point.

Political, but very linked in the system.

Cheers...Paul

It seems like there are an awfully lot of different external effects that could have a major impact:

  • Political changes, particularly in Saudi Arabia, but also in other major oil exporters, or in Iran, as you point out.
  • Weather-related changes, like the fires in Russia, or Hurricane Katrina in 2005.
  • Bankruptcies of one or more major countries -- Greece, Italy, Portugal, Britain.
  • A major accident like the Deepwater Horizon blowout, or the Chernobyl accident.
  • Problem with our current system of financial derivatives.
  • Bankruptcies of several major electricity providers.
  • Major tax increase -> less disposable income -> more recession -> lower oil prices -> less oil production.
  • End of economic growth.

The Department of Homeland Security lists 18 Critical Infrastructure and Key Resources Sectors, although I'd think the rationale for including "National Monuments and Icons" is pretty weak.

Although DHS is primarily concerned about attacks, about half of these would also be the sectors vulnerable to collapse due to their network nature.

This is a neat book on system vulnerabilities to natural disaster and sabotage.
http://books.google.com/booksid=_1DEpZmS8wUC&pg=PA55&lpg=PA55&dq=vulnera...

It is interesting to note that most very large step up transformers are not duplicated and are special ordered when needed, usually with about a one year lead time. Think about that when one 1000 megawatt transformer is taken out for one reason or another. That facility is going to be down for a long time.

Treeman: your google link yields only an error msg.

:::Tens of millions of people just getting by on crummy paying service jobs losing them or being unable to make ends meet any longer;such people buy quantities of services all out of proportrion to thier incomes, such as medical procedures and legal representation.

A lot of dentists are going to be going short of patients unless they see fit to cut thier rates-the feedback effects will be overwhelming, considering the public expenditures necessary top support the unemployeded and the loss of thier purchases from service providers.

Hairdressers and barbers may still be able to make a living, but if so they will be earning very little per customer.

Exponentially growing systems such as the economy are 'inherently unstable and prone to catostrophic collapses over time'(Meadows). Economic growth wont just end in a steady state without some serious socio-economic adjustments, but decline exponetially in the same way it grew. This decline will not be evenly spread throughout the social heirachy. The wealthy will remaining largely unaffected while those lower down will take the brunt of it. This doesn't bode well for half the worlds population who live on less than 2.5 dollars per day.

Exponentially growing systems such as the economy are 'inherently unstable and prone to catastrophic collapses over time'(Meadows)

China's exponential growth springs to mind.
Australia's economy is doing fine, but it is a one trick pony. If China does an exponential crash then Australia will go belly up.
We have no resiliency at all.

When Spain discovered Inca Gold it was the economic powerhouse of Europe. From then on Spain could import everything. This was the death of Spanish industry. Therefore commodities are poison to the real economy.
And Australia's economy depends on commodities.

I hear China's economic engine miss firing.
They are letting the Yuan free fall faster than the American dollar in order to encourage a return to the old paradigm of America as the client state buying China's produce.
(From www.x-rates)

I gird my loins.

A Great, Bounding Dog
Heavy Power In Free Flight;
chain is shortening

Hi Aeldric, nice to hear from you again.

I too am in the slow collapse camp, though speed is really a relative concept. The collapse we are witnessing now is happening "slowly" in a world that operates as fast as ours does today, with its expectations of rapid satisfaction of everything. However, if collapse occurs over 10, 20 or 30 years that is indeed fairly rapid. My kids already expect to live different lives to that which we live now.

In a more detailed sense I have been thinking about resilience. How resilient is our world? Really? We are so used to BAU that it hard to contemplate collapse, but our whole society has been geared up for maximum "efficiency". Corporations are deliberately profitable, but only just. Their balance sheets are often a cash flow knife edge. Manufacturing and retailing operates on a fantastic connected logistics support network. The UK 2000 petrol truck strike showed just how quickly the shelves empty if there is as logistics glitch.

Nevertheless us humans can adapt very quickly, find work-arounds and make do for most of our needs if the problem was merely physical. I think the problem may be financial. How resilient are the capital markets? Could it be that it is our financial systems that will let us down? Maybe every time it seems like it is all going pear shaped the government will simply throw some of own money back at us. Although I have a Masters degree in Sustainability, am a CA and have raised over $1bn in a variety of deals in the capital markets, I have absolutely no idea where we are headed. We are starting to grow food in our garden and we are installing solar power and water storage. And so are lots of other people.

Interesting post, and one I'll certainly have to give a lot of thought. How does one go about quantifying the rates at which linkage and load are changing in the oil/substitutes network?

Regarding post-Peak Oil and systemic collapse - This is why I have advocated creating a parallel Oil-Free Transportation system to our existing oil based one.

A system with a high Elasticity of Supply in an emergency. Once bicycling become a viable alternative, it's modal share can increase rapidly if oil based transportation is severely stressed. And I have a hard time conceiving how bicycling could collapse.

A quote from a forthcoming article of mine

The more society uses roads, the more expensive and slower they become. The marginal cost for extra capacity is higher, usually far higher, than the average historical cost.

The opposite is true for rail. Extra capacity is usually significantly cheaper than the base costs. The more “we” use rail, with appropriate infrastructure investments, the cheaper and faster transportation by rail becomes.

Roads are free access and create their own demand over time, so more roads is not the answer. Trains are scheduled and routed and their demand can be managed with creative dispatching, adequate track capacity and state-of-the-art signaling. So more use of rail is the answer.

I also advocate "forward leaning" investments in rail track & electrification and a Strategic Railcar Reserve (to give extra capacity in a hurry).

If a system is at risk of failure, then creating an alternative parallel one is a very viable strategy.

Best Hopes,

Alan

I cannot think of other systems (besides rail) that become faster and cheaper the more that they are used. Where the marginal cost of extra capacity is substantially lower than the base cost/unit and expanded use (with appropriate investment) also increases average speed.

And rail systems are difficult to collapse (as Allied bombing of Germany & Japan found out).

One key feature is that intranets, highways and financial systems are "free access" while rail is scheduled and managed access. Dispatch just never drops that red sand particle on a rail line that causes complete collapse.

And rail systems are difficult to collapse (as Allied bombing of Germany & Japan found out).

No more. At that time 1,000s of more or less independently running steam locos worked the trains. Today, it's mostly electrified, depending on centralized power supplies. High speed rail is even more vulnerable.

Germany and Switzerland were the leaders in European electrification. In 1911, the German State Railways standardized on a system of 15 kilovolts at 16 2/3 Hz.

http://www.lafn.org/~dave/trans/rail/electric_rr.html#ss2.5

Alan

Being somewhat of a bizarre hybrid in the world of electrical engineering, (or not so much now and maybe just a little ahead of my time), the potentials for network failures and collapse are well understood in electrical grid systems and telecommunications. Those in the enterprise network and service provider networks have come to understand these principles the hard way.

It really boils down to a clear distinction between deterministic systems and non-deterministic systems. The analog electrical grid has the potential to be completely non-deterministic but is operated to be deterministic, and that is typically the root of failures. The electrical grid to date is set up to operate within maximum limits for equipment and protection and control. Operators and dispatchers monitor and control the energy flows by various techniques. They are the first level "circuit breaker" but prone to the frailties of humanity. The equipment protection and control is set to trip when nominal equipment capacities are exceeded, and the system is typically designed to handle a single contingency during maximum conditions. And this has become an increasing mode of failure as the bean counters and biz kids keep squeezing more out of the contingency without understanding the engineering. Engineers are too passive aggressive I guess.

Ethernet is non-deterministic, but current network infrastructure tries to make it deterministic by the use of switches. Ethernet is a random common mode technology never intended for the scale of operation deployed today. The answer to overcome the shortcomings or misuse of Ethernet is to either throw more bandwidth at it or get creative with higher layer protocols, i.e. MPLS. You might start to see some parallels here between the technology world and oil. Eventually both are going to run up against physical limits.

Old school telecommunications was set up to be deterministic through the use of channelization. This led to a degree of inefficiency, but it was stable and predictable. The main broadband technology deployed in large service provider networks integrating voice and data services primarily was/is ATM (Asynchronous Transfer Mode). ATM blends the stability of channelization with the efficiencies of dispersed randomness, or over-subscription. ATM's widespread "failure to launch" was due to the complexity and challenge to the "wetware", and the high cost of end unit terminations. My concern for the changes to the electrical grid with Smart Grid technologies is they are going down the same road depending on Ethernet. We have technology at hand to dynamically change the protection settings on the relays in substations to reflect near real-time conditions. But are we just asking for more trouble by trying to operate systems beyond their structural and entropic limitations.

The parallels are numerous and well understood. Randomness and common access works well for efficiencies at the very local level whether it be computer and telecom networks, railways or roadways. Deterministic systems work well for bulk interconnection and pathways. Our networked systems are prone (designed?) to fail where we confuse or ignore these simple natural physical laws. e.g. our roadways grew from local random systems into large scale bulk random systems and will fail. Where there is random common access, Boyles Law tends to prevail, or Nature abhors a vacuum.

Finally, taking it back to the human nature element, we resist the regimentation of deterministic systems as we feel our freedoms are curtailed. This is true to some degree, but how much freedom does one have stuck in traffic all the time?

Side note: I am going to start promoting the development and use of "ATM-Lite" in IEC 61850 for substation communications between IED's (Intelligent Electronic Devices -we had it before the military, but per usual no one bothered to check).

For Fibre Channel over Ethernet, IEEE have added features to Ethernet to support lossless deterministic delivery. See Data Center Bridging (DCB) Task Group. An Introduction to Fibre Channel over Ethernet, and Fibre Channel over Convergence Enhanced Ethernet describes the rationale for the ethernet changes.

That's the problem, they keep trying to modify Ethernet until it resembles the other technologies it is supposed to be easier, cheaper, and more effective to replace. Fibre Channel over Ethernet sounds a lot like FDDI.

Do you think you will succeed given that the Eth cost curve trumps ATM? Why not focus on the system level design rather than the low-level protocols?

ATM only gets expensive when they try to do everything with it as it was intended. Strip down ATM to the bare functionality and concentrate on what it does best which is fabric switching in a deterministic and controlled fashion and it will be as cheap as Ethernet.

Use an ADSL modem? That's ATM-Lite. The modem uses ATM to the DSLAM. DSLAM typically uses ATM to the router interface and then it's whatever from that point. That's how they shape your bandwidth packages (if you still have that in your neck of the woods).

On the other hand, try doing a channelized T1 over ATM and that is when it is expensive - and complex, and annoying - generally a PITA.

I thought ATM was being phased out in favor of MPLS ot 'tag' switching. The SONET takes care of the lower network layers.

The nice thing about standards is that you have so many to choose from!

You just proved my point. MPLS stand for Multi-Protocol layer switching. It likes ALL protocols without the cell 'tax' typically associated with ATM. What is ATM anyhow? Asynchronous Transfer Mode. Send it and you do not care if it gets there. It is up to higher layers. MPLS is an evolution of such protocols. That is why the technology is considered layer 2.5. It really outgrows the traditional OSI network design.

http://en.wikipedia.org/wiki/Multiprotocol_Label_Switching

the potentials for network failures and collapse are well understood in electrical grid systems and telecommunications. Those in the enterprise network and service provider networks have come to understand these principles the hard way.

Umm... Would you mind defining *WELL*?

By coincidence I was just browsing through the syllabus for MIT OpenCourseWare
6.01 Introduction to Electrical Engineering and Computer Science I
http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6...

And came across this quote from Donald Knuth...

Donald E. Knuth is a computer scientist who is famous for, among other things, his series of textbooks (as well as for TEX, the typesetting system we use to make all of our handouts), and a variety of other contributions to theoretical computer science.
“It is perhaps significant to note that although the author had used the elevator system for years and thought he knew it well, it wasn’t until he attempted to write this section that he realized there were quite a few facts about the elevator’s system of choosing directions that he did not know. He went back to experiment with the elevator six separate times, each time believing he had finally achieved a complete understanding of its modus operandi. (Now he is reluctant to ride it for fear some new facet of its operation will appear, contradicting the algorithms given.) We often fail to realize how little we know about a thing until we attempt to simulate it on a computer.”
The Art of Computer Programming, Donald E., Knuth, Vol 1. page 295. On the elevator system in the Mathematics Building at Cal Tech. First published in 1968

One key feature is that intranets, highways and financial systems are "free access" while rail is scheduled and managed access. Dispatch just never drops that red sand particle on a rail line that causes complete collapse.

I would add control of internal routing and time to react to the list of strengths in a rail network. The electric grid has broad dispatch controls at the points where generators and load centers are attached, but much less control over the internal flows, and the speed at which things happen is much faster. Failure of a transmission link or a generator leads to almost-instant transfer of load among the remaining links in the network, and the possibilities of cascading failures.

While there were a number of factors involved in getting things started, the broad extent of the 2003 Northeast Blackout was largely the result of cascading failures, where a local action that took a power plant or load center off the grid in order to protect the local facility created conditions elsewhere in the grid that caused other local facilities to do the same thing. And quickly -- once the failures extended past Ohio, everything else happened in less than three minutes.

Speaking from personal experience, designing distributed control systems that isolate faults in ways that keep the overall network up and running while using only local information is really hard. Even IP, the most commonly used example of robust distributed control, requires that neighboring routers exchange information regularly.

For ATM, Ethernet, grids, stock markets, and trains, the risks of emergent complexity yielding novel fault behaviors is ingrained in the control systems if not the 'transport' technology itself.

Trains have an advantage in time-constants, and humans could do a decent if sub-optimal job in scheduling transits. The problem with non-deterministic systems is the degree of non-determinism. Take the stock-market systems for example -- a robust if inefficient process of trading with scraps of paper was enhanced by automated tickers, then assisted trading, then automated trading, and now complex front-running and sophisticated arbitrage algorithms. It should be no surprise that 'flash crash' arises as an emergent facet of program trading, nor that derivatives create vast but unspecified risks in quant trading.

At the core of instability there will still be feedback loops with gains and phase shifts, and poles in the wrong half-plane, but how to model those and predict behaviors is likely impossible.

The precautionary principle would say "keep important things simple", but we have blown past that in the pursuit of efficiency and advantage.

Still, for things that are important, once the complex solution fails an acceptable if suboptimal solution should arise, as long as the failures don't completely break the superior-level systems....which might be society itself.

@ McCain6925 I would add control of internal routing and time to react to the list of strengths in a rail network.

We had an example on the Tube last Friday. An unmanned Engineering train being towed northward on the Northern Line got uncoupled and started running back South along the line. They were able to clear the line ahead of it and finally switch into a siding a few stations further down. Cue partial collapse and frustrated commuters that I mentioned earlier, but at least no-one got killed.

Funnily enough, this came a few weeks after Transport for London talked up introducing driverless trains on certain lines and got the unions talking about industrial action (again, not uncommon) over safety concerns.

Funnily enough, this came a few weeks after Transport for London talked up introducing driverless trains on certain lines and got the unions talking about industrial action (again, not uncommon) over safety concerns.

The Bay Area Rapid Transit (BART) system in California was originally intended to be largely automated, but was at least initially plagued by trains that went missing (the system could no longer detect their presence) and by ghost trains (sensors indicating a train where there was none). I understand that there are still some ghost train problems, some 40 years after operation began.

Denver International Airport was originally built with an automated baggage transport system: bags were removed from planes, put in small electric rail cars, which transported them to the terminal and automatically dumped them at the proper baggage carousel. The system was computer controlled, but without any feedback systems. Once one car jammed or the track was otherwise blocked, successive cars just kept piling up at the trouble point. The system was eventually abandoned.

In a former life I did some high-reliability real-time software specification. I suspect that it is possible to design and implement safe driverless trains. But I'd never win the bid for the system, because my approach to it would be too expensive.

All complex systems have emergent features. Those we design up-front provide the impetus to build the system in the first place. It's the unanticipated ones that in small doses give systems "a personality" and in large doses create disasters.

@ AlanfromBigEasy One key feature is that intranets, highways and financial systems are "free access" while rail is scheduled and managed access. Dispatch just never drops that red sand particle on a rail line that causes complete collapse.

I agree with everything you say but even Rail has a maximum capacity. Here in London (and I should imagine in a lot of other cities), the Underground system works flat out delivering commuters during Weekday rush-hour periods. One train per minute is the norm at many stations, and it's still not uncommon to have to wait for the 4th or 5th train before you can get on. In these circumstances it only takes one defective train for a partial collapse and a lot of very frustrated people.

Having said that, I'd like to see a lot more investment in the UK's rail network (and some effective price controls so that people are encouraged and able to use it)

The risk of interconnected systems is critically interesting IMHO. It's a good reason not to mountain climb roped together with three other climbers unless absolutely necessary. You would place your best climber in the lead with the climbers at thirty five foot intervals. If he falls on a steep slope you can't stop him unless extraordinarily lucky. He falls seventy feet before the second climber could possibly begin to stop him.

http://www.traditionalmountaineering.org/Report_Hood_Bergschrund.htm

In my mind this is how our interconnected critical systems are arrayed. Leading "the rope" on a slippery slope is either the financial markets or the electric grid. If either falls the momentum builds and is likely to drag everything dependent below them to the abyss with them... or not. It certainly could lead to a cascading series of failures though.

Obviously, our interconnected systems need to be robust enough to "self-arrest", to stop the fall, and if not, the next in the system strong enough to stop the fall. For example, if the financial system collapses and credit for business dries up very quickly, what can stop the fall? Can we simply print money to reestablish the money supply? If the electric grid fails due to a CME what options are there to stop a cascading system failure?

It seems to me a system wide failure can build very quickly like the sand pile when overloaded. The failure will continue until an equilibrium is attained. The question is, how close are we to failure? How high can the pile of .... stuff, be stacked?

It seems to me that we have a lot of plans we have for the future are based on the assumption that what will happen is oil prices will rise, and oil will become a bit scarcer, but basically the whole system will stay together. Plans for more efficient cars, for example, are built on such a system. Even organic gardening seems to have a lot of BAU assumptions built in --trucks to take produce to market, and to transport soil amendments to a new location; electric fences; and irrigation using a pump, for example.

But we need to be making fairly different plans, if the real issue is the system not holding together, IMO.

Also, I would agree with you that the financial markets and the electric grid are way up there in terms of risk factors.

CME???

Coronal mass ejection.

Completely Moronic Event?

Craig

double posted - don't know why.

It's not just systems of hardware and software, I believe. I have been involved with project management in a number of matrix organisations doing fairly high-tech stuff. When one project became a problem, resources would be taken from others to assist the broken project. The result was inevitable, all the projects sharing resources became critical then started to fail. I became convinced that a sick project must be isolated from the others, just as a sick patient is isolated to avoid infection. If a sick project needs more effort, that effort has to be sought from outside the business. This approach does not go down well with those who have built matrix organisations in the expectation that they are adaptable and can achieve more with less. I was party to just such a re-structuring in a large corporation, and the results were dramatic. The projects that were in good shape surged ahead, and the projects that were sick were isolated and mended or closed down.
It seems to me also that the same applies to economic and political systems - if there is a problem, don't share it, isolate it. Pull anything genuinely healthy out of it before it gets infected. Finally, facing the fundamental 'nasty' at the root of the problem, quarantine it and deal with it with focus and thoroughness using only those resources from within that can truly be spared, and outside help otherwise. (Obviously, it is of long-term benefit to the organisation that the learning from the fixing is passed back into the organisation, so the demobilisation needs to include structured knowledge capture and dissemination. Beginning to sound like a consultant, so I'll stop now!)

But what if the "sick" areas are the financial system, the electrical system, and the oil based energy system.. We can't just isolate and ignore them--they are too important.

Gillian Tett writing in the Financial Times recently, making a point about confidence, concluded her piece:

And just to make matters worse, the memory of the May 6 “flash crash” haunts the markets. In the past three months, US regulators and bankers have scurried around trying to work out what caused equity prices to gyrate so wildly that day. But, thus far, they have not offered any convincing explanation.

http://www.ft.com/cms/s/0/5ac2207e-a0ae-11df-badd-00144feabdc0.html

At one point the Royal Bank of Scotland (a large bank in the UK) was 2 hours from collapse. That would have had a very large
cascade effect across other banks.

http://news.bbc.co.uk/today/hi/today/newsid_8914000/8914062.stm

It has been confirmed that the large dent that recently appeared in a full oil tanker in the Straits of Homuz was a nearly successful terrorist attack.

If it had succeeded, what would that have done to the financial markets?

How close ARE we to collapse?

In energy terms, not that close.
The Persian Gulf oil provides 20% of all oil production, 15 mbpd out of 75 mbpd.
All OPEC oil amounts to 29 mbpd or 38%.
During WW2 in the USA rationing domestic consumption of gasoline dropped by 30%.

But the profit-mad capital markets run by adrenelin-hyped 'bankers' that are the least reliable links in the chain of our civilization.

The rationale that we need huge banks to do huge investments isn't that born out by their behavior(they don't lend).
The banks probably should be nationalized.

I bank with the RBS and had not heard that interview or it mentioned at all so I listened with some fascination. Sadly they seem to have learned the wrong lesson from their near miss, and are getting needlessly tough with their good customers who were in no way connected with their huge losses from bad property investments.

That said, I bet they will not get so near to collapse again in my lifetime.

"I bet they will not get so near to collapse again in my lifetime."

Well, ok then, your bet's on the table. (I'm not taking it.. just noticing that you put a prediction about the financial system in terms of betting. There are lots of other bets out there that are holding the financial system together..)

As far as 'In my lifetime' goes.. it reminds me of the interview with an old Mainer..

"You lived here all your life?"
"Not yet."

(EDIT- Yikes, that came across as snarky.. just trying to say 'From your lips to God's ears', brother, you might be right..)

Ok I have a spin on the classic failure problem. Remember good old Murphy? The real one?

In 1949, the US Air Force was running a test series (Project MX981) to see what sort of acceleration (or G-forces) a human being could withstand. They were using a volunteer strapped into a rocket sled at what is now called Edward's Air Force Base, in California. The sled would accelerate up to about 1,000 kilometres per hour, and then stop suddenly. One of the volunteer human torpedoes was Colonel Stapp, who was also a medical doctor.

Now our hero, Air Force Captain Edward A. Murphy Jr., had designed a harness which strapped onto the volunteer. This harness held 16 sensors to measure the acceleration, or the G-forces, on different parts of the poor volunteer. As luck would have it, there were two ways that each sensor could be installed.

The rocket sled took off, and stopped suddenly, generating 40 Gs. Under 1 G, the average person weighs about 70 kg, but under 40 Gs, they weigh 40 times more - about 2.8 tonnes. 40 Gs is an enormous amount of acceleration - enough to push your ears onto the front of your head.

The rest of the widely known story is that there was no reading because ALL 16 sensors had been installed improperly. So let us assume for research that the chance of failure due to orientation issues was approximately 50/50. I am sure an analysis would yield a greater than 50% failure rate, this is just a hypothetical. 1/(2^16) or a 1 in 65536 chance that the entire sensor array would fail due to orientation, ceterus paribus. Does this simple example compute?

From: http://www.abc.net.au/science/k2/moments/gmis9906.htm

TFHG: "... 40 Gs is an enormous amount of acceleration - enough to push your ears onto the front of your head."

Cool! But uh, which direction was he accelerating? I'd prefer accelerating forward... which would pin your ears to the back of your head and put your eyeballs where your ears were? :) Now that's what I'm talking about!

http://www.youtube.com/watch?v=r3IxSjuMUCE&feature=related

Good observation. All the photos I have ever seen for rocket sled tests had the subjects facing forward. I was having a harder time finding the number of sensors in the harness. I wonder if anyone here @ TOD ever worked with those tests or similar human capacity stuff for pilots or astronauts.

The one in sixty thousand plus odds almost certainly are irrevelant-the odds are thousands to one that the hands on technician made ONE PRIMARY MISTAKE-he misinterpreted the directions or drawings or labels on the sensors, and therefore installed all of them , slowly and deliberately, backwards.

Of course the failure could have been on the part of an engineer, a draftsman, a technical writer, or an English teacher and our evolving language.

There are numerous expressions in our language which are ambigious, or conventional practices which are inconsistent from one field to another.

I refer the interested reader to the field of industrial psychology.

My very old Daddy simply cannot REMEMBER that turning the regulator handles on my oxyacetylene equipment to clockwise lock is "wide open" instead of tight shut-not after turning faucets and valves to right lock to shut of everything else he ever worked with for three quarters of century-running water and plumbing arrived late in his childhood ;)

I think one has to consider mass psychology in these considerations, which is almot never done.

I can't imagine that the panic and chaos that results from the first evidence of systemic collapse "in progress" would be beneficial to arrest or mitigate the process. Certainly attempts at imposition of martial law will not be helpful in restoring "business confidence". Food riots tend not to facilitate restoration of municipal services. And so on.

There some very weak points between the current suburban "life support" infrastructure and resulting levels on "non-livability" psychological response. Sewer and water service and the electrical grid are two primary pressure points that could bring a very bad outcome, very quickly, especially in the winter (no heat without electrical grid, even if you have a full tank of oil). Food distribution, of course, for obvious reaons, and very quickly. Prevalence of guns and the the wide spread anger at "others" that is likely to preceed the onset of collapse is very troubling for many areas of the country (red states mostly, but no area is really immune).

Sorry to be soo gloomy, but most native born Americans just don't tend to think in these terms - outside of their frame of reference.

Good point Dimitry especially given how quickly info (both correct and grossly in error) can disseminate thru the web. You remind me of the "War of the World" hoax back in the 30's. Imagine if the net existed then it wasn't just broadcast to some parts of the US but spread around the globe in just minutes. Consider just the affects on the stock markets. Beyond that one's imagination could take them almost anywhere. Granted it works both ways: the truth would get out quickly also. But what about that ole cascading affect folks worry about: how quickly can one unring a bell?

@ Rockman You remind me of the "War of the World" hoax back in the 30's. Imagine if the net existed then it wasn't just broadcast to some parts of the US but spread around the globe in just minutes. Consider just the affects on the stock markets. Beyond that one's imagination could take them almost anywhere. Granted it works both ways: the truth would get out quickly also.

Does anyone know the saying, "A lie can run round the world while the truth is still getting it's shoes on" ? As an example of how much damage a false alarm can cause is the failure in 2008 of Northern Rock bank in the UK after a rush to take money out caused by speculation that it was in trouble.

I belong to the fast colapse camp. And there are one reason for this. A crucial part of the network that you did not even mention: The grid.

Back in 2003 a large chunk of NE USA and into Canada had a severe and soon to be legendary blackout. Less known is that there was one the same summer in Italy, and we had one in Sweden too. Both Stockholm (Swedens capital) and Copenhagen (capital of Denmark) were out of power as well as all the land between there for a good while. Aparently it started with a failure in a larger node near one of our nuclear power plants, and was propelled with the speed of electricity in copper throughout the network. I still remeber replacing the light bulb at the toilet room and then going to check the fuse, then going outdoors to check if the neighbours had power.

I like to think about our energy suplies as circuits. And there are two main circuits; the oli and electric circuits. They are intergrated, so no one could stand a longer time without the other. It occours to me that if we lost both the oil and electric circuits at the same time, they will stay down.

Now imagine a world where every part of every system is runing on low supply but high load. Imagine maintainance beeing neglected for a long while with lots of components running well into their retirement age. Some smaller parts of the system is temporarily out, some for long time. Prices are high, and more and more costumers cancel payments. And every month we have less oil, gas and coal than last month. Then one day at peak load, some critical component fails. We have an elecric failure that cover a whole continent for weeks. We close oil systems after a while. And it spreads through the economic systems to other parts of the world. Weeks or months later, only atstronomers are happy about the global permanent blackout.

Possible scenario? What do you guys think?

The grid is the key, I agree. Most nothing in our "developed world" works without a ready source of dependable AC. The other connected systems will stop functioning as well - food distribution, heating, water, sewer, communications, etc.

It can get very bad very quickly. Even in quick temporary blackouts people start behaving very badly very fast.

I know when Atlanta stopped getting gasoline/diesel after Hurricane Katrina, the problem was an electrical outage along the oil pipeline. The solution was to bring in an electrical generator, and use it to get the electricity for the oil pipeline going again. The dependence of oil pipelines on is one reason electricity is essential for oil usage.

Another interconnection between oil and electricity is pumps at gas stations are electrically powered.

Another way an oil shortage can cause an electricity shortage: Workers don't get to work, and transmission lines don't get repaired, if there is no oil to operate trucks, and to transport new parts long distances.

Another oil-electric interconnection: Diesel is used to operate trains to transport coal long distances to produce electricity. If there is a diesel shortage, it is possible not enough coal will be transported, leading to electrical outages.

It also works the other way--refineries are big users of electricity. If the electricity is not available, you don't get gasoline and diesel from crude oil.

Here's a string of multiple bottlenecks for you, whether you're at a store, a government office, talking to the phone company or in an emergency room, how many times have you heard ..

'The Computer's Down, I can't do anything..'

While pretty much the same as the keypost, still, think of all the basic systems out there that don't have a way to function without the combined availability of -

Grid Power
Internet/Phone Connection
Working Computer/Network
Paper in the Receipt Printer
A parallel system for working without the PC-crutch

I was at a Dunkin' Donuts where they were scribbling people's CreditCard numbers onto old receipts and scrap paper, to process our sales later, when the link was working again.

For many of these things, there might actually be fairly simple and cheap workarounds.. but the prep was never considered.. like keeping a Paper Receipt Book under the CashRegister someplace (and making sure the drawer could open without power.. and then teaching the sales crew how to work the shop in 'full manual mode'

.. I had a friend trying to get help from his computer support number to get his computer online (12 years back or so..), and they had no response even then that didn't rely upon the phrase 'Go to our Website'

Diversify, diversify, diversify.

I believe there was a sudden surge in number of births, 9 months after the blackout. Badly, indeed.

Craig

See North Korea, Cuba after the collapse of the Soviet Union, Albania during their half century of isolation, much of Africa today, etc.

I disagree. Sure the grid will go down for a day or so, but not forever. 99.9+% reliability will be a memory.

Alan

As one welder to another, I think you are one very capable thinker- the scenario you outline is altogether realistic-what the odds are is anybody's guess.

What do Jedi's weld, or what gets welded on Jedis? ;)

Oh thanks. You flatter me...

I am a welder who got unemplyed, went back to school to upgrade myself to pipe welder (Sstick and TIG), done with that now, looking for work. Welding is my day job.

Our Intranet network could have been built to be reliable, but instead it was built to be "efficient"

This is the crux of the matter for a collapse. Redundancy, which is needed to make systems reliable, is expensive and therefore often omitted in system design. One day this will be bitterly regretted.

A crash happens when the system contains internal negative feed back loops which are triggered by certain events or combination of events and which tend to shut down the system.

In Australia, we do not even have a Strategic Oil Reserve. The Financial Review published my letter but no action has been taken so far because no one is taking this seriously. And it costs money.

Details are here:

http://www.crudeoilpeak.com/?p=1541

As happened in previous oil crises, one day an oil tanker may not arrive as scheduled and we are going to have disruptions. It may not be a full crash but there will be financial losses which in turn impact on the ability to prepare for declining oil production

In a technical sense, amplifiers, linear negative feedback increases stability (when used over less that three stages of amplification).

You could use the phrase "bad feedback".

However, technically, the term is probably "non-linear" feedback, as used in Chaos Theory, which comes complete with "strange attractors" and "tipping points".

It's a human fallacy and a wish for system that change slowly. Controlled change require great care, and come with many side effects, however this ignore the effect of large events.

Mother nature is good at delivering large events. So are us humans, the 1973 war and the first oil shock come to mind. Financial bubbles also pop quickly.

I believe there will be a change, and it will be quick and the "system" may reach a new equilibrium, It may be unstable fro some time before and after the change, also called "noise", or stability seeking.

We also need to define the terms "quick change" and slow change". I offer, without any basis, One Year for quick change, and One Generation for slow change.

With this definition, how many believe change will be slow?

One Year may be an appropriate definition for "quick change" but I'd dispute One Generation for slow. As a rough guess, the development of an entirely new transportation system takes a generation, so change on that timescale might be called "usual change" or "normal change" rather than quick or slow. A slow change would then span multiple generations, eh?

Or perhaps you've chosen One Generation as an appropriate timescale for approximating the limits of adaptability?

It is difficult for individuals to internalize changes at scales longer than a generation, and certainly if longer than a lifetime. You hear people say "I've never seen it like this" or "granddad says it's never been this dry" and it means something. You see "personal income normalized for 2010 dollars peaked in 1968" or whatever, and it means very little to you, as you've gotten raises and promotions over the decades, and your kids will be out of college in 3 years.

Sometimes over a beer you'll think "sure my folks expected less, but it was LOT easier for Dad to make good than for me", but that's probably as close as most will come to discerning a slide. And even then, more has changed than has decayed, so for personal reality it's hard even to discern that.

I think "long slow slide, punctuated with a few stair-steps -- some up, mostly down".

Humanity never learns. Thats what makes this so much fun.
you should reread the section on Rome in Tainter's "Collapse of Complex Societies"

My opinion is that our way of life will have to change significantly, but slowly. I don’t expect to be clubbing anybody with a femur in any foreseeable future.

Neither do I.

Because unless we have already suffered complete societal collapse and there are thousands of rotting corpses lying about in the streets, then it's probably going to be a heck of a lot easier to find, a much more efficient club, in the form of a rusty old crowbar somewhere, than it will be to find a readily available femur... Plus if it should come down to it, my crowbar will pulverize your femur.

I agree that going back to stone or bone tools won't likely happen unless all of the detritus of industrialized civilization were to suddenly vanish overnight, and that isn't very likely. However if our myriad interconnected systems begin to suffer multiple catastrophic failures, then I could easily imagine a complete, quick and irreversible loss of social cohesion leading to chaos and severe disruption of our modern JIT civilization.

Personally if I could find someone wanting to place bets on "chaos and severe disruption" , otherwise usually referred to here most often as "when tshtf", I would bet on it arriving withing fifteen years if anybody would give me odds of ten to one against.

The definition of arrival being say twenty five percent unemployment plus one thousand arrests per day for thirty days straight for rioting and looting within the borders of the lower 48;sales of new cars being off eighty percent over the average of the last decade;or something along these general lines.

I am very curious as to the odds others-especially biologists, engineers, financiers, and any others with relevant professional expertise- here would give or ask to make such bets.

Personally if I could find someone wanting to place bets on "chaos and severe disruption" , otherwise usually referred to here most often as "when tshtf", I would bet on it arriving withing fifteen years if anybody would give me odds of ten to one against.

I guess it would depend on many things not the least of which is pure luck. For example the 20 million or so people living through the unprecedented flooding happening in Pakistan right now are for all practical purposes suffering chaos and severe disruption as we speak.

Today we have about one billion humans at the edge of starvation. Could it come to pass that perhaps in twenty or so years due to combination of foreseen and unforeseen circumstances, compounded by natural and human made disasters we would have only one billion who were not enduring disruption, chaos and starvation? My emotional psychological mind wants to reject such a scenario outright but my logical dispassionate analytical mind can not simply discard that possibility out of hand.

To be clear I'm not a betting man, the one time in my life that I had to go to on a casino cruise for a company year end party, I bet $5.00, lost, and called it a night, so I'm probably not the best person to be asked to make bets.

FM, Thanks for the thoughtful reply;of course it is tough to delineate what constitutes tshitting tf;I waas thinking in US centric terms of course.

I have talked to some people who thinks the odds of a collapse are pretty high.

The guy who teaches micro biology at the local community college
(holder of a doctorate from a very well thought of univerity) thinks the odds of a really nasty contagious disease that will kill by the tens or hundreds of millions emerging (or the organism that causes it evolving to the relevant state of virulence) in the next couple of decades is at least ten percent.

But he won't say so publicly.

My own wag estimate of the possibility of crop failures on a scale capable of causing widespread starvation over continent sized areas due to outbreaks of disease or pests is fifty percent within the next twenty years;this due to constant international travel introducing numerous new invasive species of insects and diseases-one day some new bug is going to turn out to be the chesnut blight of corn, wheat,rice, or soybeans, given monoculture and lack of genetic diversity.

This does not include any allowance for lack of fuel , fertilizer, insecticides, or bad weather.

I am utterly convinced that when these variables are considered, the likelihood of widespread starvation is close to one hundred percent.

Of course here in the states we can eat corn if the wheat crop fails.

And (paradoxically) if the corn crop fails, meat will be dirt cheap for a good while as the farmers and feedlot operators liquidate thier livestock inventory.

And the food situation will be considerably worse in those countries that depend on food imports. The UK could not feed itself during WW2 without food imports, we now have 20M more people living in the UK.

The UK is very vulnerable to a global food shock. To make matters worse if the Pound collapsed we would not be able to afford imported food, so the result would be the same as a global food shock. The same goes for a major oil shock.

After WW2 a lot of farm land was used for housing, my house (built in the 50's) sits on what used to be a very productive apple orchard. A new town is being built about 30 miles away, again using a large chunk of farm land (wheat & cattle pasture).

Growing your own is less of an option as the size of home gardens have dramatically shrunk in recent years as gardens have been sold as building plots. For example, since about the 80's the village I live in hasn't increased in size, but the number of houses has increased by over 30%.

OFM - I'll give you ten to one against. My good friends working in economics and investment banking told me it can't happen...

All snarkyness aside, the kind of poo-hitting-the-fan that you speak of would probably require a sudden event to spark such a collapse. Even an extreme event such as a war in the middle east, a global pandemic, or even some kind of AGW related crop failure may not be enough to reduce the lower 48 to the conditions you describe.

Until then, I think the 'system' as we currently know it will keep lurching from one 'crisis' to another for another two or three decades. In the end, we will probably achieve 35% unemployment, 80% lower car sales, rampant crime and other such problems. The USA in 2030 or 2040 will resemble South Africa today. Of course, what that implies for South Africa in 2040 is probably very unpleasant.

In short, I'd take you up on that bet.

I think what passes for a "causal event" will decrease with time, while the impacts will increase.

A few years back, it took Katrina to knock a major city over, and it recovered (or recovers) slowly. More recently, Detroit failed simply due to a national recession coupled with a car addiction -- no water damage, but if I had to choose, I'd pick NO over Detroit as a place to pick up an inexpensive residence.

To an extent, major events will become excuses and blame-holders for an increasing frail fabric of our current society. Before much longer, you won't even be able to tell what caused the latest rent in the fabric (but we'll make up culprits anyway, because humans prefer causation to chaos).

If you build sand castles, they may get washed away by a wave or stomped on by teenagers, but sometimes they just crumble unexpectedly, and eventually they'll weather and erode regardless.

Big Stone Heads and Sand Castles - what else do we build, really?

DS,

I too am of the opinion that such a collapse will if it opccurs within the near future be triggered by some particular event;I suppose the only real difference of opinion then becomes whether there is a greater than ten percent probability of such an event.

Here is a short list of such events that probably would suffice as triggers in my estimation:

Collapse and rejection of the dollar, no more oil imports except if bartered;outbreak of super flu or comparatively destructive disease or pest of major monoculture food staple,corn wheat, rice, soybeans;hot war anywhere that closes the sea lanes for a few months; internal revolution in sand'noil country;shark fin production decline on the back side of the Hubbert curve, available to purchase oil declines faster even than Sam and Westexas expect;unavoidable cutbacks in spending here in this country, from ss to govt pensions to medicare resulting in backlash putting radical politicians in office;climate modelers wrong, too conservative, highly destructive warming shows up well ahead of schedule,California dries up.

I propose a small gentleman's bet;I win, you donate a hundred to TOD;I lose, I donate ten(I'm well fixed to live but cash poor and some assets we expected to sell eventually at a very good profit may never sell at a profit, considering the rate real estate taxes are consuming current cash reserves);assuming both still posting,and assuming TOD still up and alive.

I have seldom replied to a post of my own, but it just occcured to me that the various kinds of conservation easements available in some states and communities in the form of tax relief in exchange for giving up development rights might be well worth a discussion here some night soon.

I know people who live on small formerly profitable farms who cannot earn enough nowadays to pay thier real estate taxes out of thier other earnings;a two or three thousand dollar property tax bill cannot be managed if you must commute to Walmart to earn a living as a clerk.

I see a very large gap between those events (all certainly possible) and the collapse of the electrical grid in Gail's thoughts.

30% unemployment, both riots and potholes in the streets, WW II levels of gas rationing (3 to 4 gallons/week for A (most people), 8 gallons/week for B and much higher for C, which included railroad employees during WW II) will not be nearly enough to collapse the grid !

Actually reduced demand will make the grid much more robust.

Rotating blackouts through the suburbs are a superb management tool to make electrical demand = supply.

Alan

Ok, OFM, you're on. If we're seeing the above criteria by August 18th 2020, TOD gets 100 wingwangs!

OFM, I expect the above situation within months, let me put a July 2011 timeframe on it as the economy has begun it's next deflationary leg down. The system is remarkably fragile and is essentially on life support at the moment. Within 2 years I reckon it's a near 100pc guarantee!

While it is popular to focus on system overload or stressed systems that experience failures, how does MOL (minimum operating level) affect system collapse? How can MOL in one system create catasrophic failures in others. One example would be petroleum pipelines. Another would be credit/commodity systems. What is the minimum level of capital and energy injection into an economy before banking systems, housing markets or businesses begin to fail? Is that where we are now? How low can growth go, and is "no growth" even possible?

The computer network example used by the author has limited utility when compared to economies in this respect. The network doesn't care if individual nodes just sit there, idle. Economies tend to collapse when their participants produce little to nothing.

Growth is growth... if it is negative, it is collapse, No?

So... there is no such thing as negative growth. Or positive collapse.

There was a collapse is 1931; there was one in 2008, and it is either continuing or else it will happen again soon. Ask me how bad, and I don't know. Catastrophe is in the eyes of the recipient.

Craig

Growth is growth... if it is negative, it is collapse, No?

So... there is no such thing as negative growth. Or positive collapse.

I don't think things are so clearly delineated into pure black and white. I think that there can be gradations of gray along a spectrum. I also believe we can and should differentiate between a catastrophic and sudden collapse as opposed to a gradual contraction, which in my opinion could still be characterized as negative growth.

Growth is still growth. Negative is collapse, gradual or otherwise. Negative growth is an oximoron. Like jumbo shrimp or military intelligence.

I guess you could call it expansion or contraction (though my wife doesn't think so... says contraction is what happened when she gave birth).

Craig

You could use growth/shrinkage.

"Negative growth" is definitely doublespeak when used to communicate with people who aren't professional economists. A better term when speaking technically would probably be "scale delta" rather than growth since it has no preferred conceptual orientation.

If you listen to the media they are telling us this wil now be a double dip resession. I wonder what they are going to say when the next oscilation and step down begins. Will they have the gall to call it "a rare triple dip resession" or admit a long term decline in the world economy is underway?

"Networked Computing" has been defined as "A system for information processing where a computer you didn't know existed can bring all your work to a sudden halt".

There was a failure of a large packet switched network that was caused by a single-bit error in a patch. The error caused the "I'm overloaded, don't send me packets" message to be interpreted as "I'm overloaded, don't send anyone packets". When the patch was installed during a low traffic time, everything continued to work perfectly. As traffic built, one switch became overloaded and sent out the message to its peers. Since they could not send traffic, they became overloaded and sent out the same message to their peers. And so on, bringing down the entire network.

In communications networks, software causes more failures than hardware. Software used for traffic control, maintenance, adminstration, etc., that is seldom executed, causes more failures than software involved with normal operation.

However, in networks operational errors by people doing change, maintenance, and administrative tasks cause more failures than software.

I would expect that if an energy related collapse were to start, interventions by politicians, finanaciers, and businessmen would tend to make it more rapid and severe. This makes discussing and planning out potential scenarios vital, even if they never happen.

I've been in the sudden collapse camp for some time now, but the 08 near total collapse showed the resilience of the system. Althought it is a fiction movie, like Colonel Troutman said of Rambo when it was discovered he had survived a collapsed cave in, "The kid is resilient." So now I waver between sudden and slow collapse, but am beginning to err on the side of slow collapse.

The problem I see with slow collapse, is it's one thing to suffer a quick, hard blow that forces society to reduce in numbers then regroup, but a whole different ball of wax when it slowly breaks down. In the latter, it might be harder to regroup later due to the complete and utter destruction of all systems.

Take a look at "After Collapse: The Regeneration of Complex Societies" by Glenn Schwartz and John Nichols. The title is pretty self-explanatory, and many of the cultures they studied did collapse slowly, like a flan in a cupboard.

Interesting and thanx.

Two related articles that are not yet on-line: Skeptical Enquirer has Thinking Critically about Computer Security Trade-offs by Adam Slagell that delves into FUD (fear, uncertainty,and doubt) and risk assessment; Scientific American has Villasenor's article, The Hacker in your Hardware ... all about complexity and the soon probable viruses inserted into chips.

Some of you may remember another example of this in Noah Raford's work at the LSE on:
Collapse Dynamics: Phase Transitions in Complex Social Systems
http://news.noahraford.com/?p=48

It is very possible for a system to lock up quickly if you look at some of the modelling from chaos theory and observed real world events.

First of all, to increase the capacity of the rails more tracks must be constructed. Railroads are removing tracks not building new tracks.

Trains are limited structurally. The tong that connects the locomotive to the cars has a limited amount of strain. To limit the strain on the tong, the cars are being pushed as well as being pulled. However, as the number of cars being pushed and pulled increases, the number of locomotives increases until a limit is reached of about 3 or 4 both front and back. To increase the length of the cars further, a set of locomotives are placed in the center with locomotives both forward and aft to push and pull. There is a limit to this arrangement. I could envision one long train reaching from LA to St Louis. Of course this will never happen.

So to increase the capacity of the rails in order to reduce the container traffic of the trucks, new rail must be laid. I don’t see this happening.

Within the last couple of years BNSF has double tracked the entire length of the Trans-con; LA to Chicago.

UP has just about finished double tracking from LA to El Paso (where UP splits into 3 single track lines).

NS has just finished a double stack container clearance program (expand up) from Chicago to Norfolk VA and has announced clearance work on a spur from Columbus to Cincinnati.

Former CSX line from Jupiter to Miami was double tracked a couple of years ago.

No double track is being removed AFAIK.

http://online.wsj.com/article/NA_WSJ_PUB:SB120179835382432337.html

Best Hopes for more Rail Capacity,

Alan

A little good news, for once.
Thanks.

Stuart Kauffman's work has demonstrated that systems whose subsystem connectivity increases past about 3 become brittle. Looking out my window, it seems pretty plausible that we've moved well into the brittle regime, although it begs for some principled measurements.

aeldric
All interesting analysis but more time needs to be spent on the frequencies of the hypothetical disasters we are facing.

I don’t believe climate change is a fast moving disaster – all evidence points to the concerns manifesting themselves over decades. (the ocean heat sink for example)

Peak oil seems to be manifesting itself over a decades period of time (as prices rise other alternatives will be found that will result in an extension of time. – Ultimately we could run into the entropy limit but more likely will reach a cost mountain that cannot be climbed, but how many decades away will this be? ( Will we ever turn to nuclear electricity?)

As long as energy and some forms of communication exist, the grid, the transportation network (which I hope includes rail and more importantly electrified rail), the water and sewer network, the internet and the financial system can be repaired from local crashes.

On top of these devices that tend to prolong BAU there is the hope that we as a society can move to higher efficiencies, better designs, less necessary travel, more use of distributed PV and Wind. I would hope that eventually (if not already) we recognize the need for a new paradigm of BAU that leads to garnering efficiencies whenever possible and this by itself will move the disaster into the decades time frame.

I don’t see that this interconnectedness of our systems by itself will hurry a disaster, it becomes one more part of the complex problem and highlights the need for a multifaceted solution.

It's hard to escape interconnections in a global economy. Read Andy Xie:

We are seeing the interplay between the forces of globalization and policy mistakes. Globalization has severely restricted the effectiveness of economic stimulus. Trade plus FDI are half of the global GDP. Trade is visible in terms of stimulus leakage. But, where investment occurs in response to demand growth is far more important. Multinationals can invest anywhere in response to demand. It cuts the linkage between demand stimulus and investment response. The latter is crucial to employment growth, which is necessary for sustaining demand growth beyond stimulus. Essentially, demand is local, but supply is global. This is why the old assumptions on stimulus are no longer reliable.

That's at one level, the next level up has to do with net availability (of oil, coal, cola, hair dryers, etc.) which is dependent on four things:

- the production of (lower priced) existing reserves,

- production of (higher priced) replacement reserves,

- net exports,

- the price that supports all the above.

If the price of fuel is too high demand will drop along with price. When that price is too low the production of replacement reserves above that price is shut in, net exports above that price are not shipped and what remains is the shrinking amount of lower- priced fuel that can be produced out of existing reserves. Once it becomes apparent that this diminishing reserve of low- priced oil is all that is available it will likely be rationed carefully rather than being dumped on the market. The ability of an industrial economy to raise funds to produce reserves or import at a higher price out of output - or to craft alternatives - vanishes.

The bottom line is the amount of cheap, easily produced oil is shrinking fast. Why? Because of depletion's effect on output profits AND because the marginal barrel is now located 25,000 feet below some ocean's surface. The 'effect of production costs on profits' curve must be added to the net export curve and that of physical depletion.

Companies don't make money selling goods, they profit by the steepest yield curve in history; borrowing from the Federal Reserve @ -1% and lending money to the US Treasury @ 3% ... and by firing workers. Energy use at today's prices is unprofitable, yet the price of crude that is unprofitable to business is too low to support production of new oil.

Events of 2005- 2008 demonstrated the inability to replace current production at any price!

Start with the depletion rate of domestic production, subtract net exports, subract from the subtotal the amount of new production at a reduced cost (because of increasing poverty). The amount of oil available under this set of circumstances shrinks very fast indeed. And keep in mind, when the shortages hit, they will be permanent. In a depletion context there is no new wealth that can be shifted to oil producers to gain more available reserves.

Our industry 'profits' from the complete waste of fuel. The money return on destroying a resource must be greater than the value of the resource itself otherwise the resource cannot be brought to the market. Scarcity makes the oil resource too valuable to waste. We have created a defective model and built a defective infrastructure to support that model. Now, we believe we must use the same defective model to craft a replacement on a scale that can only be enabled by destroying the last marketable resource!

Is this insanity or what? The combined energy costs of running the model added to the energy costs of crafting a replacement are unsupportable on their face! The model cannot support itself now. The 'return on waste' nears zero! Attempting to use the model to leverage another will be the overload that collapses the system. We can only craft a replacement energy regime when the existing model and its demand on resources is powered down. Then we MIGHT be able to use what remains to craft a more capital- enhancing model.

Far more capital will be required to increase availability than is required currently. Note how much the Macondo well cost, (and factor for BP's cheating). Oil is more and more expensive to produce. If we cannot leverage commerce profitable enough to pay the capital expense our experiment in industrialization is done.

What is that oil price level? The current price of oil is a 'wealth barometer'. In 2008 the world was rich and could afford $147 for a little while. Today we can afford $87. This is not a good trend when replacement oil costs + $70 a barrel to produce.

The wealth barometer could tip down to - $50 at any time, within months. The trend has been down since 2008. The world is getting poorer; what does 'wealth' mean, anyway?

Prior to 2005 it could have meant equity in real estate, credit derivatives or access to credit, now it becomes 'cash in hand'.

The bottom line is historical concern of living within our means, to cut the fiscal, personal - and energy - deficits to what can be supported by the careful husbandry, so that there is a return on oil's USE not WASTE, where oil becomes capital to increase the same way topsoil is capital to increase. This means an economy that makes capital use of what remains of our oil resources long term. Since we use close to 20m barrels per day a reduction to a sustainable domestic rate of production (1,000 years?) would be less than one million barrels per day. This means an economy that is ONE- TWENTIETH (1/20th) of our current economy!

If the price drops low enough the shortages will start almost immediately and the system will grind to a halt in an energy- shortage compounding spiral. The time frame would be months not years.

Get ready.

Thank you Steve, for explaining the manner that could lead to disaster in a short time period.
Elsewhere,I have read about alternate oil sources being produced commercially at higher prices for oil. You seem to believe that we cannot sustain an economy at a higher price for oil-- and I would agree if we keep making the same global economic errors we may create a financial trigger that causes producers to drop out rather than increase expenditures per the current model. One would hope that higher prices spur multiple approaches to solving the problems of production and transport that a society depends on but a point could come where costs to develop alternatives out paces the dwindling profits of the current industry. Kind of like reaching the edge of the graph paper that supply Vs demand curves are plotted on.

I also believe that there is some unknown regarding cost of current production. If there were only a $ 15 spread then oil companies would already be asking for exemptions to drill for known but off limits oil reserves. (I do not see a desperation; maybe it is well hid.) I do expect to see more desperation in the eyes of energy producers well before they hit the wall you describe, but I confess to not understand the production of energy as well as you.

Other than these comments, your analysis is sobering so I would assume we no longer need worry about climate change?

My comment was in terms of pinning down this gloom laced time period. I grew up in the era of Paul Erlich and my faith in him was dashed as society overcame his prognostications-- I want my prophets to be more precise now that I am older.

Lots of McDonalds (not all) shut down in Japan. Many other fast food chains also have closed many stores. Gas stations are closing. But still some remain.

Is this collapse?

Definitely! The commerce that used to support many people is now gone. It is a spiral effect.

When everyone is really, really poor, a bunch of also quite poor oil rig workers will ask a bunch of people "would you like some oil? Give us some money and we`ll be able to get it for you."

But no one will have any money or any eggs or any bread to trade for the oil. It just won`t be possible to get the oil.

I can see that happening sooner in some countries than others.

Probably if you still have all your MCDonalds still open collapse is farther off.

Thanks Steve, that's a pretty good summary of our predicament. +10! If you don't mind I'll be sending copies of your post to a few people who might have their eyes opened by reading it.

I have recently had an article published that looks at the risk of systemic failure to military technology. Although focused on the military I believe it is applicable to technology generally.

It is titled Lasers or longbows: A paradox of military technology,
available from: http://www.adfjournal.adc.edu.au/UserFiles/issues/182%202010%20Jul_Aug.pdf (from p.44)

From the intro:

Fundamental to the argument presented in this paper is the assertion that military capability
and complexity are in a symbiotic relationship. Capability can rarely be increased without a
corresponding increase in complexity. Complexity, however, comes at a cost. The costs come in
a number of forms including finance, resources and the vulnerabilities to which the capability
is exposed. This can be summarised as increased capability equals increased complexity equals
increased vulnerability, forming a paradox of military technology. The paradox is that:

The advantage provided by the increased complexity of a military capability increases
the vulnerability of that same capability to systemic collapse due to its reliance on complex
supply chains.

"This raised a question in my mind: The Internet Protocol was originally designed to be a robust, reliable, redundant system. How does one piece of software on one machine bring down a network with thousands of nodes?"

I hate to be Downer Dan by arguing with the basic premise of your post, but as a network professional I have to say it is unlikely that a misbehaving driver on your NIC had the capability to affect much of the network, especially if the network was designed for scalability.

IP, or the "Internet Protocol," segregates network subnets at Layer 3 of the OSI model through the use of specialized computers called "routers". However, a network interface card operates with Layers 1 and 2 of the OSI model. So a misbehaving NIC has the ability to affect only its local network (subnet) and other devices directly attached to that local network. If your client computer is on that subnet or communicates to the Internet through a router attached to that subnet then your communications to the Internet may have been affected. But the rest of the network beyond that affected area would have been fine.

In a rough analogy, it is like if plague breaks out in a single city and the city is successfully quarantined by closing roads and airports in/out of the city. Yes, people in the city may well get the plague and they will not be able to leave the city, but other cities will not be affected.

Thus it stands to reason that a complex system must have inherent abilities to prevent dysfunction in one part of the system from affecting other parts of the system, otherwise that system would never have evolved to complexity in the first place.

Of course, if the dysfunction is system-wide then that is a whole other matter, but that is not what happened in the case of the misbehaving NIC in your post.

http://en.wikipedia.org/wiki/Osi_model

DD

It's above a NIC, but I have seen mis-configured equipment serve DHCP addresses in competition with proper servers, and wreak havoc across a fairly broad LAN. Such things can escalate when somebody starts boot-cycling equipment that was running fine, but then fails to find resources and hangs abnormally.

Sure, things like this SHOULDN'T happen, but they do...the frailty of humans managing complexity.

Yes, you are agreeing with me. You use the term "LAN" which means "local area network".

DHCP works with the broadcast capability of IP, but that broadcast capability happens at Layer 2 of the OSI model and is therefore limited to the local network, or "subnet". Routers do not forward broadcasts between subnets unless they are programmed to do so in violation of the basic rules of IP. So in normal operation where the rules have not been broken by an administrator a rogue DHCP server can only affect the systems on the local network.

Now, if systems out on the greater network are dependent on the systems affected by the rogue DHCP server then they will also be affected. But proper design of a fault tolerant network will ensure that these dependencies are not tied to one small area of the network and are instead distributed throughout different regions.

Disregarding entropy of equipment, computers only do what you tell them. Unfortunately humans don't always tell them to do things correctly, either through software bugs or misconfiguration.

If a complex system is able to remain static then it will continue to the run the same way forever. It is the outside influences of entropy and environment that can cause it to eventually fail if it is not able to adapt to changes.

DD

An abrupt stop to our oil supply would mean no transportation, mass chaos, hunger and death. Imagine our cars, trucks, buses, trains and planes not moving for one month.

True, although a complete stop is very unlikely. A better question is what if the Strait of Hormuz closed tomorrow?
Long term I think Peak Oil may well be an anti-climax, and the residual long lasting global warmings impacts may be
the bigger issue. There are no technical problems getting off oil, only political ones.

However Per Bak, in his book “How Nature Works,” shows that there is an instructive way to look at this question.

Word to the wise. Most of Bak's writing is better described as "How Nature Doesn't Work". He was always looking for self-organized critical phenomenon signifying fundamental phase transitions in behaviors that were almost always better described by basic considerations of disorder and entropy. No one really called him on it because he was apparently a nasty SOB when confronted at conferences.

That's the problem with a lot of this research, ala Bak's sandpile model, as the investigators are willing to risk putting out some advanced theory because the potential payoff is great: a Nobel prize perhaps. If they are wrong, we just have to wade through more crap.

Can you expound on the sand-pile experiment, and your take on it? It seems at face value to be a simple realization of chaos, and likely follows a simple entropy-driven (or is entropy seeking a better term?) model.

I was thinking of entropy, energy, capital, and resources from the simplistic economic systems transform perspective, where you put in resources on one side and get capital and waste out the other.

From an entropy perspective, high quality resources, including ores and energy, could be viewed as by-product capital outputs of earlier processes which happened to create a strong entropy gradient -- for whatever reason, these resources represent a high state of order.

The goal of many (most? all?) systems seems to be to create new capital which represents an even higher state of order -- an embodied negative entropy? -- while reducing other resource inputs to higher-entropy waste. Perhaps besides embodied energy, capital could be viewed as embodied (negative) entropy?

With bulk supplies of low-entropy resources, there is a lot to work with to create high-value capital. As those run short, it will take more and more lower-quality resources to maintain high-capital output, and eventually those will decay as the resource base falters.

Perhaps there is a gradient perspective of entropy or energy for driving such transform engines, such that the ratio of capital created to waste generated is a function of the quality of the inputs? With a Carnot engine, the temperatures between the source and sink bound maximum efficiency.

Note that in a very real way, the human population is "high-value capital" in a resource driven equation.

mercurius---- An abrupt stop of our oil supply probably will not happen soon, but the huge consequences if it did requires us to be proactive.

Oil will not run out before people misbehave. There are many places we have not looked yet, and our techniques will improve.. Burning oil and the resulting CO2 being the cause of global warning needs to be proved, not just guessed at with political science like it is now. Reducing CO2 in the atmosphere can be done with new technology.

There are many ways to replace oil for transportation, but it will take decades, and we are not doing the necessary R&D or investment now.

CO2 being the cause of global warning needs to be proved, not just guessed at with political science like it is now

Pure and simple BS!!

http://www.washingtonpost.com/wp-dyn/content/article/2010/08/01/AR201008...

http://www.guardian.co.uk/commentisfree/2010/aug/01/climate-change-robin...

Alan

Our civilization is a complex networked system far from equilibrium. It's been driven by huge injections of energy to reach this state, and along the way we've already seen bifurcations like the dot com bust, katrina debacle, the financial crash, collapsing employment. As more and more energy is needed to simply to maintain system state, chances of a big and likely unpleasant state change are almost assured, particularly as energy flows decrease.

I'm in the slow camp. At least in North America we do have significant excess capacity that can act as a buffer for a considerable time, in the form of gross inefficiency and waste. With the first effects of collapse, first waste will be addressed quickly, and then efficiency even if less quickly. These two will provide time for rationing to be applied, and that can buy a lot of time to make large system changes. Many system changes can be made quickly and without hardship. a simple example is importation of exotic or out of season food.I remember visiting friemds in Rome many years ago when residential electricity was rationed by rolling black-out at about 6 hours per residence per day. Life went on as usual.
If already technically practical efficiencies were implemented, and FF energy substituted by renewables plus some nuclear, the USA could maintain the present basic (free of a lot of pure luxuries) living standard on about 1/4th of the energy now used. The necessary changes could be implemented over 30 or so years if the will were there, and there would be nothing remotely like societal collapse.

Question for Mr. David Clarke....what do you see as the warning posts for systemic collapse?

Excellent article. Very readable. Thank you for provoking deep, interesting thought. I assume, however, that your office is not in NW Pakistan. That is a sudden, catastrophic collapse in individual lives. From outer space, looking in, the individual is as insignificant as a red or blue dot on the model. But within an individual human, that's what counts, that individual red or blue dot who is myself. That's important, too, because we seem to be falling apart one by one although the system has held up so far. When we do fall apart, it is sudden and catastrophic, not a slow decline. When and how will it all go down? My guess is individual by individual, small group by small group. I can't know for myself without closely monitoring my own world, and even then, I can't tell. I don't know what part of the system is about to backfire onto me.