Montserrat is one of the many islands sprinkled among the Caribbean West Indies. In just 39 square miles, it boasts mountains, rainforests, beaches and groves of bananas, mangoes and coconuts. The air temperature rarely dips below 78 degrees and neither does the water.
In short, Montserrat is paradise. Or it would be but for the Soufriere Hills volcano, which erupted for the first time in July 1995 and hasn't stopped since.
Soufriere Hills has rendered nearly two-thirds of the island--an area now called the Exclusion Zone--uninhabitable. Since 1995, the island's population has fallen from 11,000 to 4,000. The volcano has buried Plymouth, the former capital. It killed 20 people in one violent belch in 1997. It has suffocated the economy, once driven by tourism and rock stars like Sting, the Stones and Paul McCartney, who partied and recorded music there at Air Studios, the recording facility once owned by the Beatles' former producer George Martin but now buried by the volcano.
This dichotomy--Eden on one side of the island, the fires of Hell on the other--makes Montserrat a perfect laboratory for risk analysis. Just as much of Montserrat is buried in ash, it's also buried in probabilities. Scientists know, for example, that there's only a 3percent chance that Soufriere Hills will stop erupting in the next six months. They also know there's a 10 percent chance of injury from the volcano at the border of the Exclusion Zone, and they can draw an imaginary line across the island where the threat from the volcano equals the threat from hurricanes and earthquakes.
"Thirty years ago, you needed the biggest computer in the world to do the statistical risk analysis," says Willy Aspinall, who helped develop these figures in the shadow of Soufriere Hills. "Now all you need is a laptop and a spreadsheet." He says the risk calculations get better and more textured all the time. He uses Monte Carlo risk analysis simulation software and spreadsheets to quantify the risk levels that help decision-makers minimize the volcano's threat to people's lives.
If this type of risk analysis is good enough for Aspinall, it ought to be good enough for CIOs, especially now that they're working in an economic environment looming as ominously over their businesses as Soufriere Hills looms over Montserrat. For the most part, though, CIOs have not adopted statistical analysis tools to analyze and mitigate risk for software project management.
This is why they should.
Risky Business
Experts will tell you that statistical risk analysis is as essential to real portfolio management as a processor is to a computer. Without it, portfolio management is simply a way to organize the view of projects that will almost certainly fail. CIOs who are serious about portfolio management need to be serious about statistical risk management.
"If you don't succeed with risk management, you won't succeed with project portfolio management," says Raytheon CIO Rebecca Rhoads, who credits risk management with lowering her project failure rate and helping Raytheon IT achieve its cost-performance targets. Rhoads is ahead of the curve, but despite her engineering background, she has yet to apply the kind of sophisticated statistical analysis that Aspinall uses for his volcano.
Robert Sanchez, senior vice president and CIO of Ryder, credits risk analysis with bringing order to his company's decision-making process for projects. He would welcome statistical analysis, but he's not there yet. "Have we really embraced it completely and understood it in all of its detail?" Sanchez asks rhetorically. "No, we haven't. But we will."
CIOs should become familiar with two statistical tools. They are the colorfully named workhorses of risk analysis: Monte Carlo simulation and decision tree analysis. Probabilities figure heavily into both, which means that risk has to be quantified. CIOs must draw their own line between the Exclusion Zone, where it's too risky to venture, and the beaches, rain forests and coconut groves, where the living is easy and the threats are manageable.
The Trap of Common Sense
Even a simple task like choosing to drive to work requires a risk assessment, although not a computational one; you can do shorthand probability in your head. Though the cost of being wrong is high, the risk is relatively low (a 5 percent probability of being seriously hurt in a car accident) and easily mitigated by wearing a seat belt.
This sort of informal risk analysis can sometimes be useful. Steve Snodgrass, CIO of construction materials supplier Granite Rock, has the misfortune of managing IT for a company that literally straddles the San Andreas Fault. Snodgrass doesn't need statistics to tell him that it would be a bad idea to do nothing to mitigate the possibility that a quake will take out his critical applications. So he outsource his applications' backup far from the fault line.
However, CIOs often use this kind of commonsense reasoning as a way to avoid doing real risk analysis, say Tom DeMarco and Timothy Lister, authors of Waltzing with Bears: Managing Risk on Software Projects, a primer on statistical risk analysis for IT. "It's been very frustrating to see a best practice like statistical analysis shunned in IT," says Lister. "It seems there's this enormously strong cultural pull in IT to avoid looking at the downside."
In lieu of choosing projects based on acceptable risk, Ryder's Sanchez says, IT often uses what he calls the moral argument, in which the greatest risk lies in not doing the project. Therefore, the risk is mitigated by doing the project. This reasoning was particularly valid during the boom years when there was a palpable fear of getting left behind technologically. But it was never called risk analysis. "I came into IT and was never really comfortable with the moral argument," says Sanchez, whose background is in engineering and finance. "I was looking at it thinking, We analyze the risk of building a new office, but we don't on an ERP system that costs the same amount."
How to Create a Risk Analysis Process
As the director of foreign exchange at Merck, Art Misyan uses statistical risk analysis for evaluating the impact of foreign currency volatility. Like Sanchez, he's puzzled by IT's laissez-faire attitude toward risk analysis. "Risk gives you the ability to look at a whole range of outcomes, but IT looks at only two possible outcomes," he says. "Either you hit deadlines or budgets, or you don't."
IT needs to think in probabilities, Misyan says, not ones and zeros. The best way to start is for the CIO to formalize the risk process. "First you have to set up a process to determine and track risks," he says. The good news is that much of the risk process is built into project management methodologies CIOs have been adopting anyway, so it should be familiar. Here are the basics for developing a risk analysis process.
Gather experts to determine project risks. These brainstorming sessions should be free and creative. "You want the pessimist in the group, the dark cloud," says Anne Rogers, director of information safeguards at Waste Management, who teaches risk analysis. "You want the person that will ask, What if a truck ran into the building?"
When you don't ask the off-the-wall question, you run the risk of smacking into it. "Motorola gambled on developing Iridium satellite phones and charging $7 a minute," recalls DeMarco. "No one seemed to wonder what would happen if cell phones came along offering similar service for 10 cents a minute and free nights and weekends."
Assign researchers to uncover known risks. "We came up with 20 or 30 risks we knew we'd face by research," says Sandy Lazar, director of key systems for the District of Columbia, who is overseeing a five-year, $71.5million administrative systems modernization program. "If you read up, you realize ERP has failed over and over for the same reasons for 15 years now." In fact, there are five typical risks to software projects that every CIO should include in a risk analysis.
Divide risks into two categories--local and global. The risk of staff turnover during a project is a local risk. War is a global risk. Often, those new to risk analysis focus only on the local risks, but they need to consider the global risks and their impact. Create a template for each risk. The template should include a unique risk number, a risk owner, potential costs (in dollars and other terms), a probability of occurrence (a low-medium-high scale will do at this point), any potential red flags or signs that the risk is materializing, mitigation strategies and a postmortem for noting if the risk factor actually happened.
One important footnote for developing this process: Value consistency over accuracy. If you do things in a consistent manner and the numbers are off, at least they'll be off in a consistent--and therefore fixable--way. "The process," says Raytheon's Rhoads, "is so much more important than the math rigor. Mature, consistent processes--you need that first."
How to Use Monte Carlo Simulations
Once you have a repository of project risks, you can get statistical. The most commonly used tool for this is the Monte Carlo simulation. This technique was developed in the 1940s for the Manhattan Project. It's used today for everything from deciding where to dig for oil to optimizing the process of compacting trash at a waste treatment facility. It's a deceptively simple but powerful tool for risk analysis. All Monte Carlo really does is roll the dice (hence the name).
Here's the theory: Roll a die 100 times, and record the results. Each face will come up approximately one-sixth of the time--but not exactly. That's because of randomness. Roll the die 1,000 times, and the distribution becomes closer to one-sixth. Roll it a million times, and it gets much closer still.
The die represents risks--albeit evenly distributed, predictable risks--where each side has about a one-sixth probability of occurrence or a five-sixths probability of not occurring. What if each die were a project risk and each side represented a possible outcome of that risk? Say one die was for the risk of project delays due to staff turnover. One side would represent the possibility that the project is six months late because of 20 percent turnover. Another side could represent a two-year delay due to 80 percent turnover. The die could also be unevenly weighted so that certain outcomes are more or less likely. There would, of course, be dice for other risks--sloppy development, budget cuts or any other factor unearthed during preliminary research.
Monte Carlo simulators "roll" all those risks together and record the combined outcomes. The more you roll the dice, the more exact they make the distribution of possible outcomes. What you end up with resembles an anthill, where the highest point on the curve is the most likely outcome and the lowest ends are possible but less likely.
Once you determine a project's risk profile, you can build in extra resources (like money and time) to mitigate the risks on the highest points of the curve. If the distribution says there's a 50 percent probability the project will run six months late, you might decide to build three extra months into the schedule to mitigate that risk.
Monte Carlo simulators also let you run "sensitivity analyses"--rolling only one die while keeping the others fixed on a particular outcome to see what happens when just one risk changes. A health-care company (that requested anonymity) using a Monte Carlo simulator from Glomark ran a sensitivity analysis for a pending software project. Each die was rolled, one at a time, 500 times while the other dice were kept fixed on their most likely outcomes. The exercise showed that three of the nine risks represented 87 percent of the potential impact on the project--allowing the company to focus its energy there.
You can (and should) repeat Monte Carlo simulations for all the projects in your portfolio, ranking them from riskiest to safest. This will help you generate an "efficient frontier"--a line that shows the combination of projects that provide the highest benefit at a predetermined level of risk--something like the line across Montserrat. An efficient frontier helps you avoid unnecessary risk. It will help stop you from choosing one project portfolio that has the same risk but lower benefits than another.
Admittedly, this description glosses over some of Monte Carlo's dirty work. Someone has to determine which dots to put on the dice and how to weight the individual dots. That's your job. Canvass your experts, mine historical data, and do whatever else you can to come up with possible outcomes from each risk, and then estimate the probability of that result occurring. In other words, the risks themselves are a range of outcomes contributing to a further range of possible outcomes for any given project, or even combinations of projects.
You need only the best, worst and most likely outcomes to draw an effective triangle distribution. The best way to create the risk dice is with a triangle distribution. Determine three data points: the best case outcome, the worst case and the most likely case. Assume the best and worst cases have low probabilities and the most likely case is somewhere in between.
In the staff turnover example, the worst case might be a two-year delay due to 80 percent turnover. The best case may be no delay due to no turnover. The most likely--based on experience and research--might be the previously stated six-month delay from 20 percent turnover. Chart this on a probability distribution grid, and you get a triangle.
Take that triangle and others you create for all project risks, run Monte Carlo simulations, and you'll come up with the smooth anthill curve that shows overall risks to your project.
Vitro, a $2.6 billion glass company in Monterrey, Mexico, has done this on many IT projects (it's now required for projects valued at more than $20,000). "No one wanted to measure at first," says Gustavo Benitez, manager of Vitro's IT supply chain. "Because measuring makes you accountable. We're not that deep into it; we only use best case, worst case and most likely, and already it helps. It helps you see different scenarios."
Certain risk metrics are predetermined. DeMarco and Lister's five core risks to software projects have been given probability distributions based on historical data. If you're still worried about assigning a meaningful number to risks, Lister says relax and just guess. "Guess a number just to get going," he says. "Even that will be better than how IT approaches risk today."
How to Do Decision Tree Analysis
The other major risk analysis methodology applicable to IT project portfolio management is decision tree analysis. Where Monte Carlo excels at shaping what happens when many risks are in play at once (such as launching an ERP project), a decision tree proves most useful at mapping either-or situations and the sequential risks that follow each decision (for example, either I build a new factory or retrofit an old one). Each choice is a branch with an attached probability, derived the same way all risks are derived--through brainstorming and research. Each branch leads to other branches, which are the risks that result from choosing the original branch.
The key to analyzing decision trees is knowing that the probabilities compound. This is why Waste Management's Rogers likes them. Decision trees often show that good risk decisions are counterintuitive, if you follow the branches. "I'll give some outlandish choice, like rewiring your office or building a new one," Rogers says. "Everyone thinks they know which is less risky, but you watch the risks compound over time and guess what, it's not nearly as risky as you think to build a new office in certain situations."
Decision trees are also powerful presentation tools for executives, as long as you don't drop an entire tree on the CEO. They can be unwieldy. They branch out quickly with thousands of potential paths. ACIO must keep them under control. To demonstrate the path of certain decisions, limit the branches to only the most important risks, and simply leave out remote risks.
Final Analysis
Risk analysis takes some getting used to. Anyone not steeped in this world may misinterpret the results of any given analysis. For example, if you say there's a 90 percent chance it will rain tomorrow, and the day ends up being sunny, your analysis still may have been perfect. There was, after all, a one in 10 chance the sun would shine, and indeed it did.
The hardest part of risk, especially for CIOs, is that it doesn't provide concrete answers. Risk analysis will not tell you which project to do. It will tell you which ones fall into a certain level of risk and payoff. Provide the same IT portfolio and the same risk analysis to Rogers of Waste Management and Lynn Caddell, CIO of Yellow Freight, and they will most likely choose a different set of projects to pursue.
"We're not real risk-averse right now," Rogers notes. "If they're handled right, we're ready to take some risks."
On the other hand, Caddell's current crusade is to ensure timely delivery of projects, and she's not eager to take on anything that might compromise that mission. "Last year we were 96 percent on time with projects, and I attribute that to risk management," she says.
Ultimately, the CIO must understand his enterprise's culture and know which risks are worth taking. No amount of risk analysis will lift the decision-making burden from the CIO's shoulders or ensure the success of any project. Risk analysis is a tool, not a substitute for leadership. After all, it takes a leader to understand that although there's no such thing as a sure bet, decisions must be made.
"In these matters, the only certainty is that nothing is certain," famously noted Pliny the Elder, who, incidentally, died from ash inhalation in 79 A.D., when the volcano Mount Vesuvius erupted.