AS SOON AS Michael Baker Corp.'s IT department finished installing a Web-based procurement system from ePlus, Bruce Higgins, CIO of the$405 million engineering and construction company, was bombarded with inquiries from colleagues about the system's effectiveness. He had ample anecdotal evidence suggesting that orders were getting turned around more quickly, but he didn't have tangible proof that the system was improving efficiency.
So he decided to conduct his first ever post-implementation audit (PIA) of the new system. A PIA is a top-to-bottom evaluation of the hard and soft benefits derived from a strategic information system, the security of that system and the project management process for deploying it. From the PIA, Higgins learned that because IT miscalculated the number of people needing to use the new system, the ROI was driven down by the cost of ordering additional licenses. The PIA also showed that the system was saving the company more than $150,000 yearly, however, so Higgins was able to prove the system's worth to his colleagues. And he learned some valuable lessons on what not to do on subsequent projects.
CIOs would do well to follow Higgins' lead in realizing that a PIA is a worthwhile way to prove the value of IT. But many don't. In fact, Barbara Gomolski, a research director with Gartner, estimates that a mere 20 percent of companies take the time to conduct PIAs.
Companies avoid post-implementation audits for many reasons: They take too much time and drain away valuable personnel resources - two things currently in short supply. They require reams of documentation so that processes and results can be validated. Finally, project sponsors and implementers fear that the results of an audit, if unfavorable, will be used against them.
The CIOs at companies successfully performing audits have identified critical success factors, including: getting the right people involved, timing the audit properly, and collecting enough documentation to facilitate the smooth execution of a PIA. They pick the right projects to audit. And these CIOs share several key traits on how they ensure that PIAs become a sustained practice in their organizations: They are all committed to continuous improvement, they've made PIAs a part of their project management methodologies, and they have their CEOs' support.
But companies not performing PIAs are missing out on the important benefits such data provides. Although the challenges and risks associated with PIAs seem formidable, 66 percent of this year's CIO100 honorees always or frequently conduct them, according to a CIO survey. Since the honorees were chosen for their exceptional resourcefulness, their prevalent use of the audit establishes it as a best practice among highly effective organizations. PIAs provide a thorough approach for proving the value of high-cost, mission-critical IT investments and for gleaning project management best practices, which CIOs can then apply to keep subsequent projects on track. At a time when the value of CIOs, IT departments and IT investments are under increased scrutiny, PIAs are now more critical to CIOs' success and survival than ever before. "With the pressure on business today and the responsibility of IT to help the business units understand where their dollars are being spent, I really have trouble with anyone saying you shouldn't be doing [PIAs] in this [hard economic] time," says Jim Smith, CIO of Sun Life Financial and a strong advocate of PIAs.
How to Overcome Resistance to Audits
Many CIOs and their IT organizations shy away from audits because they think they'll be scapegoated if the audit reveals problems or lack of ROI. That's why Michael Barilla, vice president for business information services of Greif, a $1.6 billion manufacturer of industrial and paper packaging, frequently reminds himself and his IT staff that systems implementations and software deployments are not IT projects, but business projects. So if a project fails, he says, "it's going to fail as a team," and everyone - not just IT - will be accountable.
Gartner's Gomolski concurs. If an audit exposes a shortcoming, she says, that problem shouldn't only reflect on IT, especially if the project was jointly sponsored by the business, IT and finance.
"Exposing problems can be a double-edged sword," says Gomolski. "But it's better to be proactive. If you haven't achieved the value you thought you would, figure out what you can do to change that."
Another common concern about PIAs shared by CIOs, IT employees and even businesspeople is that they suck up too much time and require too much toil. But companies that perform PIAs say they shouldn't - and don't - take a lot of time to execute. Honeywell Aerospace aims to spend no more than seven to 10 days conducting an audit, and Sun Life Financial tries to complete PIAs within two weeks.
Furthermore, CIOs who conduct PIAs say the time and resources audits use get recouped on subsequent project implementations. Resistance to audits also often stems from a desire to be rid of a project once the deployment is complete and to move on to the next challenge. The trick is to make it easy for workers to conduct and provide feedback for PIAs. For example, when Sun Life Financial's IT project office needs to solicit feedback on a system from business users, it asks them to fill out an electronic survey, on their own time, that takes no more than 20 minutes to complete. The response rate for those surveys is usually better than 50 percent and is sometimes close to100 percent, says Ed Esposito, director of the project office.
Because Honeywell Aerospace's IT organization for its Aviation Aftermarket Services unit also has trouble reengaging business users in a project during an audit, the IT organization leverages its staffers who have completed training in Six Sigma, a process improvement methodology. Those with Six Sigma training, who are experts in business processes, provide the IT department with the feedback it needs to determine whether a system has streamlined workflows.
Engage the Right People
Who should perform PIAs is a matter of great debate. The most common groups of workers include one or more of the following.
At Sun Life Financial, IT's project office leads the PIA process on its own IT and non-IT projects. But it does so in conjunction with the finance department and the company's internal audit department. IT projects are audited from the beginning and on an ongoing basis, rather than at the end of an implementation, which ensures that IT follows sound project methodologies, meets user requirements, stays on budget and implements proper security controls.
Sun Life Financial's approach comes closest to being the best, according to PIA experts. Don Christian, a partner with PricewaterhouseCoopers, says the PIA team should consist of a businessperson and an IT person who were involved with the implementation, and that it should be led by someone independent, such as an internal auditor, who was not part of the project team. Christian says it's better to have a group of people from different functions participate, rather than just an IT team or just internal audit because they all provide valuable input. The advantage of having members of the IT project team involved is that they're intimately familiar with the benefits, deliverables and requirements of the project. And because they know the project so well, it is easier for them to fully evaluate a project. Having a businessperson on the audit team is important because she can more easily determine if an external factor rather than a systems failure is causing a system to not generate expected value. And an independent auditor is important because he's not afraid to ask tough questions and will prevent the members of the project team who are involved in the audit from softening any findings.
Time It Right
When to conduct an audit, just as who should lead it, is a matter of debate. When to start depends on the type of system or application deployed, the amount of time it will take before the application begins generating some results or data, and the amount of time it takes staffers to get acclimated to the new system and new processes. Generally the audit should take place well within a year of implementation.
"When we started the audit a month after the implementation, we didn't have enough data to see if the system was successful or not," says Michael Baker Corp.'s Higgins. "It's really important to make sure the system is up and running long enough to have enough data in the system that you can analyze."
Ken Cunneen, IT leader for technology and integrated supply chain systems in Honeywell Aerospace's Aviation Aftermarket Services division, says if a company implements a financial system that only gives results once a quarter, the audit team should wait at least three to six months until the system has generated enough data before performing a PIA.
On the other hand, the sooner you start the audit, the fresher the data and the easier it is to cull lessons learned. Which is why PIAs should not be a onetime event, says PWC's Christian.
Sun Life Financial's project office begins PIAs within a month of project completion to do a postmortem on the implementation process and to get feedback from users to make enhancements to the system. "When we have benefits that are realized over a period of years after the project is implemented, we do follow-up reviews," says Sun Life Financial's Esposito.
Higgins plans to take that iterative approach with Michael Baker Corp.'s ERP implementation, which is scheduled to be completed by the end of this year. "We will begin to audit the effectiveness of the new system within a couple of months after we finish implementing it," says Higgins. "Then we'll probably start to look at the ROI six to eight months after we go live so that we have enough data to perform a good analysis."
Maintain Meticulous Documentation
Another key to successful PIAs is good documentation, which takes many forms. It includes the business case that outlines the system's expected cost, benefits and ROI; the project's time line, including key metrics and milestones; a breakdown of the system's technical requirements; a record of the security and financial controls that have been put inside the system; and a compendium of all the changes that have been made to the system or project plan. But getting good documentation is one of the biggest challenges CIOs and IT departments face in conducting PIAs, particularly if they're doing one for the first time.
Sharon L. Thompson, director of IT audit for the AARP, says that if an auditor discovers that the implementation team does not have a thorough business case, a detailed project time line or meticulous records of changes and security controls, the auditor should note that discovery in the audit report and include a recommendation for the implementation team to keep better records.
To make life easier for himself and his post-implementation auditors, Higgins uses a Web-based collaboration tool called MPOWR - developed by members of his IT department to do version control and to archive changes made to the project's scope, templates, time line and budget. When it comes time to do a PIA, Higgins sends the auditors a link to the MPOWR site with a user name and password.
Sun Life Financial's IT project office also developed a project portfolio management system in-house that the project manager uses to report on a monthly basis changes to a project's cost, milestones and schedule.
Act on Lessons Learned
If you don't use PIAs to identify ways to improve project management, you're not realizing their full value. Sun Life Financial's Esposito says project team members who served on the audit and implementation teams hold a meeting during the PIA process to discuss what went well and what went wrong both during the implementation and the audit, and to identify areas for improvement.
To ensure that people take these lessons to heart, Sun Life Financial posts them on the corporate intranet and submits to senior management a summary of findings from PIAs of all projects done during the course of the year. Esposito also makes changes to project management and implementation processes if a project's PIA indicates that changes are warranted.
Sun Life Financial annually reviews the audits of different projects to identify common areas that need improvement. Gartner's Gomolski says it's important to do this type of project-to-project comparison. "The real value," she says, "is in seeing some of the patterns. For example, 'We screw up every time we do an integration project. What does that mean?' Or, 'Every time we do a project with this business unit, we suffer from amazing scope creep because they don't really know what they want, so maybe we have to work with them differently."
Sustain the Momentum
Now that you're armed with convincing arguments on the importance of PIAs and success factors for pulling them off, making them a sustained practice in your organization should look less daunting.
"If you have the project's objectives, approach, how it's going to be organized, how the business areas are involved in the project, the expected costs, and the anticipated hard and soft benefits well-defined up front, and you manage change along the way, then when you come in to do the post-project audit, there are no surprises," says Esposito.
Not only do PIAs go smoother when you've done all the hard work on the front end, but you increase your odds of actualizing the system's value. "By getting a better understanding of project costs up front," says Higgins, "you'll be more likely to hit the ROI you're trying to meet."