"By late Sunday afternoon, Microsoft had two rooms set up on campus," says Cooper. "Services guys are in one room figuring out what to say to customers. A security response team is in the other room trying to figure out how to repackage the patches and do technical damage control.

"I'm on a cell phone, and there's a guy there running me between the two rooms." Cooper laughs at the thought of it.

Repeat Mistakes

As the volume and complexity of software increases, so does the volume and complexity of patches. The problem with this, says SEI's Hernan, is that there's nothing standard about the patch infrastructure or managing the onslaught of patches.

There are no standard naming conventions for patches; vulnerability disclosure comes from whatever competitive vendor can get the news out there first (which creates another issue around whether vendors are hyping minor vulnerabilities in order to associate themselves with the discovery of a vulnerability--yet another story for another day). Distribution might be automated or manual; and installation could be a double-click .exe file or a manual process.

Microsoft alone uses a hierarchy of eight different patching mechanisms (the company says it wants to reduce that number). But that only adds to more customer confusion.