In the case of Slammer, Microsoft built three more patches in 2002--MS02-043 in August, MS02-056 in early October and MS02-061 in mid-October--for related SQL Server vulnerabilities. MS02-056 updated ssnetlib.dll to a newer version; otherwise, all of the patches played together nicely.

Then, on October 30, Microsoft released Q317748, a non security hot fix for SQL Server. Q317748 repaired a performance-degrading memory leak. But the team that built it had used an old, vulnerable version of ssnetlib.dll. When Q317748 was installed, it could overwrite the secure version of the file and thus make that server as vulnerable to a worm like Slammer as one that had never been patched.

"As bad as software can be, at least when a company develops a product, it looks at it holistically," says SEI's Hernan. "It's given the attention of senior developers and architects, and if quality metrics exist, that's when they're used."

And then there are patches.

Patch writing is appropriated to entry-level maintenance programmers, says Hernan. They fix problems where they're found. They have no authority to look for recurrences or to audit code. And the patch coders face severe time constraints--remember there's a footrace on. They don't have time to communicate with other groups writing other patches that might conflict with theirs. (Not that they're set up to communicate. Russ Cooper, who manages NT Bugtraq, the Windows vulnerability mailing list, says companies often divide maintenance byproduct group and let them develop their own tools and strategies for patching.) There's little, if any, testing of patches by the vendors that create them.