Microsoft fell victim to a software vulnerability in one of its own products on Saturday, when the W32.Slammer worm infested host machines on the company's network, flooding that network with traffic.
The company's travails with Slammer late last Friday night and Saturday morning were first revealed through internal e-mail messages obtained by news agencies and reported on Monday.
A Microsoft spokesman confirmed that the Slammer worm penetrated the company's network defenses and infected a number of SQL Server databases and desktop machines.
"There were circumstances where we were not patched," said Rick Miller, a spokesman for Microsoft.
The vulnerable machines were mostly in the company's Redmond campus and concentrated in an area of Microsoft's network used by SQL Server developers, according to Miller.
In some cases, the vulnerable machines were purposely left unpatched to try to recreate specific environments for testing purposes, Miller said.
Miller said that "a high percentage" of the SQL Server hosts used by customers were properly patched and unaffected by Slammer.
Slammer temporarily interrupted the company's Windows XP activation service, but the activation server was not vulnerable. Instead, the service was brought down by a flood of Slammer-related traffic from hosts on the same subnet, Miller said.
Given the size of the company, security experts weren't surprised that some machines on Microsoft's network were vulnerable.
"It's not surprising when you consider that most people working at Microsoft are (software) developers and that a lot of development software installs MSDE (Microsoft SQL Desktop Engine)," said David Litchfield, managing director of Next Generation Security Software Ltd. and the person who discovered the SQL Server vulnerability exploited by Slammer.
Miller confirmed that infections linked to the MSDE component were a part of the company's problem, but declined to say how many servers and desktops were affected or how much of the problem stemmed from desktops with MSDE installed.
Many have taken Microsoft's inability to properly patch SQL Servers on its own network as proof that the current system of releasing software patches is flawed.
"I do feel it's an unreal expectation to think that system administrators can monitor multiple applications and apply patches to them that vary from implementation to implementation," said Geoff Shively, chief hacking officer of PivX Solutions LLC.
The volume of software patches from Microsoft and other software vendors and the need to test patches before deploying them combine to overwhelm system administrators, leaving them to take their chances with a new worm or virus. That makes companies like Microsoft vulnerable, Shively said.
"I think that people need to be more vigilant, but some people know there's a security vulnerability and still don't patch," Litchfield said.
The Slammer infestation shows that Microsoft is not immune to that problem.
"We struggle with the same problems as the rest of the industry," Miller said. "Individuals make patch management decisions for reasons of their own. Sometimes it's a time management issue and sometimes it's oversight -- particular developers not doing what they needed to do."
In light of the Slammer outbreak, Microsoft will be re-evaluating its internal patch management policies.
"The only thing I can say is that we've learned from this. The status quo is not acceptable and going forward we're going to be looking at procedures and doing things better," Miller said.
As part of its Trustworthy Computing initiative, Microsoft is looking into ways to streamline the patch management process.
"We feel that (patching) is critically important and recognize that there are problems with patch management," Miller said.
Changes in the existing system for deploying software patches could introduce as many problems as they solve, according to Litchfield.
"If I'm running a 911 emergency system, I don't want my computer calling Microsoft and downloading a patch that breaks something. That could end up killing people," Litchfield said.
For the foreseeable future, the world -- including Microsoft -- will have to get by with the current system of manually downloading, testing and installing software patches, according to Litchfield.
In the meantime, the revelation that Microsoft's own servers were not properly patched does not undermine the company's credibility in encouraging its customer to promptly patch their systems, according to Miller.
"In our mind, what's critically important is that everybody patch their systems. The biggest lesson with this worm is that if you don't patch you're gonna get hit," Miller said.