Just minutes after VMware issued a patch late Tuesday that allowed enterprises to start up crippled virtual servers, the company's CEO said he was sorry for the problem.
"I want to apologise for the disruption and difficulty this issue may have caused to our customers and our partners," Paul Maritz, the head of VMware, said in an open letter posted to the company's web site.
Tuesday, reports flooded VMware's support forums from users unable to power-up virtual servers that had been updated to the newest software, which was released in late July. Users said they were seeing error messages claiming that the virtualisation software's license had expired as of 12 August.
Maritz, who was appointed CEO about a month ago, confirmed that VMware developers had left code in recent ESX 3.5 and ESXi Server 3.5 updates that prevented users from powering up virtual machines when the calendar flipped to 12 August. The code, he said, was left over from beta versions of the update.
"We failed in two areas," said Maritz. "[We failed in] not disabling the code in the final release of Update 2 and not catching it in our quality assurance process."
Although it's common practice for developers to code a time limit into their betas in order to force users to upgrade to the final version, the deadline -- often called a "time bomb" -- is supposed to be removed before the final version is released.
"We have kicked off a comprehensive, in-depth review of our QA and release processes, and will quickly make the needed changes," said Maritz.
Tuesday night, about two hours later than original scheduled, VMware issued a patch for the problem. The fix, available in separate versions for ESX 3.5 and ESXi Server 3.5, works, according to reports by users to VMware's support forum.
"I updated one of my five hosts and everything appears to be fine so far," said a user named Robert Greenlee on a long support message thread. "Thanks for all the hard work getting this patch out," he continued. "Unfortunately, I think you've gotten a serious black eye today. We were finally getting management happy with the idea of using ESX for production servers and this set us back a little bit."
Greenlee wasn't the only user on the support forum taking shots at VMware for the bug, which left virtual machines unusable for more than 20 hours.
VMware also quashed rumors that the time bomb was anything but a failure of its quality control process. In an entry posted to a company blog Tuesday night, the company denied that the problem was security-related.
"The ESX(i) 3.5 Update 2 power on problem that surfaced today is not related to exploitation of a security issue on ESX," said VMware. "Several customers have been worried that their ESX systems had been compromised by an attack and that this was the cause for not booting of their ESX update 2 Virtual Machines today."