Posted by: Eric Siebert
Eric Siebert, VMware ESX
VMware has announced the availability of a patch to fix the date bug that was reported earlier yesterday. They have also released a letter from their CEO, Paul Maritz, explaining the problem and apologizing for it. From VMware support:
Dear VMware Customer,
The express patches are now available for download that will resolve the ESX/ESXi 3.5 Update 2 issue which causes the product license to expire as of August 12, 2008. Please go to http://www.vmware.com/go/esxexpresspatches for more information.
The VMware ESX Product Team
The express patch page displays the following information, be sure and carefully read through the KB articles that apply to either ESX or ESXi before download the new versions. The ESXi patch is about 204 MB in size and the ESX patch is 104 MB. The patches do not require a reboot of the ESX host but it must be in maintenance mode and all virtual machines must be shut down or moved to other hosts before it can be applied. The suggested steps for applying the patch as posted by one user in the VMTN forums are:
1) Turn off the NTP client on ESX 3.5 U2 server that you are going to VMotion VMs to
2) Make sure VM’s tools do not have time synchronization checked between the virtual machine and ESX server operation system
3) Change the date on ESX server that you are going to VMotion your VMs to
4) VMotion the VMs to the ESX server with the date changed
5) Apply the patch to the ESX 3.5 U2 server that the VMs have been moved off of
6) VMotion the VM’s back to the patched ESX 3.5 U2 server
7) Patch and change the date back on the other ESX 3.5 U2 server.
Express Patch Download
Special Notice: Please Read
An issue has been uncovered with ESX/ESXi 3.5 Update 2 that causes the product license to expire on August 12, 2008.
Follow the steps below to correct this issue:
1. Read the following Knowledge Base articles first:
* Fix of virtual machine power on failure issue, refer to KB 1006716
* For VI 3.5, refer to KB 1006721 for deployment consideration and instruction
* For VI3.5i, refer to KB 1006670 for deployment consideration and instruction
2. Download and apply the express patch according to the product(s) you have:
* VMware ESXi 3.5 Update 2 Express Patch
* VMware ESX 3.5 Update 2 Express Patch
The full product downloads are still currently unavailable and are expected to be re-released as a new build number shortly.
Earlier VMware also released this update that provides some more information on the issue and answers to frequently asked questions.
Dear VMware Customers,
Please find the latest update about the product expiration issue. We are staging the express patches and expect it to complete in an hour. When the staging is done, we will send out a communication with more details.
Please see FAQ 1) for details about these express patches. In FAQ 2), we describe what upgrade media and update patch bundles to be release later are for. These are the updates since our last communication.
Complete information on the ESX/ESXi 3.5 Update 2 issue follows:
An issue has been discovered by many VMware customers and partners with ESX/ESXi 3.5 Update 2 where Virtual Machines fail to power on or VMotion successfully. This problem began to occur on August 12, 2008 for customers that had upgraded to ESX 3.5 Update 2. The problem is caused by a build timeout that was mistakenly left enabled for the release build.
The following message is displayed in the vmware.log file for the virtual machine:
This product has expired. Be sure that your host machine’s date and time are set correctly.
There is a more recent version available at the VMware web site: http://www.vmware.com/info?id=4.
Module License Power on failed.
•- VMware ESX 3.5 Update 2 & ESXi 3.5 Update 2.
* – The problem will be seen ifESX350-200806201-UG is applied to a system.
* – No other VMware products are affected.
*What has been done?*
* – VMware removed the ESX 3.5 Update and ESXi 3.5 Update 2 binaries from the download pages in the evening of August 11, 2008 PST.
* – VMware Engineering teams have isolated the cause of the problem and are working around the clock to deliver updated builds and patches for impacted customers.
* – A Knowledgebase article has been published (http://kb.vmware.com/kb/1006716) and is being refreshed regularly.
VMware Engineering has produced express patches for impacted customers that will resolve the issue.
1. What will the express patches do?
There are two express patches: one for ESX 3.5 Update 2 and one for ESXi 3.5 Update 2. They are specifically targeted for customers who have installed or fully upgraded to ESX/ESXi 3.5 Update 2 or who have applied the ESX350-200806201-UG patch to ESX/ESXi 3.5 or ESX/ESX 3.5 Update 1 hosts. For customers who haven’t done either, these express patches should not be applied.
To be noted is that these patches have been validated to work with esxupdate. However, testing with the VMware Update Manager is still under way. In subsequent communications, we will provide confirmation whether the patches work with VMware Update Manger or if a re-spin is required.
To apply the patches, no reboot of ESX/ESXi hosts is required. One can VMotion off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs need to be powered off before the patches are applied and powered back on afterwards.
We are currently testing an option to apply the patch without requiring VMotion or VM power-off and re-power-on at the point of patch application. To immediately refresh vmx on the VM, one can VMotoin off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs can be powered off before the patches are applied and powered back on afterwards.
2. When will VMware reissue the upgrade media and patch bundles?
VMware plans to reissue upgrade media by 6pm, August 13 PST and all update patch bundles later in the week. We will provide an ETA for the update patch bundles subsequently.\
* An upgrade media refers to ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files. They are for customers who haven’t installed or upgraded to ESX/ESXi 3.5 Update 2 but wish to.
* The “patch bundles” here refer to those released at GA. They are for customers who do not wish to do a full upgrade to ESX/ESXi 3.5 Update 2, but apply patches that are deemed necessary to hosts running ESX/ESXi 3.5 or ESX/ESXi 3.5 Update 1. They are not the same as the express patch which is described above.
3. Why does VMware plan to reissue the upgrade media before the patch bundles?
Since we can complete building and testing of the upgrade media before the patch bundles, we want to make that available to customers right away instead of reissuing all the binaries later in the week.
4. Can VMware issue a patch that opens the licensing backdoor in the next hour as a critical measure?
There is no licensing backdoor in our code.
5. Does this issue affect VC 2.5 Update 2?