If you’ve upgraded to the latest release of ESX and ESXi which is version 3.5 Update 2, you might have woken up to a nasty surprise today. This thread in the VMTN forums has identified a major bug in the latest version of ESX that was released on 7/25/2008.
Users that have upgraded to this version have been reporting that they can no longer start or VMotion virtual machines running on ESX 3.5 Update 2 servers. Virtual machines that are already running are not effected by this bug. The error that is displayed in the VMware Infrastructure Client (VI Client) reports that a general system error has occurred. Going through the host log file displays the true error that the product has expired. The only known workaround at this point is to disable NTP on the ESX host if it is enabled and then set the ESX host server’s clock back to a date before 8/12/2008.
VMware support is aware of the issue as they are apparently getting swamped with calls because of it. A knowledgebase article has been posted confirming the bug and currently lists the resolution as:
The issue is found with ESX Server 3.5 U2, ESX Server 3i version 3.5 Update 2 Installable, and ESX Server 3i version 3.5 Update 2 Embedded. Currently there is no workaround for the issue and VMware engineering is actively working on resolving this issue. This KB article will be updated as soon as more information is available, check back frequently for updates and additions.
In addition this response from a VMware employee in the VMTN forum thread confirms that they are aware of the problem and will be issueing a fix soon:
An issue has been uncovered with ESX/ESXi 3.5 Update 2 that causes the product license to expire on August 12. VMware engineering has isolated the root cause of this issue and will reissue the various upgrade media including the ESX 3.5 Update 2 ISO, ESXi 3.5 Update 2 ISO, ESX 3.5 Update 2 upgrade tar and zip files in the next 36 hours (by noon, August 13, PST). They will be available from the page: http://www.vmware.com/download/vi. Until then, we advise against upgrading to ESX/ESXi 3.5 Update 2. The Update patch bundles will be released separately later in the week. The issue is being tracked on KB 1006716 on http://kb.vmware.com/ We sincerely apologize for any inconvenience that has been caused.
A response from another VMware employee also stressed the seriousness of the situation:
Everyone is mobilized here at VMware. mjlin, who posted in this thread several hours ago, is the product manager. Support knows what is going on. Someone else has posted our first communication here on this thread (patch should be available within 36 hours). I know we’re preparing additional communication, so check that kb and expect more from us as we have more information.
With a bug of this magnitude a critical fix will probably be released as soon as possible to resolve it. One could guess that this issue was caused by code that was put into the beta version that was set to expire on a certain date and was never removed for the final release. This is commonly done with beta software to ensure that nobody can use it past a certain date when the gold version has been released and is known as “time bombing.” Unfortunately for VMware the timing of this bug couldn’t be worse, they released ESXi for free with this version and many people downloaded it to try it out. Now those people will be receiving error messages that they may not understand and since there is no support for the free version of ESXi it could cause them to discontinue use of the product.
This is really a big black eye for VMware as they struggle to compete in a market that has become increasingly competitive lately. While VMware has been actively trying to release new features and versions as quickly as possible to try stay ahead of the competition, their product quality has not been what it used to be. Too many changes are being made to the product code and not enough testing is being done to ensure they release a quality product.
Until a fix is released it is advised that you disable the DRS feature and avoid using VMotion. If you must power on a virtual machine, you can disable the NTP service on your ESX host and set the clock back. Then watch for a fix to be released so you can apply it to all your ESX 3.5 Update 2 hosts.