By default, ESX hosts do not allow you to log in to the Service Console using the root user account via SSH. This is done for security purposes as the root account should generally not be used because it is a superuser account. It is possible to enable this by changing a configuration file, but this is not recommended. An alternative to this is to create a separate user account on the ESX Service Console that you could use to connect to it using SSH, and than use the su command (grants root privileges to a user) to elevate your privileges. To do this, follow the below steps:
1. First, create an account on the ESX host. This can be done in two ways, either by using the service console command line or by using the VMware Infrastructure Client (VI Client). To create an account using the command line, log into the ESX host as the root account and then type the following commands: Continued »
There are three new and useful tools for the virtualization administrator’s toolbox.
The first is the VMware vCenter Mobile Access Technology Preview. Thought not quite available yet, it bodes fairly well as it works from an iPhone. My only concern about this tool is about the security. Does it use pre-shared keys/certificates? Or should I only use it over a VPN that uses pre-shared keys/certificates? Very cool technology preview, but still a few questions about it. (Click the link above to get the preview.)
The second is also something for the iPhone. Continued »
VMware plans to start releasing VMworld 2008 and VMworld Europe 2009 sessions to the general public soon — for free. Currently only attendees, vExperts and anyone who purchased a paid subscription can access them. For now VMware plans to only release approximately 10 sessions for free, they may later decide to release more every month until VMworld 2009, at which point all sessions may be made available for free. If you can’t wait until then you can always buy a subscription to them for $699 that gets you access to all the sessions and some cool swag.
VMware asked me for input as to which sessions should be in the first batch. Trying to pick only 10 or so sessions was almost impossible. I ended up selecting a lot more than ten, because all of the ones I chose are great and should be released right away. In my opinion this is some of the best training you can get; there is so much great information in these sessions and there is much you can learn from the wisdom and experiences of others.
Most of the sessions come in three formats, a .pdf file of the presentation, an .mp3 file of the audio and an .flv flash video file. I prefer to download all the files locally to my computer so I can view them anytime and without an active Internet connection. The .pdf and .mp3 files are easy to download, just right-click the links and save them. To download the .flv files there are a few methods you can use, such as using a program that can download flash videos on a page such as Orbit, or if you know the URL for the videos (its not hard to figure out) you can disable your browser Flash plug-in, put the URL in and save the file to your PC. Once you have downloaded them you can play them locally using a Flash player like Swiff Player.
So here are my picks for some of the best sessions. Keep checking the VMworld website for the official announcement of the release.
VMworld 2008 sessions:
AD2764 Managing VMware with PowerShell
BC2215 Top Tips for VMware Consolidated Backup
BC3141 Understanding Options for Virtualized Disaster Recovery
EA2244 Virtualizing SQL Server Using VMware Infrastructure
EA2263 Deploying Exchange 2007 on VMware Infrastructure 3
EA2347 Citrix Presentation Server Virtualization in VI3 – Best Practices
EA2538 Using IBM WebSphere Family Products with VMware
EA2672 VMware is the Best Platform for Java Workloads
PO1323 Best Practices for Virtualizing Active Directory
PO1520 Managing VMware ESXi in the Datacenter
PO2061 VMware VirtualCenter 2.5 Database Best Practices
PO1944 Architecting and Managing your Storage Effectively with Virtual Infrastructure
PO2218 Everyday Usage of the RCLI
PO2841 Virtualization – The Big Picture
TA1401 Understanding Host and Guest Memory Usage and Other Memory Management Concepts
TA1405 VMotion Technical Deep Dive
TA1440 ESXtop for Advanced Users
TA2213 VMware Infrastructure 3 Storage: iSCSI Implementation and Best Practices
TA2375 Intepreting Performance Statistics in VI3
TA2550 ESX Server Best Practices for Performance
TA2554 VI Networking: Advanced Configurations and Troubleshooting
TA2668 VMware ESX Architectural Directions
TA2920 Overview of VMware Product Directions
TA3807 VirtualCenter Directions
VD3261 VDI versus Terminal Services
VI2389 Licensing for a Virtual World
VI2940 VMware ESXi: The Easiest Way to Get Started
VI2715 Making the Case: Selling Virtualization When ROI isn’t Enough
LAB05 VMware Infrastructure – Security Hardening and Best Practices (VMware VirtualCenter/ESX/ESXi)
LAB09 Scripting VMware Infrastructure: Automating, Integrating, and Extending VI
VMworld Europe 2009 sessions:
AP07 Virtualized Oracle Database Server Performance and Best Practices
DC07 What’s New in vCenter Server
DC14 Overview of 2009 VMware datacenter products
DC15 Hypervisor Competitive Differences: Beyond the Data Sheet
DC26 vStorage – Storage integration for the VDC-OS
TA12 Introducing VMware Converter 4.0: What’s New and Different
TA15 Protecting your vCenter Server with Server Heartbeat
TA17 End-to-End Disaster Recovery Approach with Automated SRM Failback
TA20 Cisco Nexus 1000V Technical Preview
LAB02 Securing the Virtual DataCenter with VMware vShield Zones
LAB12 vNetwork Distributed Switch Tech Preview : Cisco Nexus 1000v
People often ask about how to set up X clustering on Y operating system with virtual machines. X and Y could be Microsoft Cluster Server (MSCS) on Windows 2008, Global File System (GFS) on Linux, etc. In either case the answer is pretty straight forward, except when it comes to Windows 2008.
Windows 2008 clustering is not officially supported within VMware ESX. That will change eventually, but hasn’t changed in VMware ESX 3.5 Update 4 according to the release notes.
That said, other shared disk clusters do work quite well. The way to set them up regardless of operating system is to use the instructions within the Setup for Microsoft Cluster Service PDF provided by VMware.
“But this PDF is for MSCS, not RedHat, SuSe, or any form of Linux,” you protest.
I reply: This PDF is for any shared disk cluster regardless of operating system. When you set up a shared disk cluster within a VMware ESX host or hosts you are mainly concerned about the possible limitations, and how to set up the virtual hardware. The limitations are well covered within this PDF as is how to set up the virtual hardware.
When you go through this PDF you want to pay close attention to all examples on how to set up the virtual hardware. If you are working with a Linux clustering system, ignore the Microsoft specific setup but pay close attention to the the setup of the virtual hardware within the prose and ALL examples. The examples give much more information that the prose, specifically on how to hook up some of the virtual hardware and what is supported.
The PDF should be retitled “Setup of shared disk clusters: MSCS, RHEL, etc.”
When the 300 vExperts were announced last month I was curious about how many were located in my area. Well what started out as simple curiosity of the number of vExperts in my area grew to include all the other ones that I could find worldwide. I gathered this information from the vExperts that checked in on the special vExpert VMTN forum as well as the members that joined the LinkedIn group. Of the 300 I’ve only been able to identify slightly more than half of them, the rest either have not realized they were named a vExpert or have not acknowledged it in the forum or the LinkedIn group.
Could you be one of the missing vExperts?
There were some reports of the vExpert announcement email being categorized as spam due to the word “Congratulations” in the subject. As a result many vExperts did not realize they were awarded this honor right away. It’s possible that there are others out there that still do not realize it and a public directory of the 300 vExperts has not been published yet. Through my research I did find out that I was one of four vExperts from the Denver area; I thought I would share the rest of my findings so far as to the geographical makeup of the vExperts.
- United States: 89 vExperts
- Arkansas – 2
- Arizona – 1
- Colorado – 4
- California – 8
- District of Columbia – 4
- Florida – 3
- Georgia – 5
- Hawaii – 1
- Iowa – 1
- Illinois – 7
- Indiana – 3
- Kansas – 1
- Kentucky – 2
- Louisiana – 1
- Massachusetts – 3
- Maine – 1
- Michigan – 1
- Minnesota – 2
- Missouri – 3
- N. Carolina – 2
- Nebraska – 3
- New York – 5
- Ohio – 2
- Oklahoma – 1
- Oregon – 2
- Pennsylvania – 4
- S. Carolina – 1
- Tennessee – 1
- Texas – 7
- Virginia – 3
- Washington – 3
- Wisconsin – 2
- Netherlands – 14 vExperts
- United Kingdom – 11 vExperts
- Germany – 6 vExperts
- Canada – 4 vExperts
- Australia – 4 vExperts
- Belgium – 3 vExperts
- Denmark – 3 vExperts
- France – 3 vExperts
- Italy – 3 vExperts
- Russia – 3 vExperts
- Luxembourg – 2 vExperts
- Croatia – 1 vExpert
- Hungary – 1 vExpert
- India – 1 vExpert
- Israel – 1 vExpert
- Malaysia – 1 vExpert
- Norway – 1 vExpert
- Slovakia – 1 vExpert
- Spain – 1 vExpert
- Sweden – 1 vExpert
- Switzerland – 1 vExpert
That’s 155 vExperts so far that I know about. Be sure and check your spam folder, you may be a vExpert and not even know it. A large majority of the vExperts are VMware User Group organizers from all over the world. The rest are mainly composed of bloggers and VMware evangelists. If you are a vExpert be sure and introduce yourself in this thread in the VMTN private community, and join the LinkedIn group.
Consider yourself lucky if you’ve never gotten the VMware HA message: An error occurred during configuration of the HA Agent on the host. But if you have, you may know that the ways to fix the error are extremely limited. Here is a method that worked for me.
The current methods of troubleshooting this issue involve checking that the DNS is working properly, that the FT_HOSTS file in /etc/opt/vmware/aam is properly written for the hosts involved in your VMware Cluster, and disabling and re-enabling VMware HA within the VMware Cluster.
The new method assumes that the VMware HA configuration is somehow at fault. I began to think this was the case when I noticed that the /opt/vmware/aam/ha/VMap process was not terminating on a reset of VMware HA. This process, as seen from the output of ps ax issued from the service console command line interface, should not exist when VMware HA is disabled. However, in my configuration it did exist. I also noticed I had problems reestablishing VMware HA after a recent reboot of a server caused by a faulty UPS. DNS was working, FT_HOSTS looked correct, and disabling and re-enabling VMware HA did no good.
Here are the steps that I followed to fix it:
- Log in to the service console of your problem hosts and verify that VMware HA is disabled using: service vmware-aam stop
- Ensure there are no VMware HA processes running by using: ps ax | grep aam | grep -v grep
- If processes exist, kill them using the Process ID returned by the previous command (first column) as the PID: kill -9 PID
- Issue the following command via the service console including the parenthesis: (cd /etc/opt/vmware/aam; mkdir .old; mv * .old; mv .[a-z]* .old)
- Using the Virtual Infrastructure Client click on the Host, then the Summary tab, and then Reconfigure for VMware HA.
Viola, VMware HA restarts and works properly! This solution may be seen as overkill as it forces VMware HA to recreate all configuration files. I may have been able to just remove the .vmware_fdport file and also Reconfigure for VMware HA, but I did not try that option. I bring this possibility up as it is NOT there on my now-running VMware HA-enabled hosts.
Now I have what looks to be a fool proof way to get VMware HA to start back up and protect my investment.
Whenever I am troubleshooting a problem with my VMware ESX hosts I find myself frequently having to look up the location of various VMware Infrastructure 3 (VI3) log files. To try and make this easier for myself and others I’ve decided to put together a list of all the log files that you might use that reside on your ESX host servers and vCenter Servers.
VMware ESX host logs:
- /var/log/vmkernel – VMkernel log, records activities related to the virtual machines and ESX hosts. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension
- /var/log/vmkwarning – VMkernel Warnings log, records activities with the virtual machines, a subset of the vmkernel log and uses the same rotation scheme.
- /var/log/vmksummary – VMkernel Summary log, used to determine uptime and availability statistics for ESX hosts human-readable summary found in /var/log/vmksummary.txt.
- /var/log/vmware/hostd.log – ESX Server host agent log, contains information on the agent that manages and configures the ESX host and its virtual machines. (Search the file date/time stamps to find the log file it is currently outputting to, or open hostd.log which is linked to the current log file.)
- /var/log/vmware/esxcfg-firewall.log – ESX Firewall log, logs all firewall rule events. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.
- /var/log/vmware/esxcfg-boot.log – ESX Boot log, logs all ESX boot events. Rotated with a numeric extension, current log has no extension and the next newest one has a .1 extension.
- /var/log/vmware/esxupdate.log – ESX Update log, logs all updates done through the esxupdate tool.
- /var/log/messages – Service console log, contains all general log messages used to troubleshoot virtual machines or an ESX host.
- /var/log/vmware/webAccess – Web Access log, records information on Web-based access to an ESX host.
- /var/log/vmware/aam – High Availability (HA) log, logs information related to the High Availability service. In ESX 3.0.x these logs were in /opt/LGTOaam512/ instead.
- /var/log/secure – Authentication log, contains records of connections that require authentication, such as VMware daemons and actions initiated by the xinetd daemon.
- /var/log/vmware/vpxa.log – Vpxa log, contains information on the agent that communicates with vCenter Server. Search the file date/time stamps to find the log file it is currently outputting to or open vpxa.log file which is linked to the current log file.
Virtual machine logs:
There is only one log file for virtual machines on a host and it is located in the working directory on the VMFS volume for the VM (i.e /vmfs/volumes/storage1/VM1/). The current log file is always vmware.log and older log files are incremented numerically (i.e. vmware-5.log).
vCenter Server logs:
The primary log file used by vCenter Server is the vpxd-#.log file (the # sign represents a number) which is located in the %allusersprofile%\Application Data\VMware\VMware VirtualCenter\Logs directory on the vCenter Servers version 2.5. The %allusersprofile% variable on most systems is the C:\Document and Settings\All Users directory. On older VirtualCenter 2.0 servers this file was located in the C:\Windows\Temp directory. New log files are automatically created when the logs grow to a certain size and the number in their filename is incremented. You can see which one is currently being used by sorting by modified date or by checking the vpxd-index file, which lists the number of the log file that is currently in use.
In addition to the vCenter Server log there are also logs for some of the other components, including Update Manager, Capacity Planner and Converter Enterprise. The logs are located in the \logs subdirectory under the various product directories which are located in %allusersprofile%\Application Data\VMware\. Here’s the list of log files that you will typically find on vCenter Servers:
- C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter\Logs – vCenter Server main log files
- C:\Program Files\VMware\Infrastructure\VirtualCenter Server\tomcat\logs – Tomcat web server log files for the web access component of vCenter Server
- C:\Documents and Settings\All Users\Application Data\VMware\VMware Capacity Planner\Logs – Capacity Planner log files
- C:\Documents and Settings\All Users\Application Data\VMware\VMware Update Manager\Logs – Update Manager log files
- C:\Documents and Settings\All Users\Application Data\VMware\VMware Converter Enterprise\Logs – Converter Enterprise log files
The License Manager Server log file is typically named lmgrd.log file and is located in the Windows temp directory (i.e. C:\Windows\Temp). You can verify the location of this file by opening the VMware License Server Tools application and click on the Config Services tab and checking the path to the debug of file field.
I’d recommend printing out this blog post or bookmarking it so the next time you are troubleshooting a problem in your virtual environment you can quickly figure out which log to look at to help figure out the cause.
What comprises a VMware vExpert’s virtualization toolbox? I actually think there are several different layers of tools depending on the particular VMware virtualization administrators job role. Mine, for example, has a number of security tools that most people would not normally need, but I do. So below is a list of what comprises my virtualization toolbox.
- VMware‘s VI Toolkit w/Microsoft PowerShell
- VMware’s VI Perl Toolkit
- VMware’s Remote CLI
- VMware Update Manager Plug-in for VMware Infrastructure Client
- Andrew Kutz’s Storage VMotion VMware Infrastructure Client Plug-in
- Hyper9‘s Search Toolbar VMware Infrastructure Client Plug-in
- Tripwire Opscheck (a must for solving VMware VMotion issues)
- PuTTY (a must for accessing host via secure shell (SSH) from Window’s systems)
- PowerGUI (GUI for PowerShell and VI Toolkit)
- Snaphunter (hunt for and destroy snapshots)
- Snapalert (find and commit snapshots with Remote CLI support)
- Adobe Acrobat (for reading VMware whitepapers, hardware compatibility lists and other documents)
- Twhirl (for Twitter access)
- Firefox for access to the VMware Communities Forums and Google Search
- VMware Infrastructure Client for real time performance graphs
- VMware’s esxtop for details on server utilization
- Vizioncore‘s vFoglight
- Nagios for service monitoring
- VMware Virtual Infrastructure Client Mapping capability
- Veeam Reporter
- Alan Renouf’s (http://www.virtu-al.net) vDiagram
- VMware Consolidated Backup
- VMware Converter
- If only ESX, then one of either Vizioncore vRangerPro, PHD Technologies’ esXpress, or Veeam Backup
- If only ESXi, then Veeam Backup 3.0
- VMware’s vcbMounter
- VMware Capacity Planner
- VMware Converter 4.0
- Ultimate Boot CD for Darik’s Boot and Nuke utility (in ISO form)
- Tripwire ConfigCheck
- My own Virtual Security Management Suite (still in pre-beta with a hope to get it out soon!)
- Hardening Script from VMware Virtual Infrastructure Security: Securing ESX and the Virtual Environment, available first half of 2009.
- DISA STIG for ESX SRR
- Backtrack for Penetration Testing (in ISO form)
- Helix for Forensics (in ISO form)
- Nuclues Kernel Linux for Data Recovery
- Virtual Firewall (smoothwall, IPcop, etc.)
There you have it: the contents of my virtualization toolbox. I am sure there are many more that I have missed as I do not use them very much or have never used them. Which of these tools will prove to be the most useful in the future? I believe it will be the VI Toolkit but I will keep all these tools within the toolbox, cleaned, and ready for use.
Editor’s note: Readers, any suggestions for other helpful tools? Comment and add to the list.
Have you ever had VMs mysteriously restart and didn’t know why? Even after checking the VMs vmware.log file and the event log in vCenter Server, you could find no evidence of the restart or any potential problems. Well, the Virtual Machine Monitor (VMM) feature could be the culprit. VMM extends the High Availability (HA) feature to be able to detect guest OS failures by monitoring a heartbeat provided by VMware Tools. In case a guest OS failure is detected (i.e. Windows blue screen of death), the heartbeat would stop and the VM would be restarted on the same host. While this is a great feature to have, it can also be troublesome and cause VMs to restart even when there is not a problem with the guest OS.
I recently experienced this problem first hand when VMs were mysteriously resetting at times even though they were not experiencing problems. The culprit? VMM. As mentioned, when this happens it can be difficult to determine as the vCenter event log will not have an entry that VMM caused the VM to reset, virtual machine state alarms will not be triggered, and the vmware.log will have no mention that the event occurred. The only evidence that a VMM event took place will be in the /var/log/vmware/hostd.log file on the ESX host and will look similar to what follows.
[2009-03-20 04:44:35.252 'TaskManager' 3076453280 info] Task Created : haTask-512-vim.VirtualMachine.reset-47992 [2009-03-20 04:44:35.323 'ha-eventmgr' 3076453280 info] Event 8420 : Win2003-1 on esx1.xyz.com in ha-datacenter is reset [2009-03-20 04:44:35.323 'vm:/vmfs/volumes/48331160-05c64c5c-edf0-001e0bd8c708/Win2003-1/Win2003-1.vmx' 3076453280 info] State Transition (VM_STATE_ON -> VM_STATE_RESETTING)
Since this feature relies on monitoring heartbeats through VMware tools there is the possibility of certain events happening that cause the heartbeat to stop longer than the configured failure interval, thereby triggering a false positive and resetting the VM. One example of these types of events is an upgrade to VMware Tools on a VM which causes heartbeats to temporarily stop while the VM is being upgraded. For this reason VMware changed the VMM feature in vCenter Server 2.5 Update 4 to also monitor the VM’s disk and network activity. Therefore even if no heartbeats are received within the failure interval, the VM does not reset unless no disk or network activity is detected for a predetermined I/O stats interval. A VM guest OS that is truly locked up will typically not have any disk or network activity in addition to the loss of heartbeat. This added level of monitoring will help eliminate false positives and make this feature even better. Additionally you can change the failure detection interval to higher than the default of 30 seconds. This setting is located in the HA advanced options section as shown below.
So if you ever have a VM mysteriously reset itself and you have VMM enabled, be sure and check the /var/log/vmware/hostd.log file on the ESX host to see if it was caused by VMM. It would be very helpful if VMware could make it so that these types of events are also logged in the vCenter Server events view.
This week I went through out annual audit process for the fourth time, and as usual the virtual hosts were mostly ignored. Why? The Payment Card Industry (PCI) security standards have yet to be updated to include the virtualization layer of environments that are audited for PCI compliance. The auditor acknowledged this fact and said at some point in the future this may eventually change so virtual hosts are more closely scrutinized.
The purpose of PCI standards is to ensure that IT environments meet a set of security standards to ensure the protection of card holder data and are a requirement for any companies that take credit cards and have a certain transaction volume.
VMware announced their participation in the PCI Council last year but so far nothing has come out of it. As to how this will affect a new PCI standard, this is anyone’s guess. The auditor I spoke with suggested that new regulations might require segregation of hosts so you do not mix development and test virtual machines (VMs) on the same hosts as productions VMs. Many large environments already separate their test and production VMs, but smaller environments that have a limited number of hosts may find this difficult. New regulations may also require further segregation so hosts that have VMs that are involved in the processing or storage of credit card data are isolated from other hosts.
Whatever comes out of VMware’s participation in the PCI council, we should finally see virtualization covered in the next update. Currently, PCI specification is at version 1.2, and was last updated in October 2008. This participation is critical to the success of the PCI standard, as applying security standards to VMs means nothing if you don’t also apply them to the hosts that the VMs reside on.
The lifecycle process document for the next version of the PCI specification indicates that the next release is not due out until 2010. Since virtualization will be added to the PCI specification in the future it would be wise for an administrator to start getting ready for the upcoming changes today. This would include assessing your virtual environment to identify hosts and VMs that are involved in credit card data, planning on how you might segregate your environment, reading through the published security standards for ESX hosts (i.e. CIS Security Benchmark, VMware’s hecurity hardening guide) and applying them and using virtualization specific security tools to monitor and secure your virtual environment.
VMware has a new Compliance Center on their website that includes white papers and presentations specifically on virtualization and PCI compliance. By doing this now you can be better prepared when the PCI standard is updated and be ahead of the game as often times you will find yourself scrambling to remediate your environment once the new PCI standard is in place and you are audited.
Editor’s note: For more information on how current auditing fails to address virtualized servers, read about how three separate auditing firms failed to address several security vulnerabilities involving virtualized servers in Virtual machine security enters the mainstream.