To the rescue was my handy iPhone and the TouchTerm application that I downloaded for free. TouchTerm provides an SSH client for accessing a remote SSH server. The application even allows the use of pre-shared keys, which alleviates the major security concern when using SSH and other SSL-based codes.

Using TouchTerm I was able to access my SSH server and then my ESX hosts service consoles to fix the security problem that occurred.

Another item I am looking forward to adding to my Virtualization Toolbox is Virtualization Manager Mobile from Andrew Kutz and H9Labs. I had a great discussion with Andrew when I was on vacation and some new features coming to VMM look very promising. I have been looking for a secure mobile virtualization manager and hopefully it will be here soon.

For now, TouchTerm adds a much-needed mobile access capability to my Virtualization Toolbox. Bonus: it also supports vSphere.

1. First, create an account on the ESX host. This can be done in two ways, either by using the service console command line or by using the VMware Infrastructure Client (VI Client). To create an account using the command line, log into the ESX host as the root account and then type the following commands:
useradd – creates the user account
passwd – sets the password for the user account
To create an account using the VI Client, you need to connect directly to the ESX host with the VI Client (not  vCenter Server) and log in is the root user. Next click on the Users & Groups tab, right-click inside the users pane and select Add. Enter a log-in name and password and check the Grant Shell Access To This User box. If you want to enter a descriptive name you can enter one in the username field; this name is not used for logging in. The UID field will automatically populate when you save the account. Once you are done, click the OK button and that’s it.

2. Now that we have our new account set up, we are ready to start using it. Let’s connect to the ESX host with a client like Veeam’s FastSCP, which allows you to elevate your privilege’s using su after you connect to a host. Run FastSCP and select Add a Server, enter the IP address of the server and select the ESX host option. At the Connection Settings screen enter the username/password for the new user that you created in the previous step. In the bottom section select the Elevate Account to Root option and enter the root user account password. At the Web Service credentials screen you can use the root user and password, as this is not used for connecting to SSH.

3. Once you connect to the host, you are using the user account you created earlier instead of the root account.

Please resist the urge to enable root SSH logins on your ESX hosts set up separate accounts for this purpose instead. It’s a security best practice to not use the root account for anything. Instead, use the su command or set up sudo; I’ll cover both methods in an upcoming tip on SearchVMware.com.

A VM with a raw disk or raw disk map (RDM) allows the guest OS to write directly to the LUN or disk partition within a LUN assigned to it, utilizing a pass-through mechanism. Using a raw disk or RDP won’t result in a noticeable performance gain, but there is often an improvement in management. In general, when a VMDK grows past a certain size set by the administrator, the administrator will opt to use a raw disk or RDM.

With VMware ESX v2.5 and ESX v3.0 it took a few simple VM configuration file edits to enable a raw disk when using local storage. Unfortunately, those methods no longer work on VMware ESX v3.5. There is, however, a solution. It is not very elegant, but it will work.

The solution is to use vmkfstools to import or copy a virtual machine disk file to the LUN or partition to use for the raw device. Here is an example that I just tested and seems to work. First, I needed to find out which device held the LUN I was going to assign to my VM.

1. Run fdisk -l to find the LUN.

# fdisk -l

Disk /dev/cciss/c0d0: 146.8 GB, 146807930880 bytes
255 heads, 63 sectors/track, 17848 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot    Start       End    Blocks   Id  System
/dev/cciss/c0d0p1   *         1        13    104391   83  Linux
/dev/cciss/c0d0p2            14       650   5116702+  83  Linux
/dev/cciss/c0d0p3           651      1287   5116702+  83  Linux
/dev/cciss/c0d0p4          1288     17848 133026232+   f  Win95 Ext'd (LBA)
/dev/cciss/c0d0p5          1288      1924   5116671   83  Linux
/dev/cciss/c0d0p6          1925      2561   5116671   83  Linux
/dev/cciss/c0d0p7          2562      3198   5116671   83  Linux
/dev/cciss/c0d0p8          3199      3453   2048256   82  Linux swap
/dev/cciss/c0d0p9          3454     17835 115523383+  fb  Unknown
/dev/cciss/c0d0p10        17836     17848    104391   fc  Unknown

Disk /dev/cciss/c0d1: 440.4 GB, 440430842880 bytes
255 heads, 63 sectors/track, 53546 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/cciss/c0d1 doesn't contain a valid partition table

2. Run esxcfg-vmhbadevs to find the vmhba device associated with the LUN.

# esxcfg-vmhbadevs
vmhba0:0:0     /dev/cciss/c0d0
vmhba0:1:0     /dev/cciss/c0d1

3. Create the VM with a standard virtual disk (VMDK).

4. Use vmkfstools from the command-line interface to import the VMDK into the vmhba device associated with the LUN using a disk target of RAW. Kill the import after you hit 1%.

# vmkfstools -i OpenFiler.vmdk -d raw:/vmfs/devices/disks/vmhba0:1:0:0 OpenFiler_1.vmdk
Destination disk format: raw disk out of '/vmfs/devices/disks/vmhba0:1:0:0'
Cloning disk 'OpenFiler.vmdk'...
Clone: 1% done.<ctrl-C>

5. Review the resultant VMDK metadata file. Note that the size of the LUN is references. For size take 860216490 and multiply by 512 for 410 GBs.

# cat OpenFiler_1.vmdk
# Disk DescriptorFile
version=1
CID=c0699ec2
parentCID=ffffffff
createType="vmfsRaw"

# Extent description
RW 860216490 VMFSRAW "/vmfs/devices/disks/vmhba0:1:0:0"

# The Disk Data Base
#DDB

ddb.virtualHWVersion = "4"
ddb.uuid = "60 00 C2 97 5c a6 e9 59-f3 de ba f6 83 ed 15 73"
ddb.geometry.cylinders = "1044"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"

5. Using the VMware Infrastructure Client (VIC), add an existing disk into the VM or add the following lines to the VMX file. (Note that you want to use a virtual SCSI controller not already in use. Also, the VM you make these changes to should be powered off and perhaps even unregistered if you are doing this from the CLI. If you use the VIC add the device following the red arrow path in the diagram below.)

scsi1.present = "true"
scsi1.sharedBus = "none"
scsi1.virtualDev = "lsilogic"
scsi1:0.present = "true"
scsi1:0.fileName = "OpenFiler_1.vmdk"
scsi1:0.deviceType = "scsi-hardDisk"

Now you have a VM that can directly access the local LUN on your VMware ESX host. I imagine this will work the same way for ESXi.

The white paper mentioned the usual recommendations that deal with memory, CPU and disk I/O, but I was surprised to see a whole section on timekeeping (which we will talk about later). JVMs are often memory hogs depending on how you set your minimum and maximum JVM heap size and they also read and write very often. JVMs also tend to be very multi-threaded and the disk I/O will vary based on the type of applications that they are running. A summary of the best practices for each resource is below:

Memory

• As JVMs are very memory intensive, make sure your JVM has access to physical memory at all times by using memory reservations. If a JVM is forced to use its disk swap file (vswp) for memory on an over-committed host its performance will be affected. Set the memory reservation for a VM running a JVM equal to the amount of memory assigned to the VM. The VM won’t be able to utilize the transparent page sharing (TPS) feature to save memory on your host, but you save memory on a VM running a JVM anyway due to the its nature.
• Make sure to give your VM enough memory based on the maximum heap size of your JVM. JVM’s have a minimum and maximum heap size value and will quickly grow to their maximum size. If you do not have enough memory assigned to it then it will not be able to grow and performance will suffer. Check your max size and allocate another 512 MB for Linux virtual machines (VMs) and an additional 1 GB for Windows VMs.
• Use large memory pages if supported by the JVM and OS. See the following hyperlinked white paper for info on how to do this on the OS and how to enable in JVMs use the option –Xlp for IBM JVMs and –XX:+UselLargePages for Sun JVMs.

CPU

• Many JVMs will run well with one vCPU depending on how many garbage collection (GC) threads are running and may not benefit from using virtual symmetric multiprocessing. GC is the process that reclaims memory inside the JVM for objects that are no longer used. Tuning this can be tricky and relies on specific Java resource monitoring tools to see how often GCs are taking place. Check to see how many GC threads are running on your JVM and either adjust this to match the number of vCPUs in the VM or increase the number of vCPUs to match the number of GC threads. Often times its best to start with on vCPU and see how the application performs and then add another vCPU to see if it improves performance.

Disk

• You want to watch the disk I/O of your application running on the JVM for potential bottleneck issues.  A JVM that is waiting to write to disk won’t perform as well as it could.

Timekeeping

As mentioned previously, timekeeping is very important to a JVM. First you should make sure you sync the clock on your VM using either VMware Tools, W32Time or another network time protocol time source. What’s important here is the affect timer interrupts have on a JVM. Higher resolution timer interrupts cause more work to be done by ESX on behalf of the VM then lower resolution timer interrupts. The guest OS determines the timer interrupt of a VM. Most Linux guests allow you to configure the timer interrupt in the OS but Windows guests must rely on a JVM setting. Due to a weird bug in the JVM the –XX:+ForceTimeHighResolution option in the JVM actually has the opposite effect of lowering the time resolution.

For more information check out VMware’s white paper on Java virtual machines and be sure to check out the documents referenced at the end of it.

1. Please slow down a bit. Produce a quality bug-free product and not try to rush out new versions, features and functionality until they are ready. Stop with the experimental features and only put them in the finished product unless they are ready and you are going to fully support them. I know it’s almost impossible to produce 100% bug-free code, especially as your product code grows larger and larger in size, but please catch the major ones that can cause outages for your customers. If you can’t slow down, at least hire more QA personnel and do more public Betas so your customers can help you with this. You can’t afford another mishap like Microsoft is currently experiencing with their Zune music players.

2. On the release of VI4 (or vSphere as you now call it): This should be an exciting upgrade and further distance you from your competitors, but please don’t release it before it’s fully done, polished and tested. I can wait an extra month or two if necessary.

3. Please, no more product name changes. Enough is enough with the name changes! You’re just confusing your customers and complicating things. Instead, get your marketing department to do more to attract new customers, keep your current ones and fight all the HyperV vs. ESX misinformation that Microsoft releases. Also please leave ESX named ESX, I know your marketing department is probably itching to change it so something like vHypervisor but resist and leave it as ESX. (For those who don’t know ESX stands for Elastic Sky X which was the name used in the development of the original version.)

4. More competitive pricing. You have lots of competition now and the hypervisor is becoming commoditized. Giving away ESXi for free was a good start. Why not give ESX away for free also and sell all the advanced features as add-ons? You also have plenty of automation and management products that you can sell to complement it. Also, please reduce the price of Workstation. It’s too expensive for many. You’d probably sell a lot more if you reduced the price so it was close to the price of Fusion.

5. On VMworld presentations: Please go back to releasing these to non-attendees after the show ends as you did in previous years. Not everyone can afford to go to it and the information in the sessions would be valuable to both your current and potential customers. It’s to your benefit to educate your customers and provide as much information to them as possible. At the very least, allow people to purchase a subscription to the sessions so they can access them right away after the show ends.

6. Relax the VMware Certified Professional (VCP) certification requirements. I shouldn’t have to take a class to become a VCP, if I have the knowledge and experience to pass the VCP exam that should be enough. Many qualified people can’t afford to take a class just so they can take the test.

Well VMware, I hope 2009 is a very good year for you, I look forward to the release of vSphere and any other great things that you will deliver to us in the upcoming year.

Best,

A VMware aficionado

As stated within the VMware KB article 1000761 it is possible to restore ESX to identical hardware; however, you need to reinstall ESX first and restore the data you backed up while making changes to how the system boots, else the Universally Unique Identifier (UUID) written by the installation will not work anymore as you have overwritten the data from your backup.

This method will restore everything effectively to identical hardware, however if you want to use new hardware, perhaps with different PCI devices, then the restoration would fail to properly configure the new devices. It may even fail to properly configure NICs if there are any IRQ differences between the supposed identical hardware.

So in these cases you would have to at least verify the configuration and fix anything that was broken. This could lead to a set of unknowns from a security perspective. You are after all trusting the backup was restored properly and if it was not, then you could end up with security issues. So the verification step would have to be extremely well documented.

It is far easier to reinstall VMware ESX to the hardware and to use a either a installation document,  kickstart, or other type of script to configure all the devices for you using either the Remote CLI or the VMware ESX CLI.

When restoring VMware ESX or VMware ESXi the best tool to have will be very good installation documentation that is easy to follow and has graphics and text for every step of the configuration.  These documents could be reviewed for security concerns, and used to derive the scripts that could do the work for you.

Systems - What products are supported for installations for ESX and ESXi platforms
Storage and SAN – Allows searches for partners and their products for ESX and ESXi-based storage devices.
I/O devices – Has brand information for supported HBAs, RAID controllers, and SCSI adapters.
VMware View – Lists supported connecting devices to the new virtual desktop product.

This new hardware compatibility guide search website also has direct links to all of the relevant other configuration information. This includes resources on CPU configuration for VMotion, supported guest operating systems, as well as my trusty PDF documents that are available as a traditional download.

More information on the new tool can be found in the online help section of the hardware compatibility guide website.

• Any book or reference from the Virtualization bookshelf
• A subscription to any number of magazines or ezines references on the Virtualization bookshelf
• VMware Workstation or Fusion licenses (if they do not already have one) for virtualizing at home
• VMware merchandise (gift cards, apparel , golf balls, mugs, mouse pads, etc)  from The VMware Store

For those who want to purchase hardware for their virtualization administrator:

• Any system on the VMware ESX Systems HCL
• Memory upgrades for servers, workstation, or laptop
• Fully loaded HP 8730w laptop (Great for running ESX within VMware Workstation)
• LiveScribe Smartpen — never miss a note/conversation again
• The iPhone, for oh-so-many reasons

Enjoy the holidays and I hope this helps you shop for your virtualization administrator!

Top Virtualization Security Links

The following links are just a sample of what is at the aforementioned site and should be read in order for those interested in securing your VMware Virtual Infrastructure and unfamiliar with VMware ESX, at the same time these are great references for the experienced administrator.

• VMware Virtual Networking Concepts
• VMware ESX Server 3: 802.1Q VLAN Solutions
• Security Design of the VMware 3 Architecture
• DMZ Virtualization with VMware Infrastructure
• VMware Infrastructure 3 Hardening
• CISecurity VMware ESX Security Benchmark followed by the CISecurity Linux Benchmark
• DISA STIG (ESX STIG depends on the UNIX STIG)
  • VMware Communities thread on ESX_SRRSecure – Script to allow ESX to pass a DISA Security Readiness Review.
  • UNIX STIG
  • ESX STIG
• Proven Practice: VI3 Security Risk Assessment – Xtravirt.com
• Remote Authentication – Full/Partial AD Integration, Secure LDAP, NIS, …

It is recommended to read as each guide or benchmark as each covers things from a slightly different but useful perspective.

VMware VI:OPs has also launched the Top 100 Virtualization Security Questions. This can also be referenced from the Top Virtualization Security Links sites as well as other blogs of interest.

Why is this? When a snapshot is created, the original disk becomes read-only, and a separate delta file is created that contains all the disk changes that are made thereafter. The delta file does not contain an ongoing history or transaction log of all the changes to data on the disk, it simply updates disk blocks as they are changed. If a particular block is changed it is written to the delta file, but if that same block is changed again later on the existing block is simply updated with the new data and a new block is not written to the delta file.

For example, if you took a snapshot of a VM with a 10 GB virtual disk, that snapshot could never grow larger than 10 GB, although it might grow slightly larger if every single disk block was changed because of the extra overhead space included in the snapshot disk file. The initial snapshot starts out small (16 MB) and grows in 16 MB increments up to the maximum size of the original virtual disk as changes are made to it.

In most cases the snapshot will not grow as large as the original disk, because typically operating system and application files are not changed once they are installed and therefore those disk blocks are not changed. If you performed a disk defragment inside the operating system, however, this could quickly and easily grow the size of the snapshot as files are being moved around on the disk which results in them being rewritten in a new location and, subsequently, the disk blocks are updated accordingly.

Now this only applies to a single snapshot. It is possible for the combined disk space total of multiple snapshots to exceed the size of the original disk file. The reason for this is that previous snapshots become read-only when new ones are created. If a particular disk block was updated from a previous snapshot, it would be written as a new block in later snapshots. That same disk block could then exist in multiple snapshots, which could make the combined total of the snapshots greater then the original disk file.

Even though copies of a single disk block can exist in multiple snapshots when it comes time to delete the snapshots, only the latest disk block is written back to the original disk and all of the others are discarded. Likewise, if you revert to a particular snapshot, then the existing disk block is discarded if it has been updated since the snapshot that you are reverting to, and the disk block from that snapshot is used instead.

This may all sound a bit confusing but the moral of this story is that a single snapshot can never exceed the size of the original disk file. For more information on snapshots be sure and check out the three-part series that I wrote about them:

How VMware snapshots work (Pt. 1)
Deleting virtual machine snapshots without wasting disk space (Pt. 2)
Troubleshooting VMware snapshots (Pt. 3)