Virtualization

Red Hat Enterprise Virtualization (RHEV) Management

Found this story while surfing the net, seems that the management software for Red Hat’s virtualization software needs to be run from a Windows Server, they say that they will have a Linux version ready around 2010, but at the time of this story and for a little while anyways, you will need some sort of Windows box to managed the virtual environment. Read the insert below and then check out the full story.

“As a major Linux vendor, one might expect that Red Hat’s new Red Hat Enterprise Virtualization (RHEV) for Servers solution would be able to run on Linux servers.

You’d be wrong. Not only is that not the case, but the Management Server piece of RHEV, which provides virtualization management capabilities, requires users to be running Microsoft’s Windows Server.

That’s no typo: A Linux vendor is requiring its users to run one of its key new products on the rival, closed source Windows operating system.”

Here is the link to the full article.

-NS

vSwitch and vSwif Problems

What happens when you cannot connect to the service console anymore, and even if you had a backup service console and it wasn’t working either, well i happened to me. When I attempted to attach a vmnic to a vSwitch, I got an error message, no matter what I tried I could not get it reconnected. Last ditch effort is to recreate the vswif interface. Below are the steps I took. I had to do the first 4 steps twice, as i had to vSwitches.

Note: On your system the vswif, vmnic, vSwitch numbers and network settings are different.

1.Run the following command to list the name of the vswif adapter:

esxcfg-vswif -l

2.Run the following command to delete the vswif adapter:

esxcfg-vswif –del vswif0

3.Run the following command to list the name of the vSwitch:

esxcfg-vswitch -l

4.Run the following command to delete the vSwitch:

esxcfg-vswitch -d vSwitch0

5.Run the following command to create the vSwitch:

esxcfg-vswitch -a vSwitch0

6.Run the following commands to create default port groups for vSwitch:

esxcfg-vswitch -A “VM Network” vSwitch0
esxcfg-vswitch -A “Service Console” vSwitch0

7.Run the following command to create the vswif adapter:

esxcfg-vswif –add –portgroup “Service Console” –ip=nnn.nnn.nnn.nnn –netmask=255.255.255.0 vswif0

8.Run the following command to verify that the settings in the network file are correct:

cat /etc/sysconfig/network

NETWORKING=yes
GATEWAYDEV=vswif0
HOSTNAME=host.domain.com
GATEWAY=nnn.nnn.nnn.nnn

9.Run the following commands to list all of the network adapters and associate a vmnic which has a link status of up:

esxcfg-nics -l
esxcfg-vswitch -L vmnic1 vSwitch0

10.Run the following command to verify that the vmnic is associated with the vSwitch:

esxcfg-vswitch -l

11.Ping an IP address to check for network connectivity. If the ping fails remove the previous vmnic from the vSwitch and try another adapter that has a link status of up.

12.Once you have verified connectivity, you can then connect to Virtual Infrastructure client of the affected ESX Server, you may or may not have a bit a work ahead of you at this point, depending on iSCSI, etc…

Good Luck

-NS

How to Properly kill a VM

We all know that you can use the command “ps -ef |grep (vmconfig filename.vmx)” to find the PID for a VM, but I get confused as to which number that comes up is actually the PID… I found a new command that works much better. See below.

The Definitive way to Kill a PID process

The steps below are based on the fact that you already tried to use “vmware-cmd vmname.vmx stop; stop hard”

1.)    A final solution is to kill the PID (process ID). Issue the following command: “ps auxfww | grep <vmname>” to locate the correct PID (BTW: this cannot be done via *ESXTOP*). The first number to appear in the output is your PID. The PID can be used to terminate the process by issuing kill -9 PID.

2.)    Run vmware-cmd <vmname>.vmx getstate to see if it is now “off”, as it should be.

3.)    You can now go back into Virtual Infrastructure Client and start the machine.

-NS

VMware Vroom!

If you haven’t subscribed to this RSS Feed, you should do so now… it includes Performance Tips for ESX and vSphere. Here is the link and read the insert below.

“We made a huge number of performance improvements in vSphere 4.0. The ESX storage stack was no exception. We ran a wide variety of micro and real world benchmarks to thoroughly evaluate and optimize vSphere’s storage subsystem. It is now even more efficient for the enterprise and ready to support the cloud.

A wide variety of I/O intensive applications will run efficiently on vSphere with all the improvements.  You can find details on the architectural changes and storage performance improvements made in this white paper.”

I suggest you add the feed to your BlackBerries(Viigo) or iPhone immediately, yes now… not later, now.

NS

vSphere and Datastore Monitoring

I just wanted to point out an article that I found while searching about vSphere and Datastore management and monitoring.

What would be better than being warned that your datastore is reaching a critical point, wouldn’t it be nice to intercept before the VMs crash?! Of course it would. Check out this pdf “Storage with VMware vSphere“, I have included an insert below.

“vSphere enables many new features for the management of virtualization storage resources. These capabilities provide vSphere administrators with new ways to increase efficiency for disk utilization and CPU cycles. With automated proactive alerts and alarms, vCenter provides more visibility and control of storage resource allocations, deployment, and monitoring. Also, new levels of reporting, topology maps, and datastore status and details screens provide an easy way to manage access and limit storage sprawl in VMware deployments. Storage VMotion has been enhanced to enable more choices for migrating VM homes from one type of storage to another. And finally, new choices exist for optimizing I/O traffic for I/O intensive Virtual Machines.”

This next link gives you a quick insight into what vSphere Offers in terms of Data Management overall, including “Responsible Thin Provisioning”.

Of course if you just cant get your hands on vSphere 4, then you can purchase Veeam Monitoring, which can monitor datastores and alert you when the datastores are almost full… see insert and check out the Veeam Monitor Homepage

“Veeam Monitor 4.0 provides full storage monitoring capabilities, including disk space, disk issues and datastore monitoring. The latter allows for viewing datastore load information aggregated from disk utilization statistics of all VMs and hosts using the specific datastore. This allows you to see the actual cumulative load on a particular datastore, and set up an alarm that fires off when the datastore load is above a specified threshold. Datastore I/O monitoring is a unique Veeam Monitor feature not available from other VMware monitoring applications.”
-NS

Understanding VMware Snapshots.

I wanted to clear the air on snapshots, everyone is always asking, how much space does it take to remove a snapshot, but no one has a real answer…  whatever the answer may be this information inserted from the Basic System Administration PDF, which is readily available from VMware and can be downloaded from this link here, will help to explain how they work, and what can be done if you run out of space and cannot commit a snapshot. Please take your time and read through the below information and you will get a much better understanding of snapshots. The information below is a copyright of VMware. I have in no way altered its content.

“The Understanding Snapshots section does not include information on delta disks. The section should contain the following content:

To take a snapshot, the state of the virtual disk at the time of taking the snapshot must be preserved. When this occurs, the guest operating system cannot write to the VMDK file. The delta disk is an additional VMDK file where the guest is given write access. The delta disk represents the difference between the current state of the virtual disk and the state at the time of the previous snapshot. If more than one snapshots exist, delta disks might represent the difference (or delta) between each snapshot. Also, the guest can write to every single block of the virtual disk causing the delta disk to grow as large as the base VMDK of the virtual machine.

NOTE To consolidate all snapshots into the base virtual machine, you might need extra disk space, as large as the base VMDK.

When a snapshot is deleted, if a user chooses to merge the changes between the snapshots to the previous disk‐state, all the data from the delta disk that contains the information about the deleted snapshot is written to the parent disk. This might involve a large amount of disk I/O and might reduce the virtual machine performance until consolidation is complete.

If the user chooses to ignore the delta disks, delta consolidation is not required.

See VMware Knowledge Base system for more information on the iterative snapshot deletion behaviour. I’ve Included the details of the (KB article 1003302).

Details

If you try to initiate a Delete All snapshot for a virtual machine using Snapshot Manager, and if that virtual machine is on a datastore that does not have sufficient space for the snapshot, the following message displays in VMware Infrastructure (VI) Client:

msg.hbacommon.outofspace: there is no more space for the redo log of <VMname>-0000xx.vmdk.

You are given the option to abort or retry.

• If you choose Abort, the virtual machine is powered off, the snapshot is aborted, and a Consolidate Helper snapshot is created. The Snapshot Manager UI displays that Consolidate Helper snapshot. You can delete the Consolidate Helper snapshot after you have made space available.
• If you click Retry, the Snapshot Manager returns to Consolidate Helper snapshot mode unless you have made more disk space available.

Solution

Free up disk space if possible, or extend the VMFS volume using VI Client.

To extend the VMFS volume:

1. Select the host on which the virtual machine resides and click the Configuration tab.
2. Select the datastore on which the virtual machine resides and click Properties.Note: If there is no available storage, a new LUN must be presented to every ESX host that can see the LUN.

3. In the dialog that appears, click Add Extent and follow the prompts in the Add Extend wizard to add an extent.
4. Perform a rescan on every ESX host that is being presented the new LUN so that the addition of the extent is detected.
5. After you have extended the VMFS volume, you can check the Retry option of the Redo log pop-up.

Caution: When using Delete All in the Snapshot Manager, the snapshot furthest from the base disk is committed to its parent, causing that parent snapshot to grow. When that commit is complete, that snapshot is removed and the process starts over on the newly updated snapshot to its parent. This continues until every snapshot has been committed. This can lead to an aggressive use of additional disk space if the snapshots are large. Use care when exercising this option if there is not much space available on the datastore.”

Veeam Monitor 4.5 + Veeam Business View

… Are now  is now available!

For the first time ever, you can see the impact of virtual infrastructure performance on individual business units…and take immediate action! Veeam’s newest release now allows you to bridge the gap between IT and business:

- Group, view, and manage virtual machines based on business criteria such as business unit, location, application (e.g. all Exchange servers), service level agreement (SLA), owner, etc.

- Create flexible alerts based on server or application type (e.g all Exchange servers) and business impact (e.g. all high SLA VMs)

- Identify the business impact of the virtual infrastructure’s performance

- Fight VM sprawl

- Discover how easy it is to see your business in a brand new way with Veeam Monitor 4.5 plus Veeam Business View.

Go Here to download Veeam Monitor and Veeam Business View

DRS, Vmotion, and Snapshots

I think setting DRS(Distributed Resource Scheduling) to manual mode when removing snapshots, should be best practice, especially when you are only running two ESX servers and they are slightly taxed. Why? Mainly because when you remove a snapshot, especially a two tiered one, it can cause the hard disk to “thrash” and thus signaling DRS to Vmotion a Virtual Machine from one ESX Server to the other ESX Server automatically. This will of course cause more stress on the hardware and eventually the client may notice the decrease in performance. Hey I’m just speaking from personal experience. When the snapshot is removed, re-enable DRS. Comments?

-NS

Openfiler, Free and Opensource

I am back in the books after being un-prepared for VCP-310 and was needing some iSCSI practice, well i dont have an MSA or HP All-In-One laying around so i improvised. My buddy Raj ( he has a blog on here also) pointed me to a program called Openfiler. I was able to create a VM on VMware Workstation, install Openfiler, and then create an iSCSI target for my ESX Box. Worked like a charm. Since I had never used this product before, I used this Petri Article, “Use OpenFiler as your Free VMware ESX SAN Server”. Read through it all, its a 2 part post.

Just wanted to share my awesome experience of configuring an iSCSI target out of a laptop and virtualization components!

NS

VMware 3.5 and Virtual Infrastruture Client 2.5 Ports

Check this link out, it has a neat port guide in PDF format with all the ports mapped and labelled. Its a visual representation of the ports below.

22. SSH access to service console.
80. HTTP access to web servers.
443. HTTPS SSL access to web servers for VI Web Access.
902. VI Client access to ESX Server or VirtualCenter, and communication between VirtualCenter and ESX Server hosts or between hosts.
903. VM Console access via VI Client or VI Web Access.
2049. Connection to NFS storage devices.
2050-5000, and 8042-8045. Traffic between ESX Server hosts for VMware HA (also utilizes EMC Automated Availability Manager).
3260. Connection to iSCSI storage devices.
8000. Incoming requests from VMotion.
8083. VirtualCenter diagnostics port.
8086. Apache Tomcat web server admin on VirtualCenter Server
27000. License transactions from ESX Server to the License Server.
27010. License transactions from the License Server.