However while going through these conversations here’s something else to keep in mind.  Odds are your company has just started going through the 2012 budget process.  Now is the time to get some requests in to attend some conferences next year.  When talking to your bosses about conferences for 2012 don’t just request one conference, request them all.  The way that the budget process works is that your boss starts with a big number, and slowly hacks that number down to get to a number that his/her boss can approve.  If you request a single conference with a budget of say $4000 for ticket, hotel and flight and it’s time to reduce that line item, there’s only one place to go, $0.  Boom, no more conferences for the year.  However if you has a team of 3 people, and you all want to go to one conference request a budget line item of $96,000 (3 people * 8 conferences * $4000 each).  When asked for a list of the conferences be prepared to provide a list.

• SQL PASS
• VM World
• EMC World
• Dev Connections (There’s a couple of those)
• Build (This is similar to what PDC used to be)
• Tech Ed
• Oracle Open World

There are also other events that you can attend which may require some budget, some a little less, some a little more depending on the event.

• SQL PASS Rally
• SQL Excursions
• SQL Cruise
• Tech Cruise
• Oracle Cruise
• SQLskills Immersion Events

I’m sure that there are plenty of other events which you could find and attend.  If you start with a nice high budget as you go through the process you’ll probably end up with enough budget for a couple of people to hit a conference or two throughout the year.

Good luck getting through the budget process, and hopefully I’ll see you at some of these conferences.

Denny

The first session that I hit was “SQL Server on VMware – Architecting for Performance” which was a bit of a let down. The first half of the session was mostly a SQL Server consolidation 101 session, and a lot of the points the speaker talked about in solution design I didn’t agree with. Some examples include her recommendation to set max server memory at ~500 megs below the memory allocated to the VM. Personally I feel that the max server memory setting should be set about 2-4 gigs below the amount of memory allocated to the VM (depending on what other software is installed on the VM, how much SQLCLR is used, etc.). There were also recommendations to enable lock pages on all servers as well as to disable the ballon drivers which I didn’t agree with either.

The second session that I went two was “VNX Block Oriented Performance” which was a great session. During this 500 level session the speaker talked about the hardware layout of the new EMC VNX storage array, specifically exactly how much data can be pushed through each internal component of the VNX and VNXe storage arrays. I’m not going to put the numbers in this post as I want to get the deck downloaded from the EMC World website so i can double check all the numbers before I do. With all the info that the speaker was giving out there was no way I could type it all fast enough on my laptop, much less my iPad which I was using to take notes.

The third session that I went to had a crazy long title. In my notes I titled it as “Building a highly available enterprise data warehouse using a bunch of shit”. The session was all about using the EMC GreenPlum database to build a distributed data warehouse and levering some of of their other products like the VMAX, Data Domain, and SRDF replication to protect through DR and back up processes. If you don’t know what GreenPlum is, it is a very scalable data warehouse product which is based on the Progress SQL platform. The system is configured as a fully redundant system which is scaled out by adding in more x86 servers into the farm. The systems scales easily into the petabyte range and EMC says they have several customers with multiple-petabyte databases running within GreenPlum. The nice thing with GreenPlum is that it comes as a software package you can install on your own hardware but also as a preconfigured appliance as a full rack. The next version will allow you to chain multiple appliance racks together to create a massive GreenPlum appliance farm. It posts some pretty expressive data load and data query rates, and i would love to put it side by side with the Microsoft Parallel Data Warehouse and see how they stack up against each other as far as load times, and data processing times but I’m guessing that neuter company will loan me one of these massive devices for a few weeks.

After the sessions were done came the hard work, figuring out which parties to attend that night. I started with a dinner that my great VAR Ahead IT threw. From there i moved to the Emulex party and ended the night at Brocades party. Don’t get me wrong I wasn’t a perfect angel at these parties but i did take it easy as I do have to be able to do this all again next week at Tech Ed and there is the official EMC party tonight.

I’ve run out of time to blog as I’ve got to get to another sessions, so I’ll wrap up here. Sorry for any spelling errors and the lack of links. I’m using my iPad to write this and may not have caught all the problems.
Denny

There were also some fantastic sessions. This year EMC is doing something new and putting the sessions online through an iPad app as well as putting them on their Facebook page.

First Session

The first session that I went to today was listed as a Federated Live Migration hands on workshop which I thought was a VMware session on doing live migration between data centers, which sounded really cool.  Turns out it was a hands on lab about doing live migrations from a DMX to a new VMAX array.  This sounded pretty cool as well so I stuck through it.  The concept is pretty cool.  When you have an EMC DMX array and you purchase a new EMC VMAX array you can migrate the data from the old array to the new array with no down time.  There was about 10-15 minutes of slides then we moved onto the lab.  Sadly one of the two EMC DMX arrays which we were using for the lab had stopped accepting logins so half of us (including me) couldn’t do the lab.  We walked through the lab process, but other than that it was kind of a bust.

Second Session

The second session that I made it to was a little disappointing as well.  The session ended up being a little more basic than I personally was looking for.  This sessions was “What’s new in VNX operating environment”.  As the VNX has been released and is shipping I assumed that this would be pretty in depth and would go over the hardware changes between the CX4 and the VNX.  This however wasn’t the case.  The session did go over some of the new features of the VNX array such as the new EMC Storage Integrator which is a Windows MMC application which allows Windows admins to provision and manage storage from the server without needing a login to the EMC VNX array.  A similar plugin is also being released for VMware’s vCenter server.

Unlike the CX line of arrays the VNX supports being both a NAS and a SAN in a single unit.  The array is basically a SAN, with a NAS bolted on.  When a LUN is created you’ll be asked if you want it to be a block level LUN which would be a traditional LUN or if you want a file system volume which would be a NAS mount point.  When you create the NAS mount point a LUN is created which is then automatically mounted to the NAS and the NAS then formats the LUN.

They also talked about the FAST cache which is available within the VNX array.  This cache takes their flash drives and mirrors them in a RAID 10 mirror which is then used as a second stage cache.  As blocks are accessed if the same block is touched three times the block is then loaded into the FAST cache so that on the fourth request for the block the block is now loaded from the cache instead of the spinning disks.  Blocks can be moved into the FAST cache because of either reads or writes as the cache is writable.  You can add up to 2.1 TB of FAST cache per VNX array.  When using sequential workloads these won’t work well with FAST cache very well because the FAST cache doesn’t work with pre-fetched data.

The really cool thing about FAST cache for SQL Server databases is that all FAST cache work is done in 64k blocks the same IO size that SQL Server users.  The downside that I see about this is that during a reindex process you might see stale data loaded into the FAST cache as the 64k blocks are read and written back during an index rebuild process, especially is an update stats is then done which would be a third operation on the block causing the block to move into the FAST cache.  This will take me getting my hands on an array and doing a lot of testing to see how this works.

One thing which I thought was really interesting was a graph that was showed were EMC tested Microsoft Exchange on an iSCSI setup, a NFS setup, and a fiber channel setup.  In these setups the iSCSI traffic had the slowest reads of the three followed by NFS then fiber channel being the faster. For writes NFS was the slowest, iSCSI was next then fiber channel was the fastest.  For the IOPs throughput fiber channel had the most bandwidth, followed by NFS with iSCSI being the slowest.  (I don’t have the config of network speeds which were used in the tests.)

Check back tomorrow for more information about day 2.

Denny

I met some great people at the party tonight and I met up with some friends that I’ve met at EMC World during prior years.

This week looks to be a great lineup of sessions based on what I’ve seen in the session schedule.  I’ll be posting what I can from the sessions during the week.

EMC World 2011 is back in Las Vegas this week.  The official hotels are pretty pricey, if you are looking for a much cheaper place to stay on the strip, I’m at the Imperial Palace which isn’t anything fancy.  It’s a basic room and shower, surely nothing special but for $30 a night instead of the $250 a night that the official hotels want it’s a damn good deal.  The best part is, its only a 10 minute walk to the convention at the Venetian which is worth saving the $1000 in my mind.

Denny

As I can only go about 130 miles on a tank of gas, I’ll have to stop a couple of times to fill up each way.  I’ve created a Google map to show my route.  I’ll be driving from home to Barstow, then to State-line, NV then on to Vegas.  If you see a pretty blue motorcycle with a leather clad motorcycle along this route with a license plate that says “mrdenny” give me a wave.

If you aren’t driving in from California and are attending EMC World, I’ll see you in Vegas at the welcome reception.

Denny

We covered some great topics like how to try and stay impartial, and when trying to stay impartial just makes you look like a chump.

There was also a lot of hey great to see you, what are you guys working on, here’s what we are working on kind of conversations going on; but there was also a lot of technical discussion going on especially around the new VMAX product and where it fits into the EMC lineup.  Frankly I probably got almost as much from sitting in the bloggers lounge talking to people than I got from the sessions that I attended.

My next big event is TechEd 2010 which is next month and I’ll be doing as much Tweeting and blogging as I can during the event just like I did this week.  Between now and TechEd I’ll be at the Portland SQL Saturday up in Portland, OR.  I’ll hopefully be tweeting like crazy while I’m up there.

If I don’t see you in Portland, hopefully I’ll see you at TechEd.

Denny

P.S. Kris and I were able to do some touristy stuff today, I’ll try and get those photo’s posted tomorrow night after we land.  We’re going to try and hit the Samuel Adams Brewery tour tomorrow morning before our flight.

The first session that I want to recap here was the SAN meets NAS sessions that I attended.  One of the big takeaways from EMC World was the technology that EMC is putting into all of the mid-tier products.  This includes the EMC Celerra which is EMC’s Network Attached Storage product.  The Celerra is basically as EMC CLARiiON with no fiber ports and a NAS connector on it, with a lite version of NaviSphere Manager running on it (unless you get the gateway only, and you then present LUNs from another fiber channel storage platform).  What the FAST package lets you do is to have the hardware automatically move less used data from expensive storage to cheaper slower storage.  This allows you to keep the data online so that your users can access it, but the access times will be just a little bit slower.  Instead of having a 5ms response time it may have a 50ms response time, but just for the files which are older and haven’t been touched in a while.

When you setup FAST on your Celerra it defaults to 80% of the least active data will be moved to slower storage, but this number can be easily adjusted in the settings for the Celerra.  As part of the Celerra FAST package you can even move the least active data to another device such as a Centera or Atmos device.

The second session was a session on getting Exchange 2010 setup and running on VMware’s vSphere 4.0.

VMware has shown both in their lab and their own production environment that they can run about 8000 users per virtual exchange server when you assign 8 vCPUs and 48 Gigs of RAM to each virtual server.  Now for smaller virtual exchange servers VMware has these configuration guidelines.

Now do keep in mind that these numbers assume a 350 Meg mailbox size, so if you allow for a larger mailbox size than this you’ll need to adjust your numbers accordingly. Something to note that when putting the 8000 user Exchange server under load using Load Runner, only 2 vCPUs were running at 100%. While the recommended spec for an 8000 virtual user Exchange Server is an 8 vCPU virtual machine, a 4 vCPU virtual machine should work just fine.

One of the reasons that Exchange 2010 works so well in a virtual environment is because Microsoft has made some major changes in Exchange 2010 so that it requires a lot less disk IO that Exchange 2007. You’ll also notice that with Exchange 2010 (as well as older versions of Exchange and anything else really that has a well written database engine) that the more RAM you add to your virtual Exchange Server the less disk IO you’ll see as the Exchange Server is able to cache more and more data into RAM.

I spent a lot of the rest of the day milling around the exhibit hall looking at the products like the Dell and Cisco blade servers (which I’m looking at for our next VMware Server Expansion) as well as the new Dell 810 servers which can give you a quad socket server in a 2 U configuration. Both of which would give a much more dense solution than I’ve got now.

I also looked at the Xsigo virtual network/HBA configuration. It looks to be an kind of expensive solution to get into, but it would dramatically reduce the number of cables that I have to run to each server. Currently when I deploy a new VMware server I have to run 2 fiber cables and 13 Ethernet cables. This Xsigo solution would cut that down to two infiniband cables, each of which can carry 20 Gigs of bandwidth. This would reduce not only the cabling but the number of network switches that I have to buy (granted I’d be buying Xsigo modules instead). It’s an interesting solution, however Emulex has a couple of blog posts (“Candygram for Mongo” and “Where They At“) where they talk about the solution which make for some interesting reading.

Sadly in my walking around the show floor I managed to not win anything beyond the free t-shirts, pens, etc.  Hopefully next time.

Last night was the traditional Wednesday Night party called the Beantown Bash.  I’ve uploaded some pictures from the party (and some other stuff) to Flickr.

If you missed my other recaps…

EMC World Day -1

EMC World Day 0

EMC World Day 1

EMC World Day 2

There will be one more small wrap up tomorrow then that’s it until EMC World 2011.

Denny

The first session yesterday was by Bruce Zimmerman.  For those of you in the SQL Server community that are reading this Bruce is to the EMC Storage community as Bob Ward is to the SQL Server Community.  Bruce talks on EMC CLARiiON performance tuning every year at the 400-500 level.  Here are some of the highlights.

When using NaviSphere Analyzer to monitor the utilization of your array you may not be getting the correct picture.  Metrics such as utilization are not true measurements of the utilization but instead a calculation.  In the case of the utilization metric Analyzer simply looks at the utilization of the RAID Group that each Storage Processor (SP) is putting on the RAID Group  and the higher of the two numbers is reported.  If you have a single LUN on the RAID Group, or all the LUNs on the RAID Group are owned by a single SP this isn’t an issue as the other number will be 0, but if the LUNs on a single RAID Group are owned by both SPs and each SP is running the RAID Group to 40%, Analyzer will show a 40% load, instead of an 80% load on the RAID Group.

If you need to dump Analyzer data to a CSV file via the naviseccli command use the -archivedump switch.  (Someone asked me about this via twitter a while back, which is why I made sure to include it.)

If you monitor the performance of your Storage Processors you may see the CPU spike to 100% on regular intervals.  This interval will correspond to the data logging interval that you have set within NaviSphere Manager.  While this CPU spike may worry you, unless your normal CPU load on the Storage Processor is very high, this CPU spike will not effect your performance through put.  If you are concerned that it is affecting your performance throughput through the storage processors, try disabling the data collection for a period of time in the SP properties.

If you look at the NaviSphere properties for the array you’ll see two settings for data logging.  One for the background process, and one for live data capture.  If these settings are different then the data logging happens at the lower of the two intervals.  Most people should set both of these options to 300 seconds unless you need capture data more frequently than that for a specific reason.

Some improvements to the FLARE version 29 that you’ll notice is that the load placed on the storage processor by the data logging process has been reduced by about 80% which is a huge savings.  You’ll also notice that with Release 29 that when doing a non-disruptive update (NDU) the CPU on each storage processor has to be slow 65%.  In older versions the CPU load had to be below 50%.  This change was made because the amount of backround processes which the array is performing as background management processes can be about 7-8% (per SP) and these processes don’t fail over.

Another naviseccli trick is to include the -np flag for all your commands.  This will tell naviseccli not to poll the array for response information.  Now if you need to get back information from the array when you run your commands you’ll want to include this.  For example if you create a LUN and have the array assign the LUN id and you want to do something with the LUN id later in the script you’ll need to exclude the -np switch, however if you specify the LUN id and don’t care about the feedback including the -np flag will save the Storage Processor quite a bit of work as the CLI requests a good deal of information from the Storage Processor for each CLI command that is issued.

I also gathered a lot of information about VMware in other sessions yesterday.

I’m not sure if this was supposed to be released, but the next release of vSphere (aka ESX) will be in Q3 of 2010 and will be vSphere 4.1.  This next release has a lot of enhancements to ease administration and improve integration between ESX and the EMC storage arrays.  You can assume that all of these integrations between vSphere and EMC CLARiiON arrays will require FLARE release 30 which should also be coming out in Q3 2010.

The first improvement is the vStorage APIs.  This is a set of APIs within vSphere 4.1and the EMC arrays that allows the vCenter server or the vSphere server it self (if not running with a vCenter server) to talk to the array directly and perform some actions.

These actions include Bulk Zero Acceleration.  This allows the vSphere host to when creating a new file to tell the array to fill the file with 0s instead of having to transmit all those zeros to the array over the fibre or iSCSI.  This is done by the vSphere host writing a single block to the array with all 0s in it, then telling the array to replicate that block n number of times.  While this won’t reduce the amount of data that the array has to write, it will reduce your network traffic and because of this may safe time.  By default this feature will be enabled in vSphere 4.1, but can be disabled in the advanced settings page of the host.

Another feature are some hardware locking changes.  Currently when vSphere needs to take a lock on the LUN it locks the entire volume then performances it’s operation then releases the lock on the LUN.  In vSphere 4.1 it will be able to lock just the specific block on the disk that it wants to work with, then release just that block.  This will allow multiple hosts to take locks on the same LUN at the same time without having to wait in line to complete the operation.  There are a few places when this benefit will be seen including boot storms (where you’ve got lots of machines booting at the exact same time), and allow for more snapshoting to take place (as when each snapshot is created a lock has to be taken on the LUN when the new file is going to be created).  By default this feature will be enabled on vSphere 4.1, but can be disabled in the advanced settings page of the host.

The next feature is called Full Copy Acceleration.  This is a great feature which will reduce the amount of traffic between the array and the host when cloning a virtual machine.  Today when you clone a file the file is copied up from the array to the host, then written from the host back to the array in the new location.  With this feature enabled (which it is by default) the API will simply tell the array to copy the blocks which make up the file from one location to another preventing the entire file from being transferred from the array up to the host.  If your network between the array and the host is bandwidth limited this will reduce the time it takes to clone the virtual machine.

Of the new VMware Features which require array integration there is only one which doesn’t require FLARE 30 and that is the Stop and Resume feature, which requires FLARE 29 on the host.  This feature cleans up the way that the guest OSs see that a thin provisioning pool is out of space and the LUN can’t consume any additional space.  Prior to vSphere 4.1 (also known as today) if a thin LUN can’t be expanded as needed on the array because there isn’t any space the guest OS will throw (within Windows at least) a blue screen of death (BSOD) because the page that it’s requesting to write to isn’t available.  In vSphere 4.1 an error message will be thrown as a popup within the guest OS which effectively says that there was a problem writing to the disks.

Something which will be coming in Q2 of 2010 (so probably within the next 6 weeks or so) will be the CLARiiON Provisioning Plugin for vSphere.  This will let you provision a new LUN on the storage array, and attach it to the VMware Cluster from a single screen which should greatly decrease the amount of time required to provision and attach storage from the array to the server.

I’m curious to see how long it takes other storage vendors to get these APIs working on their arrays (with or without VMwares assistance).

Check back tomorrow for my Day 3 post.

Denny

The first session that I hit was the futures for the EMC CLARiiON’s FLARE software.  For those that don’t know what FLARE is, FLARE is the software which runs the array and handles all the functions of the Array.  The next release will be FLARE v30.  If you are a CX3 or older customer this new release will be of no use to use as this version only supports the CX4 array.

Some of the new features which are being included are a totally new management interface called Unisphere.  This new interface will give a single interface for your EMC CX arrays as well as your Celera devices and RecoverPoint.  Eventually other EMC products will be integrated into Unisphere with products such as Replication Manager coming hopefully in 2011.

Now don’t worry, you don’t need to go out and buy all new arrays to take advantage of Unisphere.  Unisphere will work with existing arrays back to FLARE 19 for the CX arrays and DART 6.0 for the Celerras which should cover most of the arrays out there.

Unisphere will give you a single management interface, replacing the slow NaviSphere tree that we are used to seeing with a tabbed interface. However once you get into the actual tasks the screens will look very similar to what we see today.  Now for those that can’t upgrade to FLARE 30, or you aren’t ready to upgrade to FLARE 30, there’s an off array version of Unisphere that you can install on a Windows server and use that as your management endpoint to get the benefits of Unisphere without the pain of upgrading your array.

Now sadly Unisphere is still Java based, but it is a total rewrite, and with some tongue and cheek answering we were told that not all the performance problems of the current NaviSphere were because of Java.  To me this says “Some of our code blows”.  It was probably as honest as an answer you can get with marketing people wandering around the building.

Something which is a little more exciting than a new UI is the new thick pool of LUNs.  When you create a LUN pool you’ll be able to create the LUN as a thin LUN or a thick LUN.  A thin LUN is just like it is today, the LUN is created as 2 Gigs then new space is allocated in 1 Gig chunks.  With a thick LUN the LUN is created from the disks in the pool as it’s full size.  Now where this comes into play is with the new FAST technology will monitor the 1 Gig chunks of the disk, and move them around within the array so that hot chunks are moved to faster storage, and cold chunks are moved to slower storage.  This is all done within the same pool.

A single pool of disks can have both thin and thick LUNs in it, and it can have disks of different sizes and speeds.  When you create a pool you aren’t limited to the 16 disk limit that you have for a RAID Group today.

Another nice thing about the LUN pool is that you can expand the LUNs without having to create meta-luns which reduces your overall LUN count allowing you to create more LUNs.

Now with pooled LUNs (both thick and thin) the 1 Gig chunks are made up of 8k blocks.  Within the 1 Gig chunk the 8k blocks are contiguous, but the 1 Gig chunks won’t be contiguous for a single LUN.  So you will have some fragmentation issues to deal with.

The last big thing that I learned about yesterday is that with FLARE 30, the CX4s will be able to use the Enterprise Flash Disks (EFDs aka SSDs) as cache.  You can take a RAID 10 of EFDs (only RAID 1 or RAID 10 is supported) and mark the RAID Group as Flash Cache.  This will allow you to have up to 2 TB of additional cache on the array which can be used as a read or read/write cache.  Now this RAID Group will be dedicated as a Flash Cache RAID Group so you can’t create LUNs on this RAID Group.  This Flash Cache technology works best when used on very bursty work loads.  Typically it only takes 2-4 disks to create a really good, large Flash Cache.

Now the downside to Flash Cache, is the cost.  These EFDs cost about $20k each from EMC, so that’s $40k-$80k in EFDs from EMC.  During the session the speaker said that this was supposed to be a less expensive was to increase your array’s cache without downing the array and increasing the DRAM.  Now I’m not rocket scientist, but if the disks are 400 Gigs each at $20k, and I put in two for Flash Cache, 800 Gigs of RAM (400 Gigs for each storage processor) will be a lot less expensive than the $40k for the flash drives.  Now granted you won’t be able to get 400 Gigs of RAM into a single SP, but I think that EMC Marketing might want to come up with a better message than this.

Hopefully you like some of these new features, and I’ll see you tomorrow.

Denny

After the welcome reception was the concert featuring the Counting Crows.  The Counting Crows put on a pretty good show, so far I’d have to say that the Bare Naked Ladies are still my favorite concert at EMC World so far.

I took a bunch of pictures at the party and registration which I’ve posted to Flickr.

Probably my favorite picture of Day 0 is this one of me with the walking Celerra.

I’ll try and post sessions daily about everything that I’ve seen though out the sessions.

Denny