So today was the first official day of EMC world. I had a great time meeting some fantastic people like Chuck Hollis from EMC.
There were also some fantastic sessions. This year EMC is doing something new and putting the sessions online through an iPad app as well as putting them on their Facebook page.
The first session that I went to today was listed as a Federated Live Migration hands on workshop which I thought was a VMware session on doing live migration between data centers, which sounded really cool. Turns out it was a hands on lab about doing live migrations from a DMX to a new VMAX array. This sounded pretty cool as well so I stuck through it. The concept is pretty cool. When you have an EMC DMX array and you purchase a new EMC VMAX array you can migrate the data from the old array to the new array with no down time. There was about 10-15 minutes of slides then we moved onto the lab. Sadly one of the two EMC DMX arrays which we were using for the lab had stopped accepting logins so half of us (including me) couldn’t do the lab. We walked through the lab process, but other than that it was kind of a bust.
The second session that I made it to was a little disappointing as well. The session ended up being a little more basic than I personally was looking for. This sessions was “What’s new in VNX operating environment”. As the VNX has been released and is shipping I assumed that this would be pretty in depth and would go over the hardware changes between the CX4 and the VNX. This however wasn’t the case. The session did go over some of the new features of the VNX array such as the new EMC Storage Integrator which is a Windows MMC application which allows Windows admins to provision and manage storage from the server without needing a login to the EMC VNX array. A similar plugin is also being released for VMware’s vCenter server.
Unlike the CX line of arrays the VNX supports being both a NAS and a SAN in a single unit. The array is basically a SAN, with a NAS bolted on. When a LUN is created you’ll be asked if you want it to be a block level LUN which would be a traditional LUN or if you want a file system volume which would be a NAS mount point. When you create the NAS mount point a LUN is created which is then automatically mounted to the NAS and the NAS then formats the LUN.
They also talked about the FAST cache which is available within the VNX array. This cache takes their flash drives and mirrors them in a RAID 10 mirror which is then used as a second stage cache. As blocks are accessed if the same block is touched three times the block is then loaded into the FAST cache so that on the fourth request for the block the block is now loaded from the cache instead of the spinning disks. Blocks can be moved into the FAST cache because of either reads or writes as the cache is writable. You can add up to 2.1 TB of FAST cache per VNX array. When using sequential workloads these won’t work well with FAST cache very well because the FAST cache doesn’t work with pre-fetched data.
The really cool thing about FAST cache for SQL Server databases is that all FAST cache work is done in 64k blocks the same IO size that SQL Server users. The downside that I see about this is that during a reindex process you might see stale data loaded into the FAST cache as the 64k blocks are read and written back during an index rebuild process, especially is an update stats is then done which would be a third operation on the block causing the block to move into the FAST cache. This will take me getting my hands on an array and doing a lot of testing to see how this works.
One thing which I thought was really interesting was a graph that was showed were EMC tested Microsoft Exchange on an iSCSI setup, a NFS setup, and a fiber channel setup. In these setups the iSCSI traffic had the slowest reads of the three followed by NFS then fiber channel being the faster. For writes NFS was the slowest, iSCSI was next then fiber channel was the fastest. For the IOPs throughput fiber channel had the most bandwidth, followed by NFS with iSCSI being the slowest. (I don’t have the config of network speeds which were used in the tests.)
Check back tomorrow for more information about day 2.
So today began EMC World 2011. It started in true EMC style with a great party, sponsored by UniSys. The party was held at the Hotel pool which seems a little strange but EMC made it work. It was a little windy last night, apparently not even EMC can control the weather.
I met some great people at the party tonight and I met up with some friends that I’ve met at EMC World during prior years.
This week looks to be a great lineup of sessions based on what I’ve seen in the session schedule. I’ll be posting what I can from the sessions during the week.
EMC World 2011 is back in Las Vegas this week. The official hotels are pretty pricey, if you are looking for a much cheaper place to stay on the strip, I’m at the Imperial Palace which isn’t anything fancy. It’s a basic room and shower, surely nothing special but for $30 a night instead of the $250 a night that the official hotels want it’s a damn good deal. The best part is, its only a 10 minute walk to the convention at the Venetian which is worth saving the $1000 in my mind.
Enterprise environment are moving more and more towards service oriented functional groups. SAN as a service, database as a service, etc. So SAN guys are saying DBAs shouldn’t worry about this, they use gigantic arrays with 100’s of spindles, they guarantee performance, etc. What are your thoughts on this?
I would say that the SAN admins are nuts if they think that the DBAs aren’t going to worry about storage performance. There is no magic SAN dust which keeps everything running fast all of the time. If you put enough load on the array then performance will be impacted, no matter how many disks there are in the array. The physics say that eventually the little disks can only spin so quickly before they break apart.
In my opinion if they tell you not to worry about performance, then they aren’t worried about the performance, which means that they aren’t going their job.
So this year EMC World is back in Las Vegas, and I’m damn happy that I’m going again. Instead of taking the quick flight from Ontario to Las Vegas like I normally word, I’ll be getting there a little differently, I’ll be riding my motorcycle.
As I can only go about 130 miles on a tank of gas, I’ll have to stop a couple of times to fill up each way. I’ve created a Google map to show my route. I’ll be driving from home to Barstow, then to State-line, NV then on to Vegas. If you see a pretty blue motorcycle with a leather clad motorcycle along this route with a license plate that says “mrdenny” give me a wave.
If you aren’t driving in from California and are attending EMC World, I’ll see you in Vegas at the welcome reception.
If i have 3Par SAN, is there any specific recommendations, as it does its custom stripping across the drives. or any other specifics related to 3PAR?
The biggest recommendation that I have (pretty much the only one since I haven’t worked with a 3PAR in a while) is to ensure that your LUNs are sized correctly so that they are evenly spread out across all the disks in the disk set that you are using.
For example, if you have 100 disks in the disk set (I don’t remember the 3PAR term here), then you have 50 disks available (as the other 50 are the mirror). If you have a 1 Gig chunk size you want to ensure that all your LUNs are divisible by 50 Gigs. The basic math is ChunkSize*ActiveDisks then multiple the number you get until you get to the first size larger than the size that you need. You will end up wasting a little bit of space, but you will have a faster array.
We get many warnings in our SQL error log about I/O requests taking longer than 15 seconds to complete. What do we do in such scenario?
There are a few things which you will want to look at. First make sure that you have enough RAM for the database to use. Without enough RAM in the SQL Server you’ll begin putting extra load on the disks because the same data is being pulled into the buffer cache from the disk over and over.
The second thing to check is that you don’t need to add any additional indexes. If your queries are doing scan’s which they shouldn’t be then this will put extra load on the disks.
If everything else looks good it’s time to get the boss to open their checkbook and buy more disks, or faster disks.
Is there a best practice to advise a storage admin how to set a SAN up best for SQL Server OLTP & OLAP? or is this entirely company dependent?
There are some general best practices such as separate data and log LUNs, separate TempDB from everything else. For data warehouses make sure that the data is spread over as many disks as possible. Anything more specific however is very company specific and would require a good understanding of what the databases look like as well as what the storage array looks like.
For SQL, best practice is to have data on one array, logs on another array and tempdb on another array. how does this work in san world? SAN people don’t really give all these options when assigning storage for database server.
Even when using a storage array it is typically best practice to have the data on one LUN, logs on another LUN, and tempdb on another LUN. Especially if you have the option to putting each LUN on different physical disks. If you I talked a little more about this a couple of weeks ago, so check there more some more detail.
By default SQL Server Management Studio will only show you 256 characters per column. The fix for this is pretty easy. You click on Tools > Options to being up the options menu. From there navigate to Query Results > SQL Server > Results To Text. Change the “Maximum number of characters displayed in each column” value from 256 to what ever number you want up to 8192. After you click OK the setting will be saved.
Here’s where the trick comes in. The setting doesn’t actually take effect until you restart SQL Server Management Studio. There is no prompt to restart, but a restart is required because apparently SQL Server Management Studio only reads this setting on start up.
Thanks Manageability Team.
So I’m going along doing my work thing, shrinking databases rebooting servers at random, etc. and I get an email some of which I’ve included below asking if I’d be interested in this job opening. I’ve cut out the names, etc. to protect the stupid.
Below is the response that I sent back.
Industry Confidential Location Palo Alto, CA Job Title SQL Server DBA Duration 03 month(s)(Possible Extension) Profile/ Job description
- The SQL Database Administrator is part of a data center team providing application and infrastructure support for a large healthcare engagement. The SQL Database Administrator is responsible for managing and monitoring database platforms which support multiple OLTP and OLAP applications. THE SQL Database Administrator will also be responsible for installing and configuring a high availability, and disaster recovery environments and planning upgrades and patches as required to maintain each environment.
Specific responsibilities will include:
- Working experience with SLA’s
- Monitor software database license process
- Monitor performance activity
- Familiar with Clustered environment and load balancing when required
- Hands on experience with Mirroring, replication and recovery models
- Hands on experience on performance monitoring and database counters
- Manage technical action reports to resolve bugs/issues with DBMS software
- Support capacity planning
- Participate in implementing database strategies and solutions
- Provide database maintenance activities
- Monitor database logs for potential problems
- Participate in planning software upgrades and patches
- Provide outage resolution support and perform analysis to resolve recurring database issues
- Install and configure DBMS
- Provide database support for software development/application projects
- Verify database backups have completed successfully
- 2+ years Database Administrator experience
- 2+ years SQL 2000, 2005, 2010
- 2+ years MCITP – Microsoft SQL 2008 Database Administrator
- 2+ years MCDBA 2003/2005
- Adaptive Server Enterprise Administrator professional certification
- MCITP: SQL Server 2008 Database Administrator
- MCDBA 2003/2005 Certification
Professional Skill Requirements
- Operations Management
- Project Management
- Service Level Agreement (SLA) Management
- Problem Solving and Escalation Management
- Service Performance Management and Metrics
- Estimation and Planning
Travel/Shift Hours/OT Requirements Minimal travel.
- Work location is Palo Alto and candidates must live within a 100 mile radius of client site locations. Will require shift work and on-call duty and off-hours and weekend work as needed, as this project is a 24 x 7 support team.
- Detail Oriented
- Quality and Process Improvement
- Microsoft Office Proficient
If you would like to pursue this position, please reply to this e-mail and provide us details in following areas. Also, please attach a copy of your updated resume, in word format, elaborating your projects in tune with the client’s requirement. This will assist us in placing you with one of our clients.
Here’s what I sent back.
You should probably work with your client to get some actual requirements.
If you are going to list the Industry as “Confidential” you probably shouldn’t mention it in the job description.
There is no such thing as SQL 2010.
There is no certification as the MCDBA 2003.
There is no certification as the MCDBA 2005.
You want someone with way more than 2 years’ experience installing high availability and disaster recovery servers.
People with 2 years’ experience won’t have experience with mirroring or replication. They probably won’t have much experience with server monitoring or looking at performance monitor counters (there is no such thing as database counters). Someone with 2 years’ experience will have no idea how to open tickets with Microsoft, or how to work the Microsoft CSS support team to get the problems resolved. The odds of someone with 2 years’ experience being able to correctly identify a new bug in the database engine is next to 0 (if not lower than 0).
It is barely possible for people to have the SQL Server 2008 certifications for two years. They were only released a little over two years ago. If someone only has two years’ experience with SQL Server, they probably aren’t ready to take the certification exams, much less the MCITP.
You are asking for a Sybase (Adaptive Server Enterprise) certification, yet you never mention Sybase anywhere in the job description.
Please learn at least something about the position you send out so that you don’t look like a total moron. Based on this job description I would NEVER allow you to represent me on any job posting. Open job positions are not high priority. Production servers being offline, that’s a high priority email. The government closing, that’s a high priority email. Family member is sick and in the hospital, that’s a high priority email. This is not.
And yes, this email was sent as a high priority email.