July, 2009

The After Hours Compensation Conundrum

Being in IT, most of us have accepted the fact that after hours work is a part of the job.  Usually after hours work is divided into two camps: scheduled maintenance or support.

Scheduled maintenance would be, for most of us, a part of the job.  If you’re a systems administrator then you need to perform hardware upgrades, software patches and other late night activities when all the users are home and snug in their beds.  Well, not the accountants.  Odd folk…I hear they just sit with their briefcases and wait until morning comes.

Support is what it is.  Something has worked for ages and now it’s broken.  Maybe someone locks themselves out of Active Directory at 2AM because he’s got caps lock turned on.  The IT person who has the beeper (they still make those, don’t they?) or after hours phone needs to either get online and remotely fix the problem or in a worst case scenario drive to the office.

About 2 years ago our department was required to start carrying an after hours cell phone.   Our template is a flat yearly rate for all on call staff.  There’s no formula (i.e. $X for each weekday and $Y for each weekend/holiday).  It’s just a flat number the company felt was fair.  As well, any time we spend answering after hours calls could be applied as “vacation time” sometime afterward.

At first it wasn’t too bad, but after a while some issues came up.  I did a little research and found that there’s a lot of common ground in the tech community.  Here’s some of those problems:

• Banking after hours time spent as “comp” time - It’s a great idea in theory, however it’s probably the lowest form of “vacation” time.
  • People still send emails so the company supplied blackberry is buzzing like any normal work day.
  • The blackberry phone number is published in the corporate directory so users don’t mind calling it if they can’t reach you at your desk.  If they call your office phone and hear the voice mail click in, chances are they will hit the # key and leave a message expecting a callback or hang up before hearing your “out of office” message.
  • Comp time doesn’t take precedence over a staff member officially on vacation.  If you get 3 support calls on a Sunday night and need to take a few hours to recuperate, you’re still expected to be available in some capacity if the other IT people are on vacation.  The person taking comp time in this scenario will always be called before the people on official vacation.

• Even if you’re taking vacation, you’re still expected to be available in case you have the skills or the answers to keep the business going.
  • Documentation of critical systems and procedures can only go so far.  Appendectomies are fully documented, but an experienced surgeon removing an appendix would do a far better job than a drug rep with a manual.

• If you’re not on call you can still be called by the person who is on call
  • Again, you may have an answer to keep the business going.

• No form of monetary compensation is worth being tethered to the office 24/7/365
  • I’ve heard people who get upwards of $10-12k for beeper pay alone who swear they’d give it up in order to put their 50 hours a week in and go home without the beeper
  • Sadly, some people aren’t being compensated other than being able to take comp time.  It’s part of the job.  However, everyone cuts their own deal when they take a job.

• A stock departmental policy may seem unfair
  • Is the home life of a senior analyst with 10 years experience, a spouse and kids more valuable than a junior analyst with 2 years experience and is single?
  • Is the senior analyst much more effective in solving after hours issues than the junior analyst?
  • Is it fair to pay the senior analyst $75k/year and the junior analyst $35k/year, while giving them both the same compensation for after hours support?

So what’s fair?  I’ve mentioned the fact that everyone cuts their own deal.  But how much is it worth for you to be tethered to your office?  How long do you think you could live with that deal?

I’d love to see your comments about this topic.

Outside of the compensation issue, here’s a great way to cut down on after hours calls:

Invoice the department that calls you after hours. The guy on the 24/7 production floor will start to check his caps lock the 1st time he keys in an invalid password once his department manager starts getting the quarterly bill from the IT folks.

Firefox 3.5 - Buckle up your pacemakers

Yikes!  How did this ever get released?

I downloaded Firefox 3.5 last week and I’ve been irritated with the slow speed ever since.  I’ve even used IE a couple of times out of frustration.

One of the major problems is that Firefox is pulling data from various locations on each system for randomized security reasons.  I’m not sure exactly when this started and I’m not a browser security expert, but this is where most of the finger pointing has been directed.  Click here for the bug info.

Evidently until this issue is resolved, Mozilla is expecting people to do some old school system cleanup steps (clearing temporary internet files, cookies, yada, yada, yada).  Personally, I’m about to use Chrome or something else until this nonsense is put to bed.

Here’s a link to some of the workarounds.

If you’re not a fan of workarounds, then here’s 2 solutions until you want to head back to ‘Ol Faithful:

1. If you don’t have Firefox 3.5, keep your browser
2. If you have Firefox 3.5, revert back to 3.0 or find a new browser

Friday Night Lights

I had a lovely Friday, working from home in complete silence until my little boy woke up.  I then went to the office and shut my door for a few hours, banging out the finishing touches of a project due at the end of the day.

Around 3PM we get word that there’s going to be a scheduled power outage at 3:30 for repairs.  Evidently there were a couple of explosions earlier in the day and some of the electrical equipment got toasted.  The outage would last between 5 hours and 3 days.  Big window of an estimate, but what can you do?  The poor electricians were probably airing the smoke out of the substation and assessing the problem.

We have a 4 person IT shop.  A manager, myself and two other technicians.  As luck would have it, my 2 week after hours on-call shift started Friday, the manager is on vacation and the two other technicians started their vacation Friday after work.  Guess who gets to hang out after work to make sure things are in good order with the equipment?

At 3:30 the power shuts off and our UPS handles the power load for our servers for about a minute or so until the propane generators fire up.  At 3:32 the propane generator kicks in like clockwork and runs along for all of 5 minutes before shutting off again.

I run and find one of our electricians to have a look while I check the UPS.  Cool.  I have 33 minutes of battery time left.  I make a few calls and send a few emails to prep users on the possibility of a total computer shutdown on the 5 companies we support out of our office.

Looks like the generator was toasted from the power surges earlier in the day.  Knowing all is lost and I need to power down 3 AS/400’s and maybe 10-12 Windows servers with 15 minutes to spare, I head back to the server room and check the UPS.  The front panel says “15 minutes battery time remaining” so I have plenty of room to move.

I get 2 AS/400’s on the way down (and I need a full 8 minutes for them to shut off) and start working on the 3rd when the UPS starts making the awful fast beeping noise indicating an imminent shutdown.  It’s times like these when you second guess yourself on your ability to restore from backup tapes.  1 very short minute later, all machines go down…HARD.  My stomach rolled over like you’d expect.

Colorful and creative cursing ensued at the UPS for telling me I had 15 minutes when I really had 5-6.

More colorful and creative cursing ensued at the flipping generator for failing when I needed it the most.

6 long hours later after the power was restored I started to power up the machines to find the lovely amber alert light on our new AS/400 model 515.  Luckily after booting into SST it just turned out to be an indicator of power fluctuation.

Even more colorful cursing ensued at the bloke at IBM who put this feature in the new machines.  Our models 170 and 270 went through the same experience but appeared fine with no system attention light.  Put the message in the QSYSOPR message queue but don’t fire up the “uh oh” light and cry wolf.  I want to see that light come on when I have a DASD failure or something and need to take action.

With all that said, all systems were a go with no hardware or software damage.

I don’t like dodging a bullet, but the alternative is being hit by one.  I had to hunt down one of the technicians in order to put myself on the UPS email alert system in case the systems went to UPS power in the next few days until we get the generator repaired.  In that case I’d have to remote in and power down all systems and bank on only having 20 instead of 33 minutes to get the job done right.  Tethered to the computer room 30 miles away.  I’ll also have to get the UPS checked to ensure it’s giving an accurate representation of battery time based on the load.

It’s time to review our systems continuity strategy and schedule more regular testing.  I’d suggest you do the same.