Software testing

Projects and Priorities

In keeping up with these times of limited resources and high desires (if not expectations - which may also be running high!) it is critical that project priority be properly set.  Many of us have been involved with projects which have many facets to them, and depending upon the environment those facets may be very clearly defined, or perhaps somewhat nebulous.  They may be down on paper or in the developers and users heads.  Either way, setting the proper priority for the project is critical.

While there are many criteria to focus upon I believe there are two areas in particular that warrant looking at very closely.  Certainly the overall value of the application to the business is one critical criteria to be considered.  Where does the application fit?  What will it bring in the way of added value to the operation?  Can this added value be quantified?  Is this an application which when completed will result in the saving of time, labor or materials?  If so, to what extent? 

In addition to the added business value of the application I believe that the testing procedures being used with the project be clearly monitored.  Test for value to ensure that the desired or expected return can be achieved.  Test especially for data integrity, try to “break” it, ensure solid product through your testing.  Be creative in the testing wherever possible in an effort to prove usability.

I believe that any project which can’t produce a high “value add” probably shouldn’t be done (at least at this time).  However, any project given such high “value add” is worth taking the time required for testing which will produce a reliable and valuable application.  Now more than ever, extensive testing should be the order of the day.

The Illusion of Perfect Software

Perfect software doesn’t exist - that should be “…elementary my Dear Watson.”  Ok, so starting with that assumption where does one go with their quest for the “perfect” software, or perhaps the “perfect” tester, or the “perfect” whatever for that matter!  A couple of weeks ago I bookmarked a post that caught my eye entitled Book gives managers a software testing reality check.  It was right at the beginning that my eye caught the “…perfect software doesn’t exist” statement in the post.

I’ve often written here about software expectations.  In my experience end users are often looking for (maybe even expecting) perfect solutions to their software needs, but I believe that is not possible.  The expression, “one man’s junk, another man’s treasure” kind of applies here.  The definition of “perfect” among evaluators will vary as much as the individual personalities of the evaluators.  Marketers, of course, describe their “perfect” solutions with charm, grace and slick that defies logic.  Save us please!

The reality of software development and testing is that a series of trade-offs has been made to arrive at an acceptable, functional and apparently “bug free” application.  There will always be decisions made such as feature set vs cost to produce the feature set.  Sometimes software with “known issues” is released - possibly because of cost concerns, possibly because the need is so great that expediency is the driving factor.  (In those cases, however, it is quite possible that a work-a-round has been identified, or that the known issue exists only when a user goes down a path which they shouldn’t be going down to begin with).  Yes, in that case the software “should” prevent access to that path, but I’ve certainly seen many an instance where the time and cost to do it just wasn’t worth the effort when user training “should” take care of the issue.

Shortcuts, Nightmares and Reboots

I’ve just recently read a couple of posts in a newsgroup which reminded me of a few nightmares I’ve created for myself as a result of taking a shortcut (or two).  One of the situations was regarding changes made to a working server (…and of course one doesn’t want to reboot said server after simply adding some software — especially when the software doesn’t specify rebooting).  The story I read went something like this:  a few months ago a demo application program was loaded onto the server, the system was not rebooted after install, all programs worked fine - newly installed demo and existing programs - and they continued to work until the system was rebooted. 

It seems that in this case there were “updates” loaded onto the system which upon the reboot replaced the working versions of the dll’s with incompatible versions.  The “demo” program had been reviewed months ago, determined that it wasn’t desired, but left on the system anyway.  By the time of the system reboot the consultant called in to address the problem was of course told “Nothing has been changed!”  Yes, we did have to take the system down for an extended power outage — but it booted fine - “…no errors…”. 

It took a team 2 1/2 days to establish the cause of the problem - that being that at the time the “demo” software was loaded certain dll’s were being “used” and therefore would not be replaced until the next reboot of the system.  By the time that reboot happened nobody ever suspected that the “demo” software load was the cause of the problem.

Had the system been rebooted after the install the “issue” and its cause would have been immediately identified.  Months later, that wasn’t the case.  The shortcut? Not rebooting after the install.  The nightmare?  2 1/2 days of anguish with printing not working.  The solution?  Reboot after loading new software.

Software Development Project Closure Thoughts

Back in mid June I was interviewed by a freelance writer working on an article about software development project closure.  It was an interesting topic to me, and one which I have honestly never really stopped to think about.  While I’ve blogged often about my thoughts regarding user input, communications and other development topics — project closure hasn’t really been much a part of my planning.

The article for which I was interviewed is a well written piece which I will not try enhance.  (It is already full of my thoughts during the interview!).  I expect that readers of this blog will find it interesting.  The article entitled “The Beginning of the End: Defining Project Closure” is a recommended read.

Software Application Speed and ADD Users

ADD — aka Attention Deficit Disorder — is, some say, an altogether too common ailment in today’s society, at least in the U.S. While I certainly don’t intend to debate that issue, a recent experience got me to thinking about how children with ADD do and will grow up to be computer application users. These thoughts in turn had me thinking once again about application interface design and user needs.

Much is written about interface design. For years we’ve worked hard to provide intuitive “user-friendly” interfaces for our applications. Much has been written about visual presentation, and many options for changes to the visual presentation such as “skinning” have been introduced.

Perhaps, however, the most important of all the considerations for an application should be the application response time. I’m not aware of any user who doesn’t get impatient with poor response - defined here as a response time meeting their personal expectation! As more and more users (ADD or otherwise) become frustrated with either speed issues OR for some, the cluttered screen, it seems imperative that we as developers be constantly on the alert for signs of this frustration brewing. My experience would indicate that most computer applications used in a business environment are not being used by “computer” users, but rather by users who understand the task or job they are required to do, and (in some case, regrettably) must use the “computer” to accomplish the task.

Application design is no trivial task!

Updates that Fail

Don’t you hate it when an update fails? I had the experience just yesterday while doing an update on my Microsoft Vista laptop. The updates appeared to be going correctly, but at the very end I found myself in an endless loop situation. The situation was not pretty and upon further investigation in the web a Google search pointed me in the direction of a workaround. I wasn’t sure that was the right direction, but this was not something that was unique to myself. The work-around worked!

I was fortunate. I only lost about an hour in my struggle to find the answer to take corrective action. However, others were not so lucky, many losing countless hours before recovering — or not — their system. My investigation to this point indicates that while this has happened on an occasional basis for some, Microsoft has been apparently quiet regarding this problem. This was an update that I expected to work absolutely perfectly since a fellow developer friend of mine said he never liked Vista until he had installed service pack one.

It seems that on August 12 of this year VMare issued an update which caused issues regarding the licensing. The problem prompted an open letter to VMware customers from VMware’s president (…see letter). This appears to be a fine example of software quality assurance going wrong — just how this embarrassing situation occurred is certainly under investigation by VMware.

As I see it, these two examples of updates that failed are very different. In the case of the VMware issue a fix was readily found, as the issue was the result of a programming error. A quick fix of the error and the problem was resolved.

As for the Microsoft issue. I see the problem as somewhat different in that while the problem has occurred for some, there are many for whom it has not occurred. This represents a much more difficult issue to resolve. As a result of the very nature of personal PCs, testing for this error becomes much more of an issue. I can almost forgive Microsoft for not having a solution to it.

Yes, I’ve had perhaps more than my share of failed updates. I find that I am much more upset with programming or testing errors than with errors which occur as the result of network or hardware issues which I cannot duplicate. Regardless of the cause, there is no such thing as a good failed update.

Software Testing, Development and Music

I read with intense interest the “From the Editor” post entitled “Software testing lessons from music” on SearchSoftwareQuality.com this week.  The article caught my eye for its title uniqueness, but kept my attention as I read the theme of the Association for Software Testing’s annual conference - “Beyond the Boundaries: Interdisciplinary Approaches to Software Testing.”

Interdisciplinary approaches to software testing just seemd to be ”beyond the boundaries” for my brain to comprehend until reading the article.  A comment on the post was also particularly meaningful to me - stating a view that I have often been criticized for voicing myself.  “Testing, like designing and programming, is as much an art form as engineering.” The comment goes on to compare testers to choreographers and users to the dancers.  I personally love that concept!  It portrays a vision of software development which I have long had, and work daily to achieve - that is creating the perfect symphony that users can play beautifully.

User Innovation

A recent post by fellow ITKE blogger John Wilder entitled “Sparking Innovation” caught my eye this morning and I just have to comment.

His statement that “…I trust my users to come up with innovative ways of using some of these new products, and I’m not so sure that IT would ever be able to envision all the possible uses…”  immediately brought my mind into the many situations I have experienced where my users innovative ways of using a piece of software has resulted in their identifying ways in which IT (…or the developer/tester) certainly had not envisioned it would be used.  Is it a bug when software isn’t used in a manner for which it was designed?  Bug or not, the situation has often led to “changes” to the software.  I believe that users, because of their innovative approaches to use CAN be the best of software quality testers — but developer beware!

John also refers to a situation where program use is resulting in a tension between a direction that IT is promoting and another approach suggested by the users.  This kind of tension between application use has been a common experience for me when developing custom programs for my customers.  Some users will prefer one direction, and others another.  IT and the developer walk a tightrope between users every time this situation manifests itself.

Beta Testing - Not for Everybody

It’s like the stock market - it can be risky - it can be frustrating and downright annoying - and it can be rewarding.  I’ve always liked participating in beta testing of products which I find of value, or as has happend to me before, in a product that I had to use - but (let’s just say to be nice), needed a lot of work — and by beta testing (with our live data of course) we got to evaluate the latest and greatest of the software.  Most of the time it was a win-win situation, but not always.

I wrote a few weeks ago about how my chiropractor didn’t realize what it meant to be a beta test site, and consequently just how much she “suffered” with the arrangement.

For me, as a custom software developer, I think I like participating in anothers beta test because I invevitably learn from the experience.  There are 2 major learning opportunites that I consider to provide the value for me by participating — they are:

1. The opportunity to really learn a product and understand it
2. The opportunity to find out just how the software provider works

Products and companies have made it and lost it based upon my experience with beta testing their product.  If a company isn’t easy to work with when you are providing beta testing, they probably aren’t a company you’ll want to work with long term.  Likewise with the product.

Useful Tools of the Trade

A few days ago I blogged about how I was playing detective with a clients computer system, struggling with performance issues.  It has been an interesting few days since that blog, and I have discovered much, and learned along the way.

One thing that I learned about is a nifty utility offered by Microsoft which can aid in the analysis of exactly what’s going on with your Windows 2003 Server.  The utility is called Server Performance Advisor.  The utility is particularly helpful for presentation of data which is collected by the multitude of logs kept as part of normal server operations.  It also provides additional information and combines it in a much more understandable manner than just reviewing raw log data.

Starting with a system overview, it can also provide information about other common services provided by the server, like IIS, DNS, Active Directory and others.  It includes useful help!  Depending upon the selection of services the utlility collects data for either 100 or 300 seconds, and then creates analysis of the data. 

If you have a Windows Server 2003 and are not familiar with this tool I’d recommend it highly.  Follow the link to Server Performance Advisor for more information.

My search for answers this week also took me to an article on Computerworld outlining 10 Great Free Network Tools.  Since the analysis of my performance issue indicated that the issue was not network related I did not follow-up with any of these network tools, but I pass the link on for what its worth.  Hopefully it can help someone out.