August, 2008

Developing Effective Test Plans for Automation

An application test plan should contain a minimum set of optimized test cases with maximum test coverage of all critical application functions. It should be executed using a tool that easily adapts to changing data and requirements.

In order to create an effective automation test, it is first necessary to review the application test plan provided by the application owner to evaluate its suitability for automation. You don’t want to automate a “BAD” test case. Consider the exact intent of the test plan and determine if you can create an effective test case (same coverage) more simply with more reliable automation. It is not acceptable to simply write all the automation scripts directly from the manual test plans. This has the same inherent limitation as doing record/replay for every script: the test is unreliable.

Design test cases and test scripts to be modular. Instead of using one script to perform multiple functions, break the script into separate functions.

Design test cases and test scripts to be generic in terms of process and repeatable in terms of data. Read test data from a separate source: keep the scripts free of test data so that when you do have to change the process or the data, you only have to maintain one item in one place.

Consider the goal of the test

Don’t just blindly follow a manual test plan. See if there is a simple way to accomplish the objective stated in the test plan.
Focus on modularity and reusability. Create a set of evaluation criteria for functions to be considered when using the automated test tool. These criteria may include:

• Repeatability of tests
• Criticality/Risk of applications
• Simplicity of operation
• Ease of automation
• Level of documentation of the function (requirements, specifications, etc.)

Targeting Test Plans for Automation

A test plan representing a good candidate for automation would have the following characteristics:

• Contains a repeatable sequence of actions
• The sequence of actions is repeated many times
• It is technically feasible to automate the sequence of actions (tool is capable, no external hardware actions)
• The behavior of the application under test is the same with automation as without
• Testing involves non-UI aspects of the application (almost all non-UI functions can and should be executed using automated tests)
• The same tests must be run on multiple hardware configurations
• The same tests must be run with varied combinations of other applications to verify compatibility (i.e., Interoperability Testing)

Have YOU been abducted by test tools?

Have You Been ABDUCTED by Your Test Automation Tools?

Here are some simple questions that you need to ask yourself:

#1. Have you experienced missing or lost time of any length, particularly of one or more hours, waiting for a screen to refresh?

——————————————————————————–

#2. Have you awakened to find yourself seemingly paralyzed or restrained for no reason and felt an unexplainable presence (your boss, waiting for results) near you?

——————————————————————————–

#3. Have you found any unusual psychological scars or marks with no possible explanation on how you received them? If so, did you have an adverse emotional reaction to them?

——————————————————————————–

#4. Have you seen balls of light or flashes of light in your eyes while NOT staring at your monitor waiting for “something to happen”?

——————————————————————————–

#5. Have you had memories of flying through the air which did NOT seem like a dream? (Well, maybe this has nothing to do with test tools — or so they want you to believe)

——————————————————————————–

#6. Have you had a “marked memory” that would not go away? (i.e., Windows blue screens, “General run error” messages, pain points in your fingers, etc.)

——————————————————————————–

#6. Have you seen beams of light outside your office, or shining into your cube through a window?

——————————————————————————–

#7. Do you have repeated dreams of ‘for…next’ loops, where you are doing the iteration?

——————————————————————————–

#8. Have you had one or more “trouble ticket” close-encounters at anytime your life?

——————————————————————————–

#9. Have you experienced feelings of cosmic awareness or a great concerned interest in ecology, the environment, vegetarianism, or anything else totally unrelated to test automation?

——————————————————————————–

#10. Do you have a strong sense that you have an important mission or task to perform, without knowing where this compulsion came from or what your mission is? (Note: this could mean you are a developer)

——————————————————————————–

#11. Have unexplainable events occurred during business hours that have left you feeling strangely anxious afterwards?

——————————————————————————–

#12. Have you ever awakened in a place where you were supposed to be working, or don’t remember ever going to sleep? (i.e. upside down in your chair, or under your desk)

——————————————————————————–

#13. Do you experience an uncontrollable fear when your movements are watched by a supervisor?

——————————————————————————–

#14. Do you have a strong adverse reaction to PowerPoint slides or even drawings of application architecture?

——————————————————————————–

#15. Do you have inexplicably strong phobias such as: keyboards, monitors, error messages, certain sounds, bright lights, heights or being alone?

——————————————————————————–

#16. Do you experience frequent self-esteem problems, for example: your appearance, abilities, ideas or decisions that you make throughout the SDLC?

——————————————————————————–

#17. Have you ever seen any other person in your presence become paralyzed, motionless or seemingly frozen in time, especially someone you work with?

——————————————————————————–

#18. Do you find yourself obsessed with the subject of test automation and/or software requirements resulting in a compulsive urge to research and study as much material on the subject as you can possibly find, such as books, user guides, the internet, etc.?

——————————————————————————–

#19. Do you frequently get the feeling that you are being watched, by more than one set of eyes, particularly while coding test scripts?

——————————————————————————–

#20. Do you have recurring but very pleasant dreams of floating through a closed window, door or a solid wall — anywhere away from your desk?

——————————————————————————–

#21. Have you had chronic sinusitis or nasal problems?

——————————————————————————–

#22. Do you have frequent or sporadic headaches, especially in your sinus cavity, behind one eye, or in one ear?

——————————————————————————–

#23. Do you often feel like you are going crazy for even thinking about such unbelievable things like “unattended execution”, “simple functions”, “stable networks”, “requirements”, etc?

——————————————————————————–

#24. Have you ever had any paranormal or psychic experiences, including Descriptive Programming?

——————————————————————————–

#25. Are you prone to be compulsive or act obsessively? (Note: this could also mean you are a developer)

——————————————————————————–

#26. Do you believe that you have ever received or channeled telepathic requirements from subject matter experts?

——————————————————————————–

#27. Have you ever been terrified of your cubical, now or every Monday morning?

——————————————————————————–

#28. Have you been having problems maintaining a serious relationship because you have this mysterious “feeling” that if you become involved with someone it will interfere with some important “automation” that you must do?

——————————————————————————–

#29. Do you have a difficult time trusting other people, especially authority figures?

——————————————————————————–

#30. Do you get the feeling that it is forbidden to discuss automation with anyone in management?

——————————————————————————–

If you answered “Yes” to ten or more of these questions, there is a very good chance that you have been abducted by your test tools, either once or perhaps even frequently.

There are several devices you can purchase from the author (for a modest fee) to protect yourself from recurring attacks. Examples are shown below.

Inquiries welcome.

;-))

Automation Test Data Management (TDM)

Test Data Management (TDM) is fundamental to the success of automated testing. For example, consider that one of the most beneficial forms of test automation is data-driven testing, which gives testers the ability to input and manipulate massive amounts of data in a relatively short period of time. If the data is bad, then running the tests could produce a mountain of unreliable results, and a whole lot of wasted time, money and effort. It pays to get data management right, especially with test automation.

Associated with test data creation are issues of capacity (i.e., disk space), data verification, data confidentiality, and prolonged test durations. If other forms of testing, such as manual or performance, are taking place in the same environment, there can be issues of concurrency, with tests failing simply because key data has changed “behind the scenes” during test execution, with records locked or altered unexpectedly.

Checking both visible test results and the effects of test execution on the database are essential to successful automation. Every automated test must start with a known data state, and end with the data in a predictable state — or even in its pre-test state.

Some questions that will help in planning a test data strategy include:

• How will data be created and entered into the system?
• Will production data be copied? If so, how will privacy of information be ensured?
• Will data need to be created from scratch?
• Who will be responsible for test data?
• What volume of data is needed to test the application or system, and how frequently will it need to be refreshed?
• Should date be refreshed completely, or incrementally?

In some cases, the test automation tools can be leveraged to load all the pre-test data and thus create the initial state of the database prior to executing functional tests.

If you take care of your Data, it will take care of you.

Automation, the Cold Equation

One of my passions is reading science fiction, and I remember a story called The Cold Equations. In this rather sad tale, no matter how much the characters want to change the way things are, they can’t, the inevitable outcome is dictated by the hard, cold facts of space travel.

I was in a meeting about test automation that reminded me of this story. One of the attendees had engaged our group of specialists to “create automation”. He had very high hopes about how it would expand test coverage, compress deadlines, and ultimately make a great impression on the management that funded his projects.

Although I really didn’t want to harsh his buzz, as a seasoned yet humble professional I had to point out that automation is simply a method of executing tests that had already been created by QA analysts working with application requirements. A group such as ours was indeed committed to and capable of “creating automation”, but it can not be done ex nihilo in some god-like fashion. We needed a solid starting point, at the very least some verified test cases, since we were not subject matter experts in the application. This was a simple fact — the Cold Equation.

There was a stunned silence on the phone, as if the fellow had indeed been ejected into outer space despite his deepest wishes. I assured him that we all actually wanted the same things from automation, but it doesn’t replace QA processes — it enhances them. We are simply dependent on his group’s delivery of the test cases — their side of the Cold Equation.

He looked again at the telltale white hand, then rose to his feet. What he must do would be unpleasant for both of them; the sooner it was over, the better.- from The Cold Equations, by Tom Godwin

They don’t call it Computer Science for nothin’!

Engage!

How to Implement a Functional Test Automation Methodology

A lot of people talk about using a “Methodology”, but what does that really mean? There are many complicated meanings for the word “methodology” itself, in Wikipedia and elsewhere. Personally, I have always taken it to mean “writing down (the -ology part) the way things are done (the method part) for a given process.”

OK, so I sit down, write up how everything is or should be done for functional test automation, and I have my methodology. What do I do with this scholarly work? Why, implement it, of course! Here are the steps I have used successfully in many automation engagements.

Discover
* Conduct Discovery Session(s) with subject matter experts, QA analysts, testers
* Establish Functional Test Goals
* Define Application(s) Under Test
* Review Requirements, Design Specifications, and Manual Functional Tests
* Identify Business Processes to Automate
* Identify Test Resources (Tools, Staff, Skills, Environments)
* Create Test Plan
* Develop Detailed Project Plan

Develop
* Exercise AUT (that’s Application Under Test, for you newbies)
* Build Test Data
* Create Business Component Tests
* Define Test Plan Components
* Customize Test Scripts and Function Libraries
* Create Test Sets and Parameters
* Dry-Run Test Sets (Test the Tests)

Execute
* Verify Test Readiness (App Build Complete?)
* Validate Test Data
* Execute Test Cycles (Iterations)
* Review Functional Test Results
* Identify Defects

Analyze
* Analyze defects discovered
* Submit Defects to Development for Resolution
* Retest Cycle for Closure
* Validate results with stakeholders

Report
* Perform Test Coverage Analysis
* Test Execution Metrics
* Present Report(s) to stakeholders

Transition
* Knowledge Transfer
* Framework Support and Maintenance (Ongoing)

- each section includes Key Terms, Roles & Responsibilities, and Deliverables
- effort (number of tasks) diminishes as method progresses

If you are not a newbie (an oldbie?), then you might recognize that this is a kind of mashup between standard QA processes and test automation. Since test automation is really just another method for executing tests, this is a natural fit. As with all good things, a solid QA process is reusable in different situations, including automation.

function junction: File System Object

Writing functions that make use of the methods exposed by the Windows file system object might seem trivial at first, but take my word for it, it is a worthwhile activity. If you use HP QuickTest Pro for test automation, or even need to do some system maintenance with VBScript, these simple scripts will become an invaluable part of your toolkit. You will come to depend on their simplicity and robustness when they are used for mundane tasks within your test automation scripts.

This first function returns the name of a file contained in a long path statement.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile.xls”:

‘————————————————————–
Function GetLongFileName(FullSpec)
‘returns file name with the extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetLongFileName = fso.GetFileName(FullSpec)
End Function

Conversely, this next function returns the full path to a file from the fully-pathed file name.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “C:\Folder1\Folder2\Folder3″:

‘————————————————————–
Function GetParentPath(FullSpec)
‘returns just the path portion of string
‘assuming last element is file name
‘ Note: function parses out trailing backslash
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetParentPath = fso.GetParentFolderName(FullSpec)
End Function

And as another twist, the function below returns the name of the file without the extension.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile”. This is useful as part of a routine for renaming converted files:

‘————————————————————–
Function GetFileBase(filespec)
‘returns file name without extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(”Scripting.FileSystemObject”)
GetFileBase = fso.GetBaseName(filespec)
End Function

Of course, we can do much more than parse file names. Here’s a function that will add a new folder to a specified path, after verifying that the path is found and that the folder doesn’t already exist:

‘————————————————————–
Function AddNewFolder(FullPath, FolderName)
‘creates a subfolder under the specified path
Dim fso, f, fc, nf
If FolderName = “” Then
FolderName = “New Folder”
End If
Set fso = CreateObject(”Scripting.FileSystemObject”)
If (fso.FolderExists(FullPath)) Then
Set f = fso.GetFolder(FullPath)
Set fc = f.SubFolders
Else
AddNewFolder = “Path ” & FullPath & ” not found!”
Exit Function
End If

If (fso.FolderExists(FullPath & “\” & FolderName)) Then
AddNewFolder = “Folder ” & FolderName & ” exists”
Exit Function
Else
Set nf = fc.Add(FolderName)
AddNewFolder = 0 ‘ “Folder ” & FolderName & ” added”
End If
End Function

Don’t worry if your functions seem too small and simple: in the end, they really work better that way. If they do one thing, and do it well, they become a sturdy link in the chain of actions that can make up a test case!

Introduction to HP QuickTest Pro Objects

HP QuickTest Professional uses an Object Repository to store information about the various fields and controls used to build the user interface of a software application. This repository is essentially a database of the names and properties of all the objects encountered during test script creation.

“Essentially all configuration and run functionality provided via the QuickTest interface is in some way represented in the QuickTest automation object model via objects, methods, and properties.” – HP (Mercury) QuickTest Professional User’s Guide

A QuickTest Pro 9.x Object Repository file structure looks something like this:

Here’s an example of the object definition for an ‘Add’ button contained in the repository above:

Note that the value of the “name” property defaults to what was discovered during recording, but the value of any property can be changed by a QTP scripter to a more recognizable value - including the logical name - as part of the repository maintenance process.

By comparison, these properties are very similar to object definitions found using the Microsoft IE Dev Toolbar.

Application object:

Object properties discovered with Microsft IE Developer Toolbar:

Note also that whatever works in VBScript for object handling works in QTP - but not necessarily vice-versa!

Creating a Solid Test Automation Framework with QTP

HP’s QuickTest Pro can itself be used to build a test automation framework. Listed below are some of the characteristics you should take into consideration during the design phase.

Test automation scripts and functions should:

• Be Reusable and Repeatable
• Require Low Maintenance
• Allow Unattended Execution
• Support Platform Independence - IE 6 or IE 7, shouldn’t matter
• Use Data Abstraction - Keyword-Driven Framework

HP QuickTest Pro scripts should use:

• Environment Variables
• Test execution ‘Driver’ to call ‘Actions’
• ‘Action’ to call functions - i.e., function call for each action
• Imported test data (Excel) - common format
• Results reporting - QTP + Quality Center; Excel “Step Status”

VBScript functions can be:

• Simple - Specific purpose; True / False or 0 / 1 return code
• Complex - Multiple steps (e.g., login); “Fuzzy” return code that maps to business rules
• Resilient - include Error Handling

Handling Objects in the Repository is done by:

• Object changes/additions per app release
• Replace with descriptive programming whenever possible
• Functions using regular expressions (e.g., popup windows: IE 6 vs IE 7)
• Web app objects usually classifiable by type, html tag, or class (QTP test object properties)
• Reusable methods: look before you write!

Test Automation Metrics: Tracking Progress

Did you ever have one of those days/weeks/months when your boss walks nervously into your office and says in that ‘I-just-came-out-of-a-very-long-and-gruesome-meeting’ tone of voice, “Well, , how’s that automation project coming all?” You figure that just answering “Fine” probably is not enough in this situation, since the fate of the company — and certainly your job — hangs in the balance. Oh, if only you had been tracking your progress all along!

Well, have no fear, the answer is here!

In a previous blog entry, I stated that “automated testing is the programmatic execution of test cases.” That gives us the starting point for our metrics: manual test case count. Generally, these counts fall into two categories: strategically Planned Test Cases, and actually written Manual Test Cases.

Most automation effort tends to be focused on coverting written manual test cases to automated test scripts, so that gives us our third metric, Automated Test Cases.

Progress can be gauged by the ratio of actual to expected. Let’s asemble our metrics into a table and see how this plays out:

As shown above, the progress of automation is expressed as the ratio between (Automated Test Cases)/(Manual Test Cases). This implies a typical dependency of automation on the creation of separate, detailed manual tests created by subject matter experts. If your organization follows a different sort of QA process, where the person doing the automation is also writing the manual tests and converting them directly into automated test scripts, the formula could be altered to express the ratio of (Automated Test Cases)/(Planned Test Cases).

Functional Testing of Web Services: Part IV

Two other web service test tools which deserve honorable mention in our comparison tests were: soapUI, available as Open Source or in a pro version distributed by eviware; and HP’s QuickTest Professional Web Services Add-In.

soapUI

This tool does not require a detailed knowledge of complicated technologies like .NET. Although the user interface is not as intuitive as that of some other tools, a tester who has some experience in programming can learn to create, organize and perform test activities fairly quickly.

soapUI allows structuring your test project into test suites that contain test cases, which can contain test steps. This structure is well-managed: you can add, modify, delete and change the order of every item in the structure. soapUI provides the tools to manage and run your test cases, and to view the SOAP responses to your test requests. You can even include limited load testing scenarios. For added flexibility, soapUI supports Groovy Script, a scripting language similar to Java.

QuickTest Pro Web-Services Add-In

QuickTest Pro with the Web Services Add-In offers a wizard to create a test object that represents the Web Service and port you want to test, and inserts the relevant steps directly into your test or component. This accelerates the process of designing a basic test that checks the operations that your Web service supports. You then update the wizard-generated steps of your test or component by replacing the generated argument values with known valid values, updating the expected values, and selecting the nodes you want to check in your checkpoints.

A QTP XML Warehouse setting is used to store request data, and a QTP Object Repository is used to store responses as checkpoints for verification.

The wizard generates a generic XML structure as a place holder for the expected XML return values. However, before you can actually run your test or component, you must replace the default values with the appropriate values for your test. A valid SOAP request can be imported into the XML Warehouse as a separate step, and a separate XML checkpoint would need to be created for each test case. Defining tests is a little cumbersome and the UI is a little clumsy, but if you’re already using QTP for functional testing, including web service regression tests in your test suites is an easy option.