QA Focus


September 11, 2008  6:45 PM

HP QuickTest Pro 10 Beta

Sentinel627 Greg Annen Profile: Sentinel627

My current employer is about to participate in a beta test for HP’s QuickTest Professional 10 release, also known as “Atlantis”. This release was demo’ed at HP’s Software Universe expo this summer. I did not attend this event, but from what I have read, it has some long-awaited features.

Tighter integration with Quality Center 10

Source Control:

  • Check-In/Check-Out, at the user level
  • Versioning, single resource, multiple folders and project-wide

Resource Management:

  • View OR’s, datatables, and library code from QC
  • Dependency visibility and warnings
  • Automatic path tracking when moving resources
  • Shared OR user information

QuickTest Pro Features

Reports:

  • Traceability to a line of code in an action script
  • Export to Word, PDF, or through XSL
  • Include Windows Performance Monitor data, related to test steps
  • Include bitmap attachments

User Interface:

  • Improved autocomplete and intellisense
  • Tasks and Comments pane
  • Code-zone identifiers – blocks of code like ‘If…Then’ marked with blue lines
  • Custom toolbars

General Goodies:

  • Enhanced bitmap checkpoints
  • Local copy of test includes linked resources
  • Load external actions without previously associating them

We don’t yet have Quality Center 10, so I will also be interested in the beta version’s integration with previous versions of QC. And the ease of converting/migrating scripts and resources from previous versions of QTP.

I will post future articles about this beta test as it progresses.

September 1, 2008  3:36 AM

Virtualization: the ‘QuickTest Appliance’ Concept

Sentinel627 Greg Annen Profile: Sentinel627

VMWare Server is a great product. It can be used to host Virtual Machines (VMs) that operate independently from each other and from the host server, creating a tightly controlled test execution environment.

Here’s an example:
QTP VM Applicance

Some advantages of this virtual environment:

  • No need to install QTP on your personal desktop/laptop
  • Virtual Machine configurations are cloned from a template and can be easily restored/recreated
  • Test execution does not interact with your desktop applications (email, IM, browser)

More soon on this topic…


August 29, 2008  3:23 AM

Developing Effective Test Plans for Automation

Sentinel627 Greg Annen Profile: Sentinel627

An application test plan should contain a minimum set of optimized test cases with maximum test coverage of all critical application functions. It should be executed using a tool that easily adapts to changing data and requirements.

In order to create an effective automation test, it is first necessary to review the application test plan provided by the application owner to evaluate its suitability for automation. You don’t want to automate a “BAD” test case. Consider the exact intent of the test plan and determine if you can create an effective test case (same coverage) more simply with more reliable automation. It is not acceptable to simply write all the automation scripts directly from the manual test plans. This has the same inherent limitation as doing record/replay for every script: the test is unreliable.

Design test cases and test scripts to be modular. Instead of using one script to perform multiple functions, break the script into separate functions.

Design test cases and test scripts to be generic in terms of process and repeatable in terms of data. Read test data from a separate source: keep the scripts free of test data so that when you do have to change the process or the data, you only have to maintain one item in one place.

Consider the goal of the test

Don’t just blindly follow a manual test plan. See if there is a simple way to accomplish the objective stated in the test plan.
Focus on modularity and reusability. Create a set of evaluation criteria for functions to be considered when using the automated test tool. These criteria may include:

  • Repeatability of tests
  • Criticality/Risk of applications
  • Simplicity of operation
  • Ease of automation
  • Level of documentation of the function (requirements, specifications, etc.)

Targeting Test Plans for Automation

A test plan representing a good candidate for automation would have the following characteristics:

  • Contains a repeatable sequence of actions
  • The sequence of actions is repeated many times
  • It is technically feasible to automate the sequence of actions (tool is capable, no external hardware actions)
  • The behavior of the application under test is the same with automation as without
  • Testing involves non-UI aspects of the application (almost all non-UI functions can and should be executed using automated tests)
  • The same tests must be run on multiple hardware configurations
  • The same tests must be run with varied combinations of other applications to verify compatibility (i.e., Interoperability Testing)


August 29, 2008  2:51 AM

Have YOU been abducted by test tools?

Sentinel627 Greg Annen Profile: Sentinel627

Have You Been ABDUCTED by Your Test Automation Tools?

Here are some simple questions that you need to ask yourself:

#1. Have you experienced missing or lost time of any length, particularly of one or more hours, waiting for a screen to refresh?

——————————————————————————–

#2. Have you awakened to find yourself seemingly paralyzed or restrained for no reason and felt an unexplainable presence (your boss, waiting for results) near you?

——————————————————————————–

#3. Have you found any unusual psychological scars or marks with no possible explanation on how you received them? If so, did you have an adverse emotional reaction to them?

——————————————————————————–

#4. Have you seen balls of light or flashes of light in your eyes while NOT staring at your monitor waiting for “something to happen”?

——————————————————————————–

#5. Have you had memories of flying through the air which did NOT seem like a dream? (Well, maybe this has nothing to do with test tools — or so they want you to believe)

——————————————————————————–

#6. Have you had a “marked memory” that would not go away? (i.e., Windows blue screens, “General run error” messages, pain points in your fingers, etc.)

——————————————————————————–

#6. Have you seen beams of light outside your office, or shining into your cube through a window?

——————————————————————————–

#7. Do you have repeated dreams of ‘for…next’ loops, where you are doing the iteration?

——————————————————————————–

#8. Have you had one or more “trouble ticket” close-encounters at anytime your life?

——————————————————————————–

#9. Have you experienced feelings of cosmic awareness or a great concerned interest in ecology, the environment, vegetarianism, or anything else totally unrelated to test automation?

——————————————————————————–

#10. Do you have a strong sense that you have an important mission or task to perform, without knowing where this compulsion came from or what your mission is? (Note: this could mean you are a developer)

——————————————————————————–

#11. Have unexplainable events occurred during business hours that have left you feeling strangely anxious afterwards?

——————————————————————————–

#12. Have you ever awakened in a place where you were supposed to be working, or don’t remember ever going to sleep? (i.e. upside down in your chair, or under your desk)

——————————————————————————–

#13. Do you experience an uncontrollable fear when your movements are watched by a supervisor?

——————————————————————————–

#14. Do you have a strong adverse reaction to PowerPoint slides or even drawings of application architecture?

——————————————————————————–

#15. Do you have inexplicably strong phobias such as: keyboards, monitors, error messages, certain sounds, bright lights, heights or being alone?

——————————————————————————–

#16. Do you experience frequent self-esteem problems, for example: your appearance, abilities, ideas or decisions that you make throughout the SDLC?

——————————————————————————–

#17. Have you ever seen any other person in your presence become paralyzed, motionless or seemingly frozen in time, especially someone you work with?

——————————————————————————–

#18. Do you find yourself obsessed with the subject of test automation and/or software requirements resulting in a compulsive urge to research and study as much material on the subject as you can possibly find, such as books, user guides, the internet, etc.?

——————————————————————————–

#19. Do you frequently get the feeling that you are being watched, by more than one set of eyes, particularly while coding test scripts?

——————————————————————————–

#20. Do you have recurring but very pleasant dreams of floating through a closed window, door or a solid wall — anywhere away from your desk?

——————————————————————————–

#21. Have you had chronic sinusitis or nasal problems?

——————————————————————————–

#22. Do you have frequent or sporadic headaches, especially in your sinus cavity, behind one eye, or in one ear?

——————————————————————————–

#23. Do you often feel like you are going crazy for even thinking about such unbelievable things like “unattended execution”, “simple functions”, “stable networks”, “requirements”, etc?

——————————————————————————–

#24. Have you ever had any paranormal or psychic experiences, including Descriptive Programming?

——————————————————————————–

#25. Are you prone to be compulsive or act obsessively? (Note: this could also mean you are a developer)

——————————————————————————–

#26. Do you believe that you have ever received or channeled telepathic requirements from subject matter experts?

——————————————————————————–

#27. Have you ever been terrified of your cubical, now or every Monday morning?

——————————————————————————–

#28. Have you been having problems maintaining a serious relationship because you have this mysterious “feeling” that if you become involved with someone it will interfere with some important “automation” that you must do?

——————————————————————————–

#29. Do you have a difficult time trusting other people, especially authority figures?

——————————————————————————–

#30. Do you get the feeling that it is forbidden to discuss automation with anyone in management?

——————————————————————————–

If you answered “Yes” to ten or more of these questions, there is a very good chance that you have been abducted by your test tools, either once or perhaps even frequently.

There are several devices you can purchase from the author (for a modest fee) to protect yourself from recurring attacks. Examples are shown below.

Stop Test Tool Abductions! Oh, the humanity!

Inquiries welcome.

;-))


August 29, 2008  12:38 AM

Automation Test Data Management (TDM)

Sentinel627 Greg Annen Profile: Sentinel627

Test Data Management (TDM) is fundamental to the success of automated testing. For example, consider that one of the most beneficial forms of test automation is data-driven testing, which gives testers the ability to input and manipulate massive amounts of data in a relatively short period of time. If the data is bad, then running the tests could produce a mountain of unreliable results, and a whole lot of wasted time, money and effort. It pays to get data management right, especially with test automation.

Associated with test data creation are issues of capacity (i.e., disk space), data verification, data confidentiality, and prolonged test durations. If other forms of testing, such as manual or performance, are taking place in the same environment, there can be issues of concurrency, with tests failing simply because key data has changed “behind the scenes” during test execution, with records locked or altered unexpectedly.

Checking both visible test results and the effects of test execution on the database are essential to successful automation. Every automated test must start with a known data state, and end with the data in a predictable state — or even in its pre-test state.

Some questions that will help in planning a test data strategy include:

  • How will data be created and entered into the system?
  • Will production data be copied? If so, how will privacy of information be ensured?
  • Will data need to be created from scratch?
  • Who will be responsible for test data?
  • What volume of data is needed to test the application or system, and how frequently will it need to be refreshed?
  • Should date be refreshed completely, or incrementally?

In some cases, the test automation tools can be leveraged to load all the pre-test data and thus create the initial state of the database prior to executing functional tests.

If you take care of your Data, it will take care of you.

Mr Data


August 27, 2008  3:16 AM

Automation, the Cold Equation

Sentinel627 Greg Annen Profile: Sentinel627

One of my passions is reading science fiction, and I remember a story called The Cold Equations. In this rather sad tale, no matter how much the characters want to change the way things are, they can’t, the inevitable outcome is dictated by the hard, cold facts of space travel.

I was in a meeting about test automation that reminded me of this story. One of the attendees had engaged our group of specialists to “create automation”. He had very high hopes about how it would expand test coverage, compress deadlines, and ultimately make a great impression on the management that funded his projects.

Although I really didn’t want to harsh his buzz, as a seasoned yet humble professional I had to point out that automation is simply a method of executing tests that had already been created by QA analysts working with application requirements. A group such as ours was indeed committed to and capable of “creating automation”, but it can not be done ex nihilo in some god-like fashion. We needed a solid starting point, at the very least some verified test cases, since we were not subject matter experts in the application. This was a simple fact — the Cold Equation.

There was a stunned silence on the phone, as if the fellow had indeed been ejected into outer space despite his deepest wishes. I assured him that we all actually wanted the same things from automation, but it doesn’t replace QA processes — it enhances them. We are simply dependent on his group’s delivery of the test cases — their side of the Cold Equation.

He looked again at the telltale white hand, then rose to his feet. What he must do would be unpleasant for both of them; the sooner it was over, the better.– from The Cold Equations, by Tom Godwin

They don’t call it Computer Science for nothin’!

Engage!


August 23, 2008  12:28 AM

How to Implement a Functional Test Automation Methodology

Sentinel627 Greg Annen Profile: Sentinel627

A lot of people talk about using a “Methodology”, but what does that really mean? There are many complicated meanings for the word “methodology” itself, in Wikipedia and elsewhere. Personally, I have always taken it to mean “writing down (the -ology part) the way things are done (the method part) for a given process.”

OK, so I sit down, write up how everything is or should be done for functional test automation, and I have my methodology. What do I do with this scholarly work? Why, implement it, of course! Here are the steps I have used successfully in many automation engagements.

Discover
* Conduct Discovery Session(s) with subject matter experts, QA analysts, testers
* Establish Functional Test Goals
* Define Application(s) Under Test
* Review Requirements, Design Specifications, and Manual Functional Tests
* Identify Business Processes to Automate
* Identify Test Resources (Tools, Staff, Skills, Environments)
* Create Test Plan
* Develop Detailed Project Plan

Develop
* Exercise AUT (that’s Application Under Test, for you newbies)
* Build Test Data
* Create Business Component Tests
* Define Test Plan Components
* Customize Test Scripts and Function Libraries
* Create Test Sets and Parameters
* Dry-Run Test Sets (Test the Tests)

Execute
* Verify Test Readiness (App Build Complete?)
* Validate Test Data
* Execute Test Cycles (Iterations)
* Review Functional Test Results
* Identify Defects

Analyze
* Analyze defects discovered
* Submit Defects to Development for Resolution
* Retest Cycle for Closure
* Validate results with stakeholders

Report
* Perform Test Coverage Analysis
* Test Execution Metrics
* Present Report(s) to stakeholders

Transition
* Knowledge Transfer
* Framework Support and Maintenance (Ongoing)

– each section includes Key Terms, Roles & Responsibilities, and Deliverables
– effort (number of tasks) diminishes as method progresses

If you are not a newbie (an oldbie?), then you might recognize that this is a kind of mashup between standard QA processes and test automation. Since test automation is really just another method for executing tests, this is a natural fit. As with all good things, a solid QA process is reusable in different situations, including automation.


August 21, 2008  3:25 AM

function junction: File System Object

Sentinel627 Greg Annen Profile: Sentinel627

Writing functions that make use of the methods exposed by the Windows file system object might seem trivial at first, but take my word for it, it is a worthwhile activity. If you use HP QuickTest Pro for test automation, or even need to do some system maintenance with VBScript, these simple scripts will become an invaluable part of your toolkit. You will come to depend on their simplicity and robustness when they are used for mundane tasks within your test automation scripts.

This first function returns the name of a file contained in a long path statement.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile.xls”:

‘————————————————————–
Function GetLongFileName(FullSpec)
‘returns file name with the extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(“Scripting.FileSystemObject”)
GetLongFileName = fso.GetFileName(FullSpec)
End Function

Conversely, this next function returns the full path to a file from the fully-pathed file name.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “C:\Folder1\Folder2\Folder3”:

‘————————————————————–
Function GetParentPath(FullSpec)
‘returns just the path portion of string
‘assuming last element is file name
‘ Note: function parses out trailing backslash
Dim fso
Set fso = CreateObject(“Scripting.FileSystemObject”)
GetParentPath = fso.GetParentFolderName(FullSpec)
End Function

And as another twist, the function below returns the name of the file without the extension.
For example, if FullSpec = “C:\Folder1\Folder2\Folder3\MyFile.xls”, the function will return the value “MyFile”. This is useful as part of a routine for renaming converted files:

‘————————————————————–
Function GetFileBase(filespec)
‘returns file name without extension from full path
‘assuming last element is file name
Dim fso
Set fso = CreateObject(“Scripting.FileSystemObject”)
GetFileBase = fso.GetBaseName(filespec)
End Function

Of course, we can do much more than parse file names. Here’s a function that will add a new folder to a specified path, after verifying that the path is found and that the folder doesn’t already exist:

‘————————————————————–
Function AddNewFolder(FullPath, FolderName)
‘creates a subfolder under the specified path
Dim fso, f, fc, nf
If FolderName = “” Then
FolderName = “New Folder”
End If
Set fso = CreateObject(“Scripting.FileSystemObject”)
If (fso.FolderExists(FullPath)) Then
Set f = fso.GetFolder(FullPath)
Set fc = f.SubFolders
Else
AddNewFolder = “Path ” & FullPath & ” not found!”
Exit Function
End If

If (fso.FolderExists(FullPath & “\” & FolderName)) Then
AddNewFolder = “Folder ” & FolderName & ” exists”
Exit Function
Else
Set nf = fc.Add(FolderName)
AddNewFolder = 0 ‘ “Folder ” & FolderName & ” added”
End If
End Function

Don’t worry if your functions seem too small and simple: in the end, they really work better that way. If they do one thing, and do it well, they become a sturdy link in the chain of actions that can make up a test case!


August 16, 2008  6:43 PM

Introduction to HP QuickTest Pro Objects

Sentinel627 Greg Annen Profile: Sentinel627

HP QuickTest Professional uses an Object Repository to store information about the various fields and controls used to build the user interface of a software application. This repository is essentially a database of the names and properties of all the objects encountered during test script creation.

“Essentially all configuration and run functionality provided via the QuickTest interface is in some way represented in the QuickTest automation object model via objects, methods, and properties.” – HP (Mercury) QuickTest Professional User’s Guide

A QuickTest Pro 9.x Object Repository file structure looks something like this:
QTP Object Repository

Here’s an example of the object definition for an ‘Add’ button contained in the repository above:
Add button defintion

Note that the value of the “name” property defaults to what was discovered during recording, but the value of any property can be changed by a QTP scripter to a more recognizable value – including the logical name – as part of the repository maintenance process.

By comparison, these properties are very similar to object definitions found using the Microsoft IE Dev Toolbar.

Application object:
App Add button object

Object properties discovered with Microsft IE Developer Toolbar:
IE Dev Toolbar properties

Note also that whatever works in VBScript for object handling works in QTP – but not necessarily vice-versa!


August 16, 2008  5:47 PM

Creating a Solid Test Automation Framework with QTP

Sentinel627 Greg Annen Profile: Sentinel627
IMAP4

HP’s QuickTest Pro can itself be used to build a test automation framework. Listed below are some of the characteristics you should take into consideration during the design phase.

Test automation scripts and functions should:

  • Be Reusable and Repeatable
  • Require Low Maintenance
  • Allow Unattended Execution
  • Support Platform Independence – IE 6 or IE 7, shouldn’t matter
  • Use Data Abstraction – Keyword-Driven Framework

HP QuickTest Pro scripts should use:

  • Environment Variables
  • Test execution ‘Driver’ to call ‘Actions’
  • ‘Action’ to call functions – i.e., function call for each action
  • Imported test data (Excel) – common format
  • Results reporting – QTP + Quality Center; Excel “Step Status”

VBScript functions can be:

  • Simple – Specific purpose; True / False or 0 / 1 return code
  • Complex – Multiple steps (e.g., login); “Fuzzy” return code that maps to business rules
  • Resilient – include Error Handling

Handling Objects in the Repository is done by:

  • Object changes/additions per app release
  • Replace with descriptive programming whenever possible
    • Functions using regular expressions (e.g., popup windows: IE 6 vs IE 7)
    • Web app objects usually classifiable by type, html tag, or class (QTP test object properties)
    • Reusable methods: look before you write!


Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to: