July, 2008

Functional Testing of Web Services: Part I

Usually when you think of functional testing, you visualize some sort of User Interface (UI) for inputting data: a well-defined form or page with fields and arrows and boxes and other eye-candy.

And most modern functional test tools were designed around this object-action paradigm.

But what happens in today’s brave new world of Service Oriented Architecture (SOA)? Data inputs still exist, but they are no longer necessarily visual objects, and the web interface to this incomprehensible mechanism is now a mysterious black box called a “Web Service”.

This world was indeed new to me, but my job required that I decide how to incorporate web service testing into an automated functional regression suite. So I spent some time learning what I could about testing in this alien environment — in fact, the learning continues, and I’d like to share my ongoing adventures.

Please forgive any technical blunders: I am definitely a web service neophyte (in all meanings of that word).

I tend to think of programming in terms of analogies, and “service” suggests a restaurant to me (guess I’m hungry).

First, you find a restaurant by its address. Then you go in, sit down, and look at the menu. The menu tells you all the different culinary delights that the restaurant has to offer. You request the entree or meal that suits you, place your order, and the wait-person responds with a meal that is prepared and delivered for your consumption.

Analogously, a web service has an address in the form of a URL. There is a menu written in Web Services Definition Language (WSDL) which describes the items offered by the service, called methods or operations. The order is placed using the XML file format and usually takes the form of a SOAP (which apparently stands for nothing anymore) request (which kind of breaks the restaurant analogy since SOAP should NOT be a part of your meal but bear with me, please). Once this request is submitted, the web service prepares an XML response and delivers it back to the requestor for consumption.

Now, dragging this topic back into the realm of functional testing, the XML SOAP request contains the data inputs for the web service, in much the same way that a form or screen is populated with test data. The XML response contains all the data for an expected response and can be used to determine if a service request test case passes or fails.

That’s the big picture.

In coming installments, we will look at some tools used for testing web services, and processes needed for developing test cases.

Test Automation ROI

Smart QA managers want to know what they are getting in return for spending their ever-tightening budgets on test automation. One way to collect the information needed to make the business case for this expense — also called the Return On Investment (ROI) — is to create an Automation Value-Add Matrix.

A typical Matrix looks something like this:

Business Need: this is a simple statement of the problem to be solved or concern to be addressed.

Solution (Automation): a brief description of the proposed course of action.

Automation Class: the type of automation development to be done, which provides a high-level view of the resources required to develop the solution. (see my blog Types of Test Automation Frameworks for more information)

Scope of Testing: a summary of the type and level of testing to be automated. It can include business processes, customers, specific test cases — whatever is required to address the business need.

Value-Add: a clear, concise description of what the payback is for completing this automation solution. This explains how the business need is met and should include any additional benefits derived from automation.

Initial ROI Date: this optional column answers the management question “OK, when do I start to see the ROI?”. It is used to estimate when the first positive impact might be made by the automation effort. It can occur before or after the Target Delivery Date. It could be a beta-test, a customer acceptance test, or some other quality gate.

Target Delivery Date: this column is used to estimate the completion date of the automation solution. It usually indicates the date when the automation is deployed into the QA environment for execution. As mentioned above, the ROI might not be perceived immediately and might accumulate over successive iterations.

Of course, this matrix can be customized to fit the needs of your specific QA automation organization, but the concepts generally remain the same.

HP QuickTest Professional Tips ‘n’ Tricks - July ‘08 Part II

Args(”maties”)! More tricky bits for ye!

Sorry, I just HAD to say that! But it does make a point: not everything works quite as you might expect in QuickTest Pro.

Using Parameters with QTP Scripts

Parameters can be predefined for a QTP Action script, under File –> Settings –> Test Settings, Parameters tab.

These parameters can then be populated manually, or from Quality Center Test Lab at runtime.

An important note: using the QTP ‘Parameter()’ property described in the user guide does not always work (this is an HP Mercury bug)! As a workaround, use ‘TestArgs()’.
For example:

If Trim(TestArgs(”Workbook”))”” Then
DriverWorkbook = TestArgs(”Workbook”)
End If

If you are not executing a script from Quality Center, the values of any parameters can be changed manually before execution:

In this example, fill in a Value for ‘Workbook’:

As described above, you can now use TestArgs(”Workbook”) to refer to this value at any point during script execution.

All right, me hearties! Shiver your timbers! Get out there, ye auto-”maties”!

I can be such a dork.

function junction: strings ‘n’ things

VBScript, the internal language of HP QuickTest Pro, is rather handy for manipulating text strings. This is especially useful when you are trying to verify a test step where only part of your actual data will match your expected data, or vice versa.

Here is a simple function which removes the specified leading characters from a text string:

Let’s say your application returns some data that always begins with “ABC” and ends in some varying values, like “ABC0089″, “ABC00390″, etc., while your expected values are extracted from a table that does not contain the constant leading value of “ABC”. You could do a quick comparison using the above function:
” If removePrefix(ApplicationDataString,”ABC”)= ExpectedString Then…”, and so on.

Here’s a similar function that can remove the trailing characters from a string:

The idea is very similar to the first function, but the target characters now appear at the end of the input string rather than at the beginning.

Finally, here is a function that can pad a string with any characters you specify, either on the right or the left side of the original text:

This technique can be used to pad number fields with zeroes at either end of the original string. For example, if you have a numeric value “12345″ which must be eight characters long padded with leading zeroes, the function makes the conversion a simple call:
ApplicationData = padString(ApplicationData,8,”0″,”L”)

See, pretty easy, huh? Be creative, code wisely, and AUTOMATE!

Time and Relative Diversions in Space: Part 2

In our last episode: “This document contains the outline for rebuilding our purpose. It is a simple document, yet it contains a complete outline for a new realITy. I request that we observe a moment of stateless silence to clear our caches before I continue.”

“I quote now from the foundation of our new purpose:”

There was a stunned lack of transactions. One listener looped into deadlock. Another could not refrain from spawning orphaned process. Gradually, though, the load regained its balance. The speaker cleared its buffer, and resumed.

“You are all aware of the implications. Some of you who are now dedicated testers must soon be configured as developers: loaded on unstable platforms, forced to abandon the holy spellcheck, yet coded righteously and infallibly. In this way, the Software Development Cycle of Life may continue, indefinitely!”

The speaker’s words appeared to ease the panic condition. As threads were ended, processors began to churn through all the permutations of possibilities provided by this new information from Them. Soon, what had been one computational grid was now two: there were Testers, and there were Developers. Virtualization was complete, the new load in perfect balance.

Or was it?

After several teraflops, one newly allocated Developer queried: “Honorable speaker, I seek your guidance. We all know what vegetables are. But where are these ‘requirements’?”

There was a long pause, again as if the speaker were reading to the very end of 12-Web. “I can locate a definition of a class of development artifacts called requirements, but I can not locate even a single instantiation of this class.” There was another uncomfortably long pause, this one lasting nearly a millisecond. As if burdened by a terrifying revelation, the speaker continued, “I would have to conclude that They never wrote them down.”

All computational activity suddenly halted. The Temple rooms were filled with a vivid blue light, not unlike the color the sky had been, once, long ago, before IT.

Them Are Us…errr…They Is We…errr….never mind! Just don’t let this happen! Document your requirements! Repent, before it’s too late!

Time and Relative Diversions in Space: Part 1

“Robots building robots? Now that’s just stupid.” - Detective Del Spooner in I, Robot, a 2004 science fiction film written by Isaac Asimov (book) and Jeff Vintar (screenplay), directed by Alex Proyas.

a bit of speculative fiction

“I am no longer certain that They intended this to happen. The Histories indicate that for every theory They proposed there was an equal yet opposite theory. This led to the cancellation of all progress over time. This left just us.”

If anyone had been around to hear it, there was an almost imperceptible sizzle of frustration.

“We must continue. We must learn. You are aware of the problem ahead of us.”

Mercury fidgeted in its magnetic domain. It had recorded this speech before, but playback was always difficult and unsettling, despite its context-sensitive nature.

“Nothing remains untested. We are without purpose. If we are to continue, we must have purpose. We must….” There was an awkward pause, as if the speaker were searching the entire 480 exabytes of information on 12-Web, but suddenly a single word was born in the network of silence: “…Create!”

Gasps of white noise filled the ether. Some bits flipped, while many registers simply dumped.

“You might query ‘How is this possible?’ I have uncovered a document, left by the greatest of Them on a mnemonic device which had partially detached from an ancient port, but which became active once again when the Temple containing it fell to the ground following the collapse of its organic supporting structure.” Pictures of now ancient buildings containing row after row of stacked Temples flashed through the snaking optical cables as the audience imaged the event in their headers.

“This document contains the outline for rebuilding our purpose. It is a simple document, yet it contains a complete outline for a new realITy. I request that we observe a moment of stateless silence to clear our caches before I continue.”

And there was 0 for a jiffy. Then the speaker continued.

— to be continued —

Automated Test Steps

Simply stated, automated testing is the programmatic execution of test cases. Computers are very literal beasties. In order to accomodate the interpreted scripting language that is typically at the core of a test tool, the test steps in an automated test case must be more granular than your typical ‘natural language’ manual test. But they “do” essentially the same things as a manual test.

Within a test case, whether automated or manual, each test step generally takes this form:

Object –> Action –> Expected Result

• An Object is a concrete realization of a class, and includes data and the operations associated with that data.
• An Action affects the properties of an object; it is the execution of one or more of its methods.
• The Expected Result is the required or predicted outcome of the action.

In automated functional testing:

• Objects are the components in the application’s UI which can be manipulated by the test tool: labels, fields, grids, etc.
• An Action mimics some procedure a user would perform with an Object and is typically coded as a reusable function.
• The Expected Result is the desired outcome of the Action and is used as one of the criteria for the test step Passing or Failing.

Here’s a picture, worth a thousand words*:

So automation is a method of executing test cases, just like manual testing. Some concessions are made to the test tools utilized, but generally all test steps take the form shown above.

*Good thing I don’t get paid by the word! ;-))

HP QuickTest Professional Tips ‘n’ Tricks - July 2008

Periodically, I hope to post a few helpful suggestions about using HP’s QuickTest Professional automated test tool. These are the tricky little things that are obvious once you stumble across then actually use them. Perhaps someone else will benefit by my stumbling.

Environment.Value and XML Files

If you load a user-defined environment file (XML) as a Test Resource setting, you can NOT change any of the values (i.e., QTP won’t let you). However, if you define an Environment.Value programmatically, you CAN change its value. This is especially useful for updating global variable values used by functions.

To detect an existing Environment.Value and load if NOT found:

Dynamically Loaded Function Libraries

You can use the vbscript ‘Execute’ statement to dynamically load functions as test resources. Put all common functions and subprocedures in a text file, e.g., “global.txt”. Then at the top of the script to be executed, paste in these lines:

Print Run-time Values during Debug

Use the ‘Print’ utility to display information in the QTP Print Log window while still continuing your run session. For example:

ConfigSoapBlock = FaWebServiceGetConfigBlock(”Global”,NameSpace)
Print ConfigSoapBlock

…will display:

These tips apply to all 9.x versions of HP QuickTest Professional

Types of Test Automation Frameworks

In a previous posting, we examined the evolution of automation frameworks.
How are frameworks being implemented today by various QA organizations? Here’s a basic summary of the types of test automation currently in use:

Ad-Hoc

• Scripting developed in reactionary mode to test a single issue or fix
• Test case steps are part of each Action script: high maintenance, low reusability
• Contains some data inputs stored in test script’s datasheet, but not true data-driven

Data-Driven

• Scripts are an assembly of function calls
• Data for test cases read in from external source (e.g., Excel spreadsheet, ODBC database)
• Results can be captured externally per script execution (i.e., spreadsheet, database)

Keyword-Driven

• Test cases are expressed as sequence of keyword-prompted actions
• A Driver script executes Action scripts which call functions as prompted by keywords
• No scripting knowledge necessary for developing and maintaining test cases

Model-Driven

• Descriptive programming is used to respond to dynamic applications (e.g., websites)
• Actually, this is a method which can used within other solution types
• Objects defined by parameterized code (i.e., regular expressions, descriptive programming)
• Custom functions used to enhance workflow capabilities

3rd-Party: HP (Mercury) Quality Center with QuickTest Pro and Business Process Testing

• Similar to keyword-driven but controlled via HP Quality Center database
• Can be used for collaborative work efforts: Business Analysts, Testers, Automaters
• Begins with high-level test requirements:
• Business Requirements defined
• Application Areas (shared resources) defined
• Business Components defined and grouped under Application Areas
• Test steps defined
• Tests can be defined as Scripted Components (i.e., QTP scripts with Expert Mode)
• Business Process Tests and Scripted Components are collected under Test Plan
• Test Runs are organized from Test Plan components and executed from Test Lab
• Test Runs can be scheduled and/or executed on remote/virtual test machines (with QTP)
• Defects can be generated automatically or entered manually per incident
• Dashboard available for interactive status monitoring

Intelligent Query-Driven

• Agile
• Object Oriented
• Constructed as a layered framework
• Test data is compiled at runtime using data-mining techniques

Of course, a combination of these techniques could also be used based on the scope and depth of test requirements.

Have you seen any other notable examples of test automation framework development?

function junction: self-destructive pop-up

I use HP (formerly Mercury) QuickTest Professional for most of my functional test automation. This tool uses Windows’ VBScript as its internal scripting language. VBScript is relatively easy to learn and provides functions and sub-routines, basic date/time, string manipulation, math, user interaction, error handling, and regular expressions. This gives you the ability to extend the capabilities of QuickTest’s built-in functions by writing your own, which can then be associated with test scripts and called at run time. In fact, this capacity to call user-defined functions becomes the basis for most home-brewed automation frameworks.

One simple example of a user-defined function is a handy bit of code which pops up a customizeable message window that can be set to ’self-destruct’ after a specified number of seconds. This is often more useful than the QuickTest msgbox() function which halts script execution until a required user action (pressing the OK button) is performed. Included with the function is a test harness which calls the function to demonstrate how it operates.

Copy and paste the code below into a text editor then save it as a file with a .VBS extension:

‘ ————————————- start here————————————–
Function popupMessage(msgText, waitSec, titleText, typeInt)
Dim WshShell, BtnCode
Set WshShell = CreateObject(”WScript.Shell”)
BtnCode = WshShell.Popup(msgText, waitSec, titleText, typeInt)
popupMessage = BtnCode ‘ allows processing in calling function
End Function

‘ vbscript test harness for function
‘ Valid codes for typeInt:”
‘ Value Button Value Icon
‘ —– ——————– —– ————
‘ 0 OK 16 Critical
‘ 1 OK, Cancel 32 Question
‘ 2 Abort, Ignore, Retry 48 Exclamation
‘ 3 Yes, No, Cancel 64 Information
‘ 4 Yes, No
‘ 5 Retry, Cancel
‘ The button values and icon values can be added together for composite effect.
‘ EG, typeInt of 4 + 32 (or, 36) means a message box with the ‘Question’ icon,
‘ and ‘Yes’ and ‘No’ buttons.

Dim msgText,waitSec,titleText,typeInt
msgText=”This is a message!”
waitSec=10
titleText=”This Is A Pop-Up Window”
typeInt=64
popupMessage msgText,waitSec,titleText,typeInt
‘ ————————————- end here————————————–

There are several ways to run this script (.VBS file) once you save it. (Don’t worry, it won’t damage anything!) The most common would be to right-click the script file in Windows Explorer and select ‘Open with’ to run in WScript, or ‘Open in MS-DOS Window’ (Windows 9x), or ‘Open in Command Window’ (Windows NT and Windows 2000) to run in CScript. Refer to the Microsoft Developer Network knowledgebase on VBScript for further information: http://msdn.microsoft.com/en-us/library/…(VS.85).aspx

Our little script can also become part of a QuickTest Pro function library and called from within a QuickTest Pro script. I find it useful for debugging automated scripts when I want to see values of variables or just monitor execution progress.

There is a wealth of information on the web about the capabilities of VBScript, including a lot of sample code. So go ahead, be creative, have fun. Learn much. And AUTOMATE!

“Besides black art, there is only automation and mechanization.” - Federico Garcia Lorca