On October 4th, the cloud-based performance test organization SOASTA announced its integration with Selenium, the popular open source testing tool. This will allow for functional and performance test results to be combined for enhanced analytics, as well as the ability for results to be fed back into continuous integration servers such as Hudson and Jenkins.
But perhaps even more interesting for users are the extensions SOASTA has added to Selenium’s functionality, including a visual test creation environment that allows testers to create tests without traditional coding or scripting.
I spoke with Tal Broder, VP of Engineering at SOASTA, who said:
We have significantly enhanced the capability of Selenium in terms of recording the detection of what element was actually interacted with on the page. We added a visual test environment, which we already had for load testing, and we also enriched analytics. We believe with our offering, even though we are using all the power of Selenium for driving browsers, we have a much faster and easier test creation without having to write a single line of code.
This announcement comes on the heels of two other recent announcements in the ALM test tool market: Replay Solutions and Coverity. Both of these announcements also were about improved automated testing, enhanced analytics, ALM tool integration and feeding results back into a continuous integration tool. I asked Broder about what was driving this trend. He answered:
I think this is all driven by Agile and this whole DevOps movement where people want to build very, very often and release small chunks of code into the user community in a very fast and efficient way. They need the tools that will allow them to find bugs in an automated way and test the performance before you push it, or even after you push it, into production so that you can protect your users from functional problems, from performance problems, and I think that’s why we’re seeing a lot of automation in the industry. I think that trend will continue.
I asked Gartner analyst Tom Murphy to compare and contrast the three announcements. He explains that each tool catches bugs in different ways, but they are very complementary:
There are a number of different places to find defects or ways to find them. Coverity is focused on the analysis of source code to find defects, Replay is focused on identifying defects that occur while the application is running by capturing what is happening in the environment and SOASTA is just “reading” functional testing to their story line, which is a way to automate tests from a user perspective. So rather than looking at the source like Coverity, I use SOASTA (or say HP QTP, or others), to drive the application and monitor for deviations from the expected behavior. I use Replay Solutions while I run tests to provide a detailed recording to the developer so that when a defect is found they can identify what is going wrong faster (ie. I don’t have to reproduce the defect), and I can also use this in production to capture crash info, etc. Now with their most recent product, ReplayLightning, there is more of a connection between what is happening at execution time, which is kind of a Dynamic Source Analysis rather than the Static Analysis that Coverity and others provide. This is important because in dynamic languages there are things you don’t know until run time.
In short, the tools are all complementary to each other.
SSQ spoke with Jeff Kuligowski, who recently became Senior Vice President of Sales and Marketing at MobileCause, a Web service that enables mobile giving, engagement, CRM and donor communication for non-profit organizations. Using their Software as a Service platform, MobileCause has facilitated thousands of campaigns that allow donors to give to non-profit agencies via text message pledges as well as on-the-spot event donating.
Kuligowski explained the importance of testing in regard to non-profit brand management:
In particular, when you’re looking at the non-profit marketplace as your customer base, testing becomes very important. I equate to sort of being the quarterback of a football team; you probably get too much credit when everything is going well, and you get too much blame when it goes bad. The same case is true for the brand of a non-profit. If somebody has a bad experience interacting with the organization through a tool—like a mobile donation tool—the brand is actually the one that gets the blame. And the same thing is true when you do a good job; you create value and add to the brand.
Testing informs many of the decisions made at MobileCause. For example, they chose the SaaS model to avoid the slippery slope of testing on the numerous available devices. The drawback to delivering services in this manner is that they do not take advantage of the many capabilities and aesthetically novel features that new smartphone models have to offer; their site is simple and straightforward.
However, the advantage is reliability. Containing the software to one manageable domain ensures quality, availability and integrity without running the risks inherent in running applications on multiple, frequently changing devices. This consistency of quality is imperative for a sector that relies so heavily on reputation and positive image.
On October 3rd, Coverity announced Coverity 5.5, a new release of their development testing platform, along with integration with major ALM vendor HP. Coverity provides static analysis of machine code uncovering defects early in the development lifecycle.
Though code analysis is available for Java, C, C++ and C# code, Coverity has a growing user base in the Java community. Coverity has a booth at this week’s JavaOne conference and attending are Jennifer Johnson, Coverity’s VP of Product Marketing and Zach Samocha, VP of Product Management.
I spoke with Johnson and Samocha about how the announcement will affect the Java community. Johnson explained the importance of seamless work flow that Java developers are looking for in ALM solutions.
What we’ve seen historically in the static analysis market is that if the technology doesn’t seamlessly fit into the developer work flow and slows them down, developers aren’t going to adopt the tool, especially in Java where they’re under extreme time-to-market pressure, [and doing] Agile development [and] iterative testing. They need to be able to test their code in a way that’s not intrusive.
We’ve created a series of integrations either into our platform or out into other development tools into the Java tool chain to help the Java workflow.
Integrations that Java users specifically will be interested in are those with the popular open source tool FindBugs, the Eclipse IDE, as well as with Jenkins Continuous Integration Server.
Samocha said they are receiving very positive customer reactions from the Java community, with their support of Android and mobile testing as well as their integration with FindBugs, Eclipse and Jenkins.
I think [users have wanted] the adoption of open source technologies within knowledge enterprises which has been a challenge so far, so it’s been a big push from the customer base and we’re getting really good feedback from the actual users at the JavaOne conference.
Samocha emphasized that Coverity 5.5 has introduced a big performance improvement with up to 10x improvements in analysis speed.
Coverity 5.5 also integrates with Visual Studio IDEs and, of course, the biggest integration announcement, HP’s ALM tool suite.
Gartner analyst Tom Murphy said of the HP integration:
For Coverity, it is really about the relationship with HP which is hopefully broader market access. From an HP perspective as they try to move into a stronger overall ALM market they need better connection to developers so this provides a nice connect from the developer world and view of code quality, now integrated into the overall view of quality.
And Theresa Lanowitz, Founder and Analyst at voke, inc. noted the integration of HP and Coverity as complementary:
ALM solutions are not a “one size/one vendor fits all” philosophy. Innovative technologies from a variety of vendors prove to be complementary in a number of areas for ALM vendors. The Coverity/HP ALM integration is a perfect example of that complementary effect. Coverity has an innovative solution that solves a classic computing problem. HP has an ALM solution that focuses on traceability throughout the lifecycle. The two vendors have technology that is completely complementary and delivers complete insight on quality throughout the lifecycle. Most importantly, the traceability of assets and the visibility of code quality prevent egregious issues and defects from reaching production and impacting the brand and the business.
When asked if this trend will continue, Lanowitz answered:
We should expect to see more ALM integrations with complementary technologies to solve some very difficult problems. IT organizations should be actively evaluating and adopting these complementary solutions. IT organizations should also make sure they are on the latest release of the tool sets they use. Recent releases have made significant advancements and will deliver strategic value to an organization when used effectively.
Recently, SSQ spoke with Marc Schulman, an application developer with MultiEducator, Inc., to learn more about new iPad apps that enable students to study American history through their mobile devices.
These apps allow for interaction with the material in ways that text books simply can’t match. Students can access in-depth historical documents, texts and visual aids through these apps, which provide videos, photos, music and other artifacts, appealing to the various learning styles of students. While traditional text books are costly, physically heavy and feature permanent print, apps like History on the Go have a different pricing model, are contained within the user’s one mobile device and can be corrected or updated instantaneously.
MultiEducator has released applications on the Civil War and the Constitution, and this summer announced the release of their new Revolutionary War app. These apps provide multimedia presentations of the events, battles, documents, historical figures, related paintings, biographies and other relevant information from these historical eras to help students engage with history and learn through a multi-faceted medium.
According to Marc Schulman:
When you use the multimedia products we have and will be developing, you can do multiple modalities in terms of learning, in the same place. You can have the written text, for kids who do best by reading; you can have a visual presentation of the same material for those kids who do best with the visual or audio learning. You can have kids do interactive projects for kids who do best with interactive methods. You have the ability to have the multiple modalities of learning in one place, on one device. I think that will transform the learning opportunities for kids.
In terms of testing these types of applications, Schulman explained that Apple has a strong security system in place, which facilitates app generation for developers. After putting apps through the rigorous process, the security is for the most part taken care of. At this time, History on the Go apps are available only through Apple.
For more information on mobile application testing, check out these related articles on SSQ:
“When facing an Agile transition, QA managers get worried. They don’t see a job description for a ‘QA manager’ in the new Agile organization,” writes Agile expert Lisa Crispin in her tip, Transitioning to Agile development: What about quality assurance?
Crispin warns against eliminating QA departments, giving several benefits of keeping QA teams, yet still eliminating a silo mentality, and instead working using a “whole-team approach.” Crispin further expands upon this concept with a two-part series in which she explains the meaning of a “whole-team approach” and follows up with benefits and examples from her own personal experience.
“Above all, realize that successful Agile transitions don’t happen overnight. It takes years for the new teams to gel, for communication and collaboration among different roles and specialties to develop,” she advises.
One approach to gaining acceptance with organizational change is to use the KAA model: Knowledge, Attitude and Action. In Change management: Agile adoption with knowledge, attitude and action, I describe how this model can be used to help manage an Agile transition.
What are your thoughts? Has your organization recently gone through an Agile transition? What challenges did you face?
On September 27th, Replay Solutions announced the ReplayLIGHTNING application diagnostics portal, an analytics front end to their ReplayDIRECTOR products. The company also announced ReplayCONTROL, a set of open API’s that will allow third party products to add replay functionality to their solution.
This is an example of the trends we’re seeing in ALM tools, bringing DevOps into the application lifecycle and allowing for diagnostic data to be displayed back into the development environment for debugging. The open APIs, too, will allow for the integration of tools, which is another area that development teams are looking for when they select ALM tools.
SSQ spoke to Larry Lunetta, president and CEO of Replay Solutions, and Jonathan Lindo, co-founder and Senior Vice President of Technology, about the announcement.
“ReplayLIGHTNING is a lightweight monitoring solution that looks for critical events and then allows you to access some of the deep data that we capture in our recordings,” said Lindo. The data that is provided includes:
- Rapid application diagnostics around critical events
- Performance profiling data
- Memory leak location
- Code coverage information
This month SSQ has been specifically covering continuous integration solutions, so we asked Lindo about how the ReplaySolutions toolset played into this space. He answered:
A lot of times when you’re running CI it’s going to be recording test failures between builds.
It’s useful to look at the differences between two builds. What are the differences in not only test failures, but application behavior? How does my application behave differently? Did it turn more exceptions? Did it expose more logging? Was there higher database activity as a result of that code change? You can use the reporting that comes out of ReplayLIGHTNING to compare and contrast and generate differences or “diffs” between different builds that are passing through the CI system.
We also spoke to voke inc. founder and analyst Theresa Lanowitz about the announcement, who said:
Lifecycle virtualization is a new hub that delivers easy solutions to the age old problems of application development.
Through our research we see great value and immediate ROI in this technology, including virtual lab management, services virtualization, and automated defect identification solutions, such as Replay.
The days of struggling with elusive defects can be left as a challenge of the past. Leveraging tools like Replay that combine, automated defect identification, monitoring and rapid diagnostics empower teams with unprecedented information to eliminate even the most elusive defects.
Regardless of whether an organization uses a full ALM suite or no tools at all, all organizations should harness the power of lifecycle virtualization to modernize their practices.
What do you do once you’ve mastered Scrum? How about incorporating some Lean practices using Kanban into your methodology? At least that’s what appears to be happening more and more as organizations continue to look for ways to improve their software development processes. I’d heard quite a bit about this trend towards mixing lean methods such as Kanban with Agile frameworks such as Scrum (some call it ScrumBan) at Agile 2011.
Last week, a group of leaders from Oppenheimer Funds came to speak at our local SQuAD (Software Quality Association of Denver) meeting. This organization seems to be early adopters of the latest industry recommendations, working at continually evaluating and improving their current software development processes. They have spoken to the SQuAD group before about their Agile adoption and transition program. This month they talked to us both about their A/TDD (Acceptance/Test Driven Development) processes, as well as how they are incorporating Kanban and lean concepts into their methodologies.
In the following video, you’ll hear a short explanation from Oppenheimer Funds’ Doug Huffman about how and why they are using Kanban.
[kml_flashembed movie="http://www.youtube.com/v/T3VFyeJkCho" width="425" height="350" wmode="transparent" /]
A team from Oppenheimer Funds has come to speak at the monthly Software Quality Association of Denver (SQuAD) meetings a couple of times, telling us about some of their early adoption experiences. At the September meeting, we found out both about their incorporation of lean concepts into their software development processes using Kanban, as well as what they are doing with Acceptance test-driven development and [unit] test-driven development.
In both cases, test-driven development involves writing tests before the code. Writing the tests up front (whether at the acceptance level or using code at the unit level) will help in creating the design of the code. Any tests will fail when first written, but once the code has been designed and written, you will know they have been written correctly when the tests pass. This provides a set of automated regression tests then that can be used in your continuous integration process with each build.
Below you’ll find a video clip from Thom Vaught of Oppenheimer Funds explaining the concept. We have plenty of coverage of this popular technique on SSQ, and I have listed a few tips you may want to read to learn more.
[kml_flashembed movie="http://www.youtube.com/v/jM5vL4GajM4" width="425" height="350" wmode="transparent" /]
Effective test-driven development and continous integration
We hear a lot about continuous integration these days, but some of us are still confused about what exactly CI is. Is it just a fancy way of doing builds? Does it mean you have to have all your regression tests automated? I got a chance to sit down with Howard Deiner at Agile 2011 and asked him more about continuous integration. He has written a series of four tips for us in which he explains the benefits of continuous integration and provides an implementation plan from beginning to end.
Check out this informative set of tips:
For a quick overview of what continuous integration means, take a look this video clip I took of Howard at the conference:
[kml_flashembed movie="http://www.youtube.com/v/ba_iJhbTpBc" width="425" height="350" wmode="transparent" /]
At the Agile 2011 conference, plenty of ALM vendors were available to talk about their latest products and enhancements. Trends in ALM include broadening of ALM, both at the front and back ends of the development lifecycle. On the front end, vendors are including more support for portfolio management and idea management. On the back end, we’re seeing more focus on release management, DevOps, and maintenance. Continuous integration and automation of the release process continue to get attention, and, in fact, SSQ is focusing on continuous integration and automation as our primary content theme for September.
Lean development is also getting a lot of attention, and vendors are looking at including Kanban support as part of their wares.
In the video below, Forrester analyst Dave West speaks about what he sees as ALM trends, including more support for cloud computing and mobile development.
[kml_flashembed movie=”http://www.youtube.com/v/keQIwVbOAVU” width=”425″ height=”350″ wmode=”transparent” /]
How about you? What enhancements would you like to see from ALM vendors? What trends are you seeing?