Oct 7 2011   1:48PM GMT

SOASTA integrates with Selenium

Posted by: Yvette Francino
ALM integration, ALM, SOASTA, Continuous Integration

On October 4th, the cloud-based performance test organization SOASTA announced its integration with Selenium, the popular open source testing tool. This will allow for functional and performance test results to be combined for enhanced analytics, as well as the ability for results to be fed back into continuous integration servers such as Hudson and Jenkins.

But perhaps even more interesting for users are the extensions SOASTA has added to Selenium’s functionality, including a visual test creation environment that allows testers to create tests without traditional coding or scripting.

I spoke with Tal Broder, VP of Engineering at SOASTA, who said:

We have significantly enhanced the capability of Selenium in terms of recording the detection of what element was actually interacted with on the page. We added a visual test environment, which we already had for load testing, and we also enriched analytics. We believe with our offering, even though we are using all the power of Selenium for driving browsers, we have a much faster and easier test creation without having to write a single line of code.

This announcement comes on the heels of two other recent announcements in the ALM test tool market: Replay Solutions and Coverity. Both of these announcements also were about improved automated testing, enhanced analytics, ALM tool integration and feeding results back into a continuous integration tool. I asked Broder about what was driving this trend. He answered:

I think this is all driven by Agile and this whole DevOps movement where people want to build very, very often and release small chunks of code into the user community in a very fast and efficient way. They need the tools that will allow them to find bugs in an automated way and test the performance before you push it, or even after you push it, into production so that you can protect your users from functional problems, from performance problems, and I think that’s why we’re seeing a lot of automation in the industry. I think that trend will continue.

I asked Gartner analyst Tom Murphy to compare and contrast the three announcements. He explains that each tool catches bugs in different ways, but they are very complementary:

There are a number of different places to find defects or ways to find them. Coverity is focused on the analysis of source code to find defects, Replay is focused on identifying defects that occur while the application is running by capturing what is happening in the environment and SOASTA is just “reading” functional testing to their story line, which is a way to automate tests from a user perspective. So rather than looking at the source like Coverity, I use SOASTA (or say HP QTP, or others), to drive the application and monitor for deviations from the expected behavior. I use Replay Solutions while I run tests to provide a detailed recording to the developer so that when a defect is found they can identify what is going wrong faster (ie. I don’t have to reproduce the defect), and I can also use this in production to capture crash info, etc.  Now with their most recent product, ReplayLightning, there is more of a connection between what is happening at execution time, which is kind of a Dynamic Source Analysis rather than the Static Analysis that Coverity and others provide. This is important because in dynamic languages there are things you don’t know until run time.

In short, the tools are all complementary to each other.

Oct 5 2011   11:49PM GMT

Coverity announcement includes integrations with HP and other ALM tools

Posted by: Yvette Francino
Coverity, ALM, JavaOne, Static Analysis, Continuous Integration, ALM integration, HP

On October 3rd, Coverity announced Coverity 5.5, a new release of their development testing platform, along with integration with major ALM vendor HP. Coverity provides static analysis of machine code uncovering defects early in the development lifecycle.

Though code analysis is available for  Java, C, C++ and C# code, Coverity has a growing user base in the Java community. Coverity has a booth at this week’s JavaOne conference and attending are Jennifer Johnson, Coverity’s VP of Product Marketing and Zach Samocha, VP of Product Management.

I spoke with Johnson and Samocha about how the announcement will affect the Java community. Johnson explained the importance of seamless work flow that Java developers are looking for in ALM solutions.

What we’ve seen historically in the static analysis market is that if the technology doesn’t seamlessly fit into the developer work flow and slows them down, developers aren’t going to adopt the tool, especially in Java where they’re under extreme time-to-market pressure, [and doing] Agile development [and] iterative testing. They need to be able to test their code in a way that’s not intrusive.

We’ve created a series of integrations either into our platform or out into other development tools into the Java tool chain to help the Java workflow.

Integrations that Java users specifically will be interested in are those with the popular open source tool FindBugs, the Eclipse IDE, as well as with Jenkins Continuous Integration Server.

Samocha said they are receiving very positive customer reactions from the Java community, with their support of Android and mobile testing as well as their integration with FindBugs, Eclipse and Jenkins.

I think [users have wanted] the adoption of open source technologies within knowledge enterprises which has been a challenge so far, so it’s been a big push from the customer base and we’re getting really good feedback from the actual users at the JavaOne conference.

Samocha emphasized that Coverity 5.5 has introduced a big performance improvement with up to 10x improvements in analysis speed.

Coverity 5.5 also integrates with Visual Studio IDEs and, of course, the biggest integration announcement, HP’s ALM tool suite.

Gartner analyst Tom Murphy said of the HP integration:

For Coverity, it is really about the relationship with HP which is hopefully broader market access. From an HP perspective as they try to move into a stronger overall ALM market they need better connection to developers so this provides a nice connect from the developer world and view of code quality, now integrated into the overall view of quality.

And Theresa Lanowitz, Founder and Analyst at voke, inc. noted the integration of HP and Coverity as complementary:

ALM solutions are not a “one size/one vendor fits all” philosophy. Innovative technologies from a variety of vendors prove to be complementary in a number of areas for ALM vendors. The Coverity/HP ALM integration is a perfect example of that complementary effect. Coverity has an innovative solution that solves a classic computing problem. HP has an ALM solution that focuses on traceability throughout the lifecycle. The two vendors have technology that is completely complementary and delivers complete insight on quality throughout the lifecycle. Most importantly, the traceability of assets and the visibility of code quality prevent egregious issues and defects from reaching production and impacting the brand and the business.

When asked if this trend will continue, Lanowitz answered:

We should expect to see more ALM integrations with complementary technologies to solve some very difficult problems. IT organizations should be actively evaluating and adopting these complementary solutions. IT organizations should also make sure they are on the latest release of the tool sets they use. Recent releases have made significant advancements and will deliver strategic value to an organization when used effectively.

Sep 28 2011   9:39PM GMT

Replay Solutions announces diagnostics portal and APIs for replay

Posted by: Yvette Francino
ReplaySolutions, announcements, DevOps, Continuous Integration

On September 27th, Replay Solutions announced the ReplayLIGHTNING application diagnostics portal, an analytics front end to their ReplayDIRECTOR products. The company also announced ReplayCONTROL, a set of open API’s that will allow third party products to add replay functionality to their solution.

This is an example of the trends we’re seeing in ALM tools, bringing DevOps into the application lifecycle and allowing for diagnostic data to be displayed back into the development environment for debugging. The open APIs, too, will allow for the integration of tools, which is another area that development teams are looking for when they select ALM tools.

SSQ spoke to Larry Lunetta, president and CEO of Replay Solutions, and Jonathan Lindo, co-founder and Senior Vice President of Technology, about the announcement. 

“ReplayLIGHTNING is a lightweight monitoring solution that looks for critical events and then allows you to access some of the deep data that we capture in our recordings,” said Lindo. The data that is provided includes:

• Rapid application diagnostics around critical events
• Performance profiling data
• Memory leak location
• Code coverage information

This month SSQ has been specifically covering continuous integration solutions, so we asked Lindo about how the ReplaySolutions toolset played into this space. He answered:

A lot of times when you’re running CI it’s going to be recording test failures between builds.

It’s useful to look at the differences between two builds. What are the differences in not only test failures, but application behavior? How does my application behave differently? Did it turn more exceptions? Did it expose more logging? Was there higher database activity as a result of that code change? You can use the reporting that comes out of ReplayLIGHTNING to compare and contrast and generate differences or “diffs” between different builds that are passing through the CI system.
 

We also spoke to voke inc. founder and analyst Theresa Lanowitz about the announcement, who said:

Lifecycle virtualization is a new hub that delivers easy solutions to the age old problems of application development.

Through our research we see great value and immediate ROI in this technology, including virtual lab management, services virtualization, and automated defect identification solutions, such as Replay.

The days of struggling with elusive defects can be left as a challenge of the past. Leveraging tools like Replay that combine, automated defect identification, monitoring and rapid diagnostics empower teams with unprecedented information to eliminate even the most elusive defects.

Regardless of whether an organization uses a full ALM suite or no tools at all, all organizations should harness the power of lifecycle virtualization to modernize their practices.

Sep 15 2011   3:39PM GMT

Continuous integration step by step with Howard Deiner

Posted by: Yvette Francino
Continuous Integration, Howard Deiner, Agile 2011

We hear a lot about continuous integration these days, but some of us are still confused about what exactly CI is. Is it just a fancy way of doing builds? Does it mean you have to have all your regression tests automated? I got a chance to sit down with Howard Deiner at Agile 2011 and asked him more about continuous integration. He has written a series of four tips for us in which he explains the benefits of continuous integration and provides an implementation plan from beginning to end.

Check out this informative set of tips:

Continuous integration: Quality from the start with automated regression

Automating your release management processes with CI

Intro to integration: Automation from version control to deployment

DevOps: Adding database automation to your continuous delivery strategy

For a quick overview of what continuous integration means, take a look this video clip I took of Howard at the conference:

Jun 2 2011   8:23PM GMT

The complexities of developing Agile embedded software

Posted by: Yvette Francino
embedded, Continuous Integration, agile, Howard Deiner

Developing traditional software applications is difficult enough. Developing embedded software systems, applications that run on specialized devices, adds an additional layer of complication. In May, SearchSoftwareQuality.com published a series of articles by contributor Howard Deiner aimed at describing Agile embedded software development, some of the challenges that come along with those efforts, and how those challenges are best overcome.

* The fallacy of one size fits all for Agile embedded development

In this tip, Agile consultant Howard Deiner describes how Agile software development can and should be done with the iteration cycles using emulators or virtualized hardware so that the benefits of Agile will still be obtained, regardless of the hardware cycle.

* Seven deadly sins of embedded software development

In this tip, consultant Howard Deiner looks at how each of the deadly sins might be committed in embedded software development and suggests that by practicing Agile, organizations can avoid the nine circles of Dante hell, and instead bring their organizations to a state of celestial bliss.

* Embedded software test: To simulate or emulate, that is the question

Understanding the differences between simulator and emulator embedded software development tools is critical to effectively using them with the Agile engineering best practices of test first development and continuous integration. This article dives into the issue and provides some suggestions to follow.

* The role of continuous integration in embedded software development — better, faster and cheaper

* The motivation behind continuous integration in embedded software development

In this two-part series of tips, Deiner describes the benefits of continuous integration in embedded software development. By understanding how to best implement continuous integration, organizations will lower their risks and produce higher-quality products with a quicker time-to-market.

May 10 2011   9:38PM GMT

Kristan Vingrys from Thoughtworks: An Agile build pipeline

Posted by: Yvette Francino
Thoughtworks, STAREast, STAREAST 2011, Kristan Vingrys, Continuous Integration, continuous delivery

Another presenter that I was able to speak with at last week’s STAREAST conference in Orlando was Kristan Vingrys from Thoughtworks. Vingrys is the Global Test Practice Lead for ThoughtWorks. In his session titled, “The Agile Build Pipeline: A Tester’s Lessons Learned,” he describes the experience of Insurance Australia Group as they launched a new complex service requiring early integration and strong testing capabilities.

The project called for continuous integration and a build pipeline that allowed for production deployment on the same day the code changes were made with a high degree of confidence that there would be no regression bugs.

I was reminded of my conversation with Thoughtworks Studio’s Jez Humble last summer about continuous delivery and the difference between continuous delivery and continuous deployment.

As both Vingrys and Humble point out, it’s the customer’s decision about whether or not to flip the switch to deploy to production, but having an automated build process that allows for quick delivery can certainly be a benefit to the business.

Sep 23 2010   3:39PM GMT

Agile practices: Continuous integration, automation, and TDD

Posted by: Yvette Francino
Continuous Integration, TDD, software test

As I was researching material needed to write, Distributed agile: Fostering development collaboration without collocation, I had the privilege of interviewing two of the authors of the IBM Press book, “A Practical Guide to Distributed Scrum.” I was able to catch up with Elizabeth Woodward at Agile 2010 and pick her brain about her experiences working in distributed agile environments.

More recently, I had the pleasure of doing a podcast interview with Steffan Surdek. In the interview with Surdek, we spoke about three practices which were described in the book to be particularly valuable to fostering collaboration during a Sprint: continuous integration, test automation and test-driven development (TDD). Surdek describes each of these and their importance in software development.

I realized, as Surdek described these practices, that although they are often associated with agile development, they could just as easily be used in traditional software development and certainly they could be used regardless of your team distribution. I managed teams who used JUnit and other programming techniques to automate unit tests and integration tests many years ago. Though our overarching methodology was more traditional with phases and gate reviews, we still used practices to ensure quality was being considered and tested for from the beginning.

Quality gurus continue to argue the pros and cons of various methodologies, but one thing seems to be clear: the earlier we can find defects and get them fixed, the better. Regardless of the methodology you use, practices such as continuous integration, automation and TDD will help you test for quality throughout the lifecycle.

Apr 28 2010   9:38PM GMT

STAREast keynotes: Continuous integration and documentation in Agile

Posted by: Yvette Francino
STAREast, agile, CI, Continuous Integration

Agile practices, including continuous integration and documentation in agile environments, were the kick-off topics at STAREast in Orlando, Fla., today.

I’ve been sitting here in the second row of a ballroom at the STAREast conference with the Rebel Alliance. We’ve been listening to two keynote speakers, Jeff Payne and Elisabeth Hendrickson. 

Payne started his presentation — “You Can’t Test Quality into Your Systems” — talking about some of the changes we’re seeing in development, asking if they were fads or trends.  Fads come and go, but trends “change the way we work,” he said. Obviously, agile development is one of the trends that he mentioned and we are hearing about a lot in the industry.

“Continuous integration is HOT,” said Payne as he spoke about SecureCI, which is an open source tool which integrates several other opensource tools. Continuous integration is the “hub” that ties together development, test, release engineering and management.

Payne quipped that developers originally made up agile as “retaliation of quality people” to get auditors out of their hair. Then the developers found out that, indeed, they’re not off the hook for unit testing and documentation! There’s actually a lot of “rigor” in agile done right. Payne said that people often think that agile means “no documentation.” A common misconception is that you can’t use an agile approach when documentation is required due to regulation. However, if documentation is a requirement, then that documentation is included as a high-priority story in agile.  Payne gave an example of a project in which the documentation was stellar, impressing the ranks. When they asked about the methodology, they were told “agile!”

Hendrickson spoke next on “Agile Testing: Uncertainty, Risk, and Why It All Works.”

When describing documentation, Hendrickson hopped all over the stage imitating all the various documentation that needs to be updated in traditional environments. As she described her frustrations, she joked about a product she once tested using Excel as the documentation tool: “I found more bugs in Excel than in the product I was actually testing.” She described the documentation process as “exhausting.”

One of the concepts used to boost documentation efficiency, Hendrickson said, is Acceptance Test Driven Development (ATDD). With ATDD we get the shared understanding of what we’re building and can automate expectations.  Hendrickson talked of an “executable specification” saying: “We finally can actually do it. We have real evidence, executed running testing features.” ATDD provides leveraged documentation resulting in executable requirements.

Hendrickson described how much more quickly feedback is received with continuous integration and automated regression. “The automated system level documentation reduces the feedback latency to the time the developer checks in code until there is evidence that the code works.” 

Both keynote speakers seem to agree that they’re seeing usage of agile development growing rapidly. Learning how to take advantage of continuous integration and using documentation efficiently are two practices that they see adding to the success of agile projects.

For more STAREast coverage, check out these links:
James Bach - The buccaneer tester
STAREast keynotes: Continuous integration and documentation in Agile
STAREast: Is your software development organization agile?