Automated integration testing sort of presents us with a paradox. The problem is that automated testing makes the most sense for large organizations where there’s a lot of testing to be done; however, these big enterprises tend to have large, complicated development environments where it tends to be hard to build automated tests for integration. So, it’s not quite a catch 22, but it is rather inconvenient that the more an organization needs their integration testing automated, the harder it’s going to be automate it. A growing open source project might have found the trick to cutting the knot – if its name doesn’t get in the way too much.
I sat in on a nice session this afternoon on an intriguing test automation framework concept from Anand Bagmar this afternoon at Agile 2013. Bagmar is a software quality evangelist and principal consultant at ThoughtWorks. He’s got a nifty tool that he created to ease some of the integration testing challenges he’s faced with large organizations. What it does is to create a system of Web services with configurable service contracts that testers can call to run automated tests. Right now the tools is really new and lacks important features to make it a commercial success with enterprise developers, but the concept shows promise.
Bagmar originally called his open source project TaaS – Testing as a Service – because the tool runs on its own server and a specialized TaaS client handles the service calls that let it automate testing. So it’s a classic client/server application where the testing happens on the server, as a service. But Bagmar dropped the phrase Testing as a Service and has shortened the projects name to simply TaaS because he doesn’t want to imply that TaaS is a fully provisioned, pay-per-use service. It’s actually a REST service that software quality teams can install, configure and run on their own server hardware.
From an on-premises server, Bagmar says, TaaS makes it relatively easy to configure automated integration tests that take various technology stacks and platforms and test them together as well as separately. The example use case Bagmar mentioned was Microsoft’s Outlook. This application has a main version that runs on the Windows operating system, but there are also versions for Mac, iOS, Android, Web clients and others. All those versions have to sync together properly, despite the fact that they may have been built with completely different technology stacks. Automating the integration tests is tough because no one integration testing tool is built to optimize testing in all of those stacks.
In fact, TaaS isn’t really built to handle them all on its own. TaaS is built to integrate the testing tools that work well in one stack with the testing tools that work well in the others. So to test that Outlook is syncing properly across all platforms, one could potentially automate a process that creates an email draft in one version, checks that each of the others are able to sync, modifies the draft and checks that the sync worked across all versions, and continues until all the tests are proven (or some fail and produce error reports).
Test writers program TaaS via yml contracts. The contract includes a timeout interval so if the tests get caught up somehow it won’t just blow up, it will time out after predetermined time has lapsed. (Bagmar used examples around sixty to one hundred seconds.) The contract is also where the test writer would parse input parameters and set the output parameters to be returned. That’s pretty much it on the server side. And the client side is even simpler.
But all that simplicity comes at a price. There’s a short list of things that TaaS is definitely lacking. Bagmar recognized each of these as important features that TaaS will need and called for developers to help out by contributing these features to the TaaS project. At the top of that list is missing error code messages. Right now the project lacks error codes which will be really necessary for serious testing and debugging. Then there’s security. TaaS was built under the assumption that it would be run inside the firewall and only with test data or well masked data. There’s also no current solution for service discovery; you just have to know where the service is running to be able to call it. Finally, TaaS only exists in its original Ruby version. Alternate versions in Java and/or .Net would really ramp up the project’s audience.
Still, running integration tests as a combination of technology specific tests rather than attempting to make a tool that works “good enough” for everything is a neat idea. I think it’s going to pick up traction in the near future. IT might not be TaaS, per say, but making integration testing sort of test tool agnostic may be a huge help for large organizations with complex architecture. That’s it for tonight. So until next time, remember to stay calm and carry a big stick.