Thanks to a 7thSpace news post about an academic paper, An innovative approach for testing bioinformatics programs using metamorphic testing by Tsong Yueh Chen, Joshua WK Ho, Huai Liu and Xiaoyuan Xie, I was able to find a gem of a paper on software testing. Here’s the full article on metamorphic testing.
I’ve read several academic papers on software testing in the past, and as a general rule I really don’t like them. However, this one’s really worth reading. In the article the authors suggest the use of a new software testing technique called metamorphic testing to test bioinformatics programs. I’ve tested a couple of bioinformatics programs in the past and it’s some of the most difficult testing I’ve ever done.
In the paper, the authors begin by outlining the general problem:
In software testing, an oracle is a mechanism to decide if the output of the target program is correct given any possible input. When a test oracle exists, we can apply a large number and variety of test cases to test a program since the correctness of the output can be veriﬁed using the oracle. Without a tangible oracle, the choice of test cases is greatly limited to those special test cases  where the expected outputs are known or there exists a way to easily verify the correctness of the testing results. In particular, an oracle problem is said to exist when : (1) “there does not exist an oracle” or (2) “it is theoretically possible, but practically too difficult to determine the correct output” .
(For those of you who might not be familiar with the oracle problem, I recommend Doug Hoffman’s work in test oracles as a primer.)
After some discussion about why traditional techniques aren’t enough, the authors describe their new method of metamorphic testing (MT):
Instead of using the traditional test oracle, MT uses some problem domain speciﬁc properties, namely metamorphic relations (MRs), to verify the testing outputs. The end users, together with the testers or program developers, ﬁrst need to identify some properties of the software under test. Then, MRs can be derived according to these properties.
The article then reviews a couple of studies and talks about the limitations of the technique:
It should be noted that satisfying all test cases based on a set of MRs does not guarantee the correctness of the program under test. MRs are necessary properties, hence satisfying all of them is not sufficient to guarantee program correctness. This problem is, in fact, a limitation of all software testing methods. Nonetheless, the ability to systematically produce a large number of test cases should increase our chance of detecting a fault in the target program, and hence improve its quality.
Well worth the read.