Analyzing Testability, Part 1.

A cornerstone of Science is the Experiment. The experiment serves to test a hypothesis which is an explanation of some phenomenon that is yet to be accepted as scientific knowledge. For it to be a scientific hypothesis it must be falsifiable or testable as opposed to the hypothesis “In the beginning, God created the heavens and the earth”.  A typical formulation is the If-then shaped hypothesis. Such as “If I raise the temperature of a cup of water, then the amount of sugar that can be dissolved in it will be increased.” In software we have unit tests which serves a similar purpose, that is, produce trustworthy knowledge of the functional behaviour of the software under test. Imagine a test named Abs_should_return_positive_number(). The hypothesis in this case might be “If i apply the function Abs with the argument of -2 the return value will be 2”. Generalising this statement will result in the more interesting “For any input x, if i apply the function Abs the return value will be the absolute value of x”. This is expressible in several unit test frameworks as a parameterized unit test. E.g JUnit:

public void Abs_always_returns_absolute_value(int x)

Sadly it’s left to the developer to supply the actual values. Automatically supplying the values is an area of active research and a few tools have been built such as Pex from Microsoft Research.
An important aspect of a controlled experiment is the need to isolate the system under test and the ability to observe output or effects. Let’s start with the observability of methods. The following 4 methods differs fundamentally in how we can observe their behaviour.

public int Abs(int x)
public int Next()
public void Print(object x)
public void MutateSomething()

Abs lets us easily observe the result from supplying a parameter x whose creation is under our control. This lets us easily setup different tests and verify the expected output. Looking at Next we see that it lets us observe the result, but without control of its context, such as a class or module, we have a difficult time of distinguishing between different test runs or experiments. This leads us to Print where no result is directly observable, but we may be able to observe sideeffects through the supplied parameter x. If you prefer verbose, complicated mocking frameworks to exercise your coding skills this is the shape of methods you should prefer. The last method MutateSomething is unobservable on its own.

In conclusion, prefer methods with well defined inputs and outputs, in the parts of your software where knowing the important parts of the functional behaviour should be cheap and easy to come by.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s