Life Imitates Code. Essays by Ka Wai CheungEssays by Ka Wai Cheung from around the web.

Testing ● October 2014

The hard thing about testing

Why are bugs in software inevitable? One of my arguments is that knowing what to test is one of the most difficult parts of testing itself. Bugs creep up in the areas we never thought to look. As software gets more complex, those undiscovered areas get bigger and more disparate.

I recently had the privilege of speaking about this topic at the QA Summit for Digital Marketing. I wanted to summarize this particular argument here, alongside a few of the slides from my session.

You can listen to the entire session and view the slides here.

It’s easiest to see the loopholes in our test plans by looking at an ostensibly obvious solution to a problem. So, let’s get away from software for a moment altogether.

A lightbulb moment

Suppose we are aliens from a far-away planet and we’ve finally decided to land our UFO (for us aliens, it refers to an Ultra-Fast Object) onto planet Earth. As all UFOs do, we land near some remote farmhouse in a cornfield in Iowa. We exit our aircraft, quietly head into the barn, and come upon a simple invention that we’ve never quite seen before: The lightbulb and switch.

We observe how the light bulb works in relation to the switch. What are the behaviors we notice? Flip the switch on; the lightbulb glows. Flip the switch off; the lightbulb shuts off. We can screw in the lightbulb and take the lightbulb out. Through centuries of advancement in our own home planet—you know…full of robotic maids, self-cleaning homes, and invisible paint—it never dawned on us to create a switchable light source!

We take copious notes on this novel idea before escaping back to our home planet. The next morning, after a generous helping of alien pancakes and coffee, we get started on replicating this great find. (At this point, let’s just assume we’ve got all the other necessary tools, like electricity, screws, and filaments, to build the components of our alien light bulb and switch. We’ve got robotic maids after all!)

After a few hours, we’ve come up with a perfect replica—we think. Here’s how we confirm that our lightbulb and switch works just like the one we saw outside of Sioux City the day earlier:

At this point, we’ve solved it! Our lightbulb and switch work as it was intended to! Or, has it? Are there scenarios we may be missing?

Here’s a more rigorous set of tests to perform:

The ridiculous isn’t so ridiculous

While you may think that some of these potential what-ifs sound ludicrous, they could fail. And if they do, there is a fathomable explanation as to why a seemingly obvious result doesn’t actually happen in the way we expect.

In software, these at-first-seemingly-ridiculous cases happen so much more frequently. A flawed conditional, a mis-scoped variable, an uncaught exception—there are too many little mistakes we make in software that can lead to very odd behavior. As developers, we need to get out of the tunnel-vision mindset of testing (the on/off scenario), and start uncovering the “unlikely” corner cases that happen quite often in our work.

Originally published Oct 15, 2014 at DoneDone. Go to the next essay in Testing, “Want fewer bugs? Test these five cases”.