What do Tester’s Find?
This section comes from two blog posts written by James Bach and Michael Bolton, respectively.
Some of you may be familiar with the 2003 movie "Holes". If not, I won’t spoil the plot for you, but the main point is that a number of youth are serving at a juvenile detention camp. While there, they are made each day to dig a hole that’s five feet round, primarily to "build character", but to also let the warden know if they "find anything interesting".
Yes, sometimes, software testing can feel like this. We do a lot of digging, and sometimes it is repetitious, but quite often, we are rewarded for our digging when we "find something interesting".
So what constitutes "interesting" to a tester? When we are looking for something in a software application, what are we likely to find?
Testers find bugs.
We look for anything that threatens the value of the product. "Quality means value to some person, at some time, which matters". –Jerry Weinberg
From James Blog: Some people like to say that testers find “defects.” That is also true, but I avoid that word. It tends to make programmers and lawyers upset, and I have trouble enough.
Example: a list of countries in a form is missing “France.”
Testers find risks.
We notice situations that seem likely to produce bugs. We notice behaviors of the product that look likely to go wrong in important ways, even if we haven’t yet seen that happen.
Example: A web form is using a deprecated HTML tag, which works fine in current browsers, but may stop working in future browsers. This suggests that we ought to do a validation scan. Maybe there are more things like that on the site.
Testers find issues.
An issue is something that threatens the value of the project, rather than the product itself.
Example: There’s a lot of real-time content on eBay. Ads and such. Am I supposed to test that stuff? How should I test it?
Testers find testability problems.
It’s a kind of issue, but it’s worth highlighting. Testers should point out aspects of the product that make it hard to observe and hard to control. There may be small things that the developers can do (adding scriptable interfaces and log files, for instance) that can improve testability. If you don’t ask for testability, it’s your fault that you don’t get it.
Example: You’re staring at a readout that changes five times a second, wondering how to tell if it’s presenting accurate figures. For that, you need a log file.
Testers find artifacts.
Also a kind of issue, but also worth highlighting, we sometimes see things that look like problems, but turn out to be manifestations of how we happen to be testing.
Example: I’m getting certificate errors on the site, but it turns out to be an interaction between the site and Burp Proxy, which is my recording tool.
Testers find curios.
We notice surprising and interesting things about our products that don’t threaten the value of the product, but may suggest hidden features or interesting ways of using the product. Some of them may represent features that the programmers themselves don’t know about. They may also suggest new ways of testing.
Example: Hmm. I notice that a lot of complex content is stored in Iframes on eBay. Maybe I can do a scan for Iframes and systematically discover important scripts that I need to test.
[Editors Note: These examples are placeholders and are the exact entries from the two blog posts. This list needs to be reviewed and potentially different examples used.]
WHAT TESTERS FIND (James Bach)
MORE OF WHAT TESTERS FIND (Michael Bolton)