Software testing: between edge cases and the critical path

Once you’ve been working for more than a few years, it becomes more and more difficult to avoid screwing up.

Of course, we all screw up, but I mean the kind of thing that can do a lot of damage to your standing with current and future employers.

Here’s the dilemma: stay in a job or at a company for too long, and you risk becoming set in your ways (whether you are or not); move around too often, and you become unreliable and fickle.

The variety of IT roles

One thing I like about working in IT is that you can work in different roles within the same industry, and each new role can give you a wealth of experience that you simply wouldn’t get if you stayed put.

I’m not talking about being a trainee developer, then a junior, then a “mid-range”, then a senior. I think a lot of people only look at the path ahead of them, and not the other paths they could take.

I strongly believe that to get ahead, you need to look beyond your natural path. This is why, in October 2004, I spent two years working as a test manager. Having worked for several years as a developer, it was a very new experience for me – but it was extremely rewarding.

I’d really like to see more developers working as testers, even just for a short time. Let me explain why…

What is the purpose of testing?

Over the years, many of the developers I’ve met have a fairly poor understanding of testing. The problem is, testing is a subject that a lot of IT staff think they know – when really, they don’t.

Testing is all about software quality. What is the purpose of testing? I already know what some people are thinking:

The purpose of testing is to check that the system works.

This is what I used to think, too. The problem with this statement is that it’s very difficult to determine if a system works. You may think it works, but in reality, failures are occurring that you don’t even know about. You then find them at some point down the line, and it’s a bloody nightmare to figure out how they occurred.

With the first approach, the tester tests to see if the system does what it is supposed to do. OK, great. It works. System goes live. How many times have you heard of massive problems occurring after a launch? Now, why is that?

Let’s try again. What is the purpose of testing?

The purpose of testing is to find faults.

With this second approach, the tester should still go through the core features of the system – for instance, a small online store might get the once over to ensure you can actually order something. But that’s the bare minimum you should be testing. Hopefully you’ll be testing a lot more…

When does testing begin?

This is the part that too many people get wrong. If the tester is only involved at the end of the project, they’re not going to be able to add a lot of value. But if they come in much earlier in the project, they can ask questions about key functionality and get some of the idiosyncracies worked out before they become bugs. This can be very tough to do when it’s a project you’ve only just been assigned to – but it’s easier if you have a well-written brief.

The subject of requirements discovery – requirements that come to light after the initial requirements gathering exercise – is a topic for another blog post. But in brief, it’s worth avoiding the discovery of unexpected requirements at a late stage in the process – or even post go-live. The cost to build in those changes can be astronomical.

Note that this does not necessarily apply to change requests – unless those requests are likely to radically alter the fundamental components of the system.

The skill of the software tester

If the tester wants to find faults, this is where the fun begins. The skill in testing lies in finding absolutely every bug you can get your hands on, and documenting them in such a way that each and every bug can be reproduced on demand.

Yes, a tester has  to be ruthless, and developers will really hate you if they’re not used to it. But listen up guys – testers represent a very, very important buffer between developers and users.

You would really know about it if your code went straight to live. Users are far less forgiving and far less descriptive in their bug reports. I’m only too familiar with “it doesn’t work – fix it!”. It’s not up to the users to provide detailed bug reports – it’s not where their expertise lies. They are skilled in using the system. They probably know it better than anyone else. It’s the tester who a developer relies on to report the bugs in the system.

Exhaustive testing: how much can a tester do?

I’ve seen both extremes. One approach is to mindlessly test everything, and continue doing so whenever a new release is deployed. This isn’t just a waste of time – it’s extremely dull and repetitive. Boring, repetitive tasks are much more likely to result in faults being missed. This is a good argument for automated testing, but that comes with its own set of challenges. I’ll come back to that in a future post.

The other extreme is to ignore everything except cross-browser issues, which many developers won’t catch as they often stick to one browser only. This isn’t a bad idea, but it shouldn’t be the only thing that gets tested.

Mitigating risks: the scope of software testing

All too many projects suffer from a lack of time, or a lack of people to get the job done. Conversely, some projects suffer from having too many people involved. The end result is that testing is often done at the end of the project, and it’s one of the first stages of a project to have time taken from it to get the other parts out on time.

When time is an issue, it may seem impossible to perform adequate testing. But if you had all the time in the world, how much testing would you do? Remember the approach – we’re looking for bugs. All software has bugs, so is testing ever really finished?

If time is an issue (and let’s face it, this is usually the case) we can only do our best. In testing, this means reducing the risk of a serious fault occurring in production. Major issues are far, far more costly to address once a system goes live than if they can be identified sooner, and addressed. Smaller issues are worth fixing too, but it’s the big issues that we really want to look for.

To find faults, we need to identify the areas that are within the scope of our testing, as priority 1 functionality. Continuing with the example of an online store, placing an order is likely to be critical; being able to upload a profile picture is not quite as important. But on the other hand, we do need to be on the lookout for potential security issues. For instance, it shouldn’t be possible to upload a malicious file as your profile picture.

It’s not just about finding issues with the core functionality. It’s worth talking to the developers to identify any features that are unusually complex or error-prone, as this means they can be targeted for some in-depth testing, and not just a cursory glance.

What should you do when time is tight?

That really depends on the project – but as a starting point, I’d suggest the following:

1. Exploratory testing: this is when a tester plays around with a site they haven’t tested before, without the use of test scripts. This can be hit and miss, but a skilled tester can find a lot of bugs in a relatively short timeframe.
2. Re-testing of fixed issues: of course, this does require some issues to be logged and fixed first. But having a clear focus for your testing – such as 5-10 issues that were fixed and are due to be released with the next version of the system – is a good way to maximise value when time is tight. Did those fixes introduce new bugs? Were the original issues actually resolved? At least you know what to look at – instead of perpetually testing and re-testing “absolutely everything”.

When does testing end?

It never really ends while the system is in use. If the system is closed, then it would be difficult to test it, for sure! Testing should continue after the system goes live. If time is tight – and it often is – why stop testing if you still have more to test?

The difficulty here is with systems that are built for a price – can you keep charging a customer for ongoing testing? It may seem like you can’t, but it could become part of an ongoing maintenance fee.

Being proactive in finding and fixing issues, rather than waiting for users to find them, means you’re more likely to receive detailed bug reports (because a tester will be writing them), and you’ll protect your company’s reputation because users will find fewer bugs in your products.

Of course, these things are difficult to quantify, which means that most systems won’t be tested once they go live – until further changes are made.

In summary

This was a fairly detailed overview of software testing, with various hooks that I’d like to cover in future posts. I hope you found it interesting. Let me know your thoughts in the comments below.