Software Engineering - Software Testing Concepts

Software testing concepts

The fundamental rule of testing is that 'a successful test is one that causes the software to fail.

You can have a testing team who didn't write the software do this.

An agile solution would be to write the tests before the code. This helps to clarify requirements too.

Bugginess

The tradition is that there are 3 - 5 bugs per 100 lines of C code. Or it's like saying 3-5% of the code is buggy.

(approx) 90% can be found by debugging.

90% of the remaining ones can be found by testing.

90% of the last few can be found by testing really well. It's expensive to get to a 99.9% bug free  state though.

What can affect bugginess?

Good design = Less bugs

Application type. Concurrent systems tend to be buggier.

Language. Java is buggier than C (right!). C is buggier than Perl.

A new programmer will make less bugs than an experienced one.

Note: Programming in pairs reduces bugs... but you become half as efficient as there's two of you.

Can you tolerate bugs?

If you find a bug in Firefox, it probably isn't the end of the world. But what if the fuel management system in the harrier jet you happen to be flying has a bug? So basically you can't tolerate bugs in applications where people's safety is relying on it.

Triple redundancy

The Airbus A320 is a massive plane. Airbus said that the software would only fail once every 10^9 hours and no more frequently than that.

This is really high, 10^4.5 is more likely, so how was it done? They added a second computer with different software coded by a different team. In theory they will both fail at the same time once in 10^9 hours.

But which one is wrong? You need a 3rd computer that you can use to take a majority vote. This is triple redundancy.

It has some advantages:

• When one computer goes down you have backup systems


• Free testing: You find a big every time you see a discrepancy

Triple redundancy is vital when it comes to increasing the safety levels of systems where safety is a priority

Testing

You can't test EVERY possible situation, it just isn't viable.

If you have a method that takes an int, there are 4294967296 possibilities. And that's just when the program contains one int.

So what we do is test the more interesting parts of the search space. There are a few ways to split the testing:

• Split the space into areas where you might get similar results from similar testing. If 4/5 works, 7/10 probably does too


• Concentrate on the boundaries between different parts of the space. This is called boundary value analysis

Black / White boxes

Black box testing:

• When you test the software without access to the code


• This means tests are written without preconceptions about how the code is going to work


• The tests stay valid even if the code is changed

White box testing:

• As you may have guessed, this is testing with access to the code


• Means you can do more tests


• When doing tests you can apple more pressure to the parts of the code that look like they might 'break'

Testing Types

• Unit testing: Test one unit at a time. 1 unit = 1 class in OO


• Integration: testing that the parts of a system work together correctly


• Regression / Smoke: Making sure you haven't broken the application


• System: Checking the system works in the context it's needed


• Alpha, beta: Testing with users


• Acceptance: Getting money for your software

Unit testing

This is quite simple, but don't forget that most classes rely on other classes (inheritance etc).

Boundary Value Analysis is good here.

You can be quite confident that a class is bug free. JUnit (not to be confused with G Unit is good for testing Java classes)

Integration testing

This is harder than unit testing because units are well defined whereas with integration testing it's up to you to decide where to start and stop testing.

You should concentrate on the critical features and make sure the use cases work.

Regression

This is more important after making significant changes during testing. You're just checking that you didn't break the original functionality.

Often done in a daily build of the system. Might be automated.

System

Varies on the context the system operates in. It can involve different OSs, hardware, etc.

You need to check the documentation as well though. Some things that work in a development environment might fail in a customer environment.

Alpha and beta tests

Alpha is with a small group and the developers are present. Beta testing is done remotely with a bigger group who are asked to submit bug reports.

Acceptance

(This is also the name of the 5th stage of grief. Don't mix them up.)

If a SW product's customers are large but a small group, they might specify a set of tests the software must pass before they pay. There are some problems with this:

• Users don't know in advance what they want


• Managers and end users have different views


• If you only focus on passing the acceptance test you might miss bigger issues