Talk:How much testing is too much?

The session began with a brief presentation from Zhimin Zhan from AgileWay. Zhimin helped answer the question about how to write functional web tests that are easy to write, but more importantly, easy to maintain. It was recognised that one of the significant barriers to writing lots of automated web tests was the effort involved in maintaining these tests as the code changed over time. Therefore, in order to avoid automated web tests from becoming more trouble than they were worth, web tests needed to be easily written and maintained, and preferable by business users and other non-technical staff. Zhimin showed how these two issues can be addressed.

First, to enable non-technical staff to have the ability to read and write functional tests, it helps to use a scripting language such as Ruby that does not require compiling and can easily be edited and saved in any text environment.

Second, to allow the tests to be easily maintained, the key is to refactor the various actions and assertions into methods that exist inside a "page" object. It helps to use recording tools such as iTest2 that can generate the script. But the script generated by the recording tools should not simply be pasted into a very large method and saved. Instead, the script should be dissected and refactored into various methods that represent the actions available on the page. This has several positive effects: it makes it easier to re-use previously recorded code to perform similar actions; it makes it easier to maintain multiple test cases when a change occurs on the page; and, significantly, it makes the body of tests much more readable, allowing test writers to think more about the actions and assertions they are trying to test, instead of being bogged down in the detail of achieving it.

Soon after this presentation, the discussion moved into a debate about the issues to consider when allocating time and effort to the various levels of testing, specifically: unit tests, integration tests, and functional tests.

It was recognised that all levels of testing were important. Some participants had experienced spending too much time getting high levels of unit tests and integration tests, only to find that when the deadlines arrived the code wasn't working at the functional level. Others has experienced too much focus on functional testing, which itself was very time consuming and expensive to provide confidence of all the many scenarios.

So then it comes down to just how much time the business wants to devote to testing. In some instances it may be more important to get a product to market early, than it is to release bug free code. However, in other cases, the consequence of releasing even minor bugs may be a very costly. So the business really must decide what level of confidence they want in their product and how much they are willing to invest to achieve that level of confidence.

This brings us to the next point: the business needs to be well informed about the effort required to achieve certain levels of confidence and reliability. It can help everyone involved in the project if the developers can effectively estimate the development effort required to achieve various levels of confidence in the code. If the business can be presented with figures, and they know that can have , for example, 90% chance of code working 90% of the time, then they can make better decisions about how much time and money they invest in code quality. Also, being up-front about the potential consequences of short-cutting testing can help spread the risk between the developers and the business.

Also discussed:
- code coverage should not be used as the primary indicator for quality testing.
- testing should focus on the important/critical areas of code.
- if test time is reduced, then all levels of testing should be reduced evenly, rather than simply eliminating one level (eg. unit tests, or web tests)
- exploratory testing may be good to save time, but requires skilled knowledge of the product being tested
- use case scenarios can help highlight the important areas to test
- the code may only be as good as the tests, but who is checking the tests? Business should get involved in approving (or even writing!) the tests.
- feedback loop time is important. can be hard for testers to keep up with the developers unless they are part of the team and involved at the beginning of the project.