Fragile Test

Less Fragile Tests

Tests Asserts Component Integration Functional UAT Non-Functional Perf Security Usability

See ISO 9126

Good Tests

   Pass Repeatedly

Timing (e.g. Selenium) External Dependencies External Systems Reliability Availability Data Set Values Incorrect State Heavy Mocking Utility to build mocks Easy switching mock to real

Bad Test Hidden cause of failure Visible Replay using TimeSnapper Visible Replay Dump state data at failure

Tool advice Time Snapper EasyMock Pretend Classes DBUnit gets prod snapshot regularly (security issues: scrubbing) Table dependancies makes it a significant task

Data Domain (name, addr, etc class pulled from DB (abstracted data from test) Data Driven Approach Fitness Integration Testing

Strong correlation between importance of tests depending on the types of testing and what kind of thing you are doing (Product type development = high focus on test vs bespoke = low focus

Suggestion Use a virtual machine, setup - refresh test machine using VMWARe to simulate MSG based, 20 mins business transactions

Test Data Repository Used keyword substitution to create the real data

Typical Data Edge / Border cases, Exception testing

Broken windows Continually failing, low priority tests/defects cause a 'Broken windows' effect which degrades the value of CI.