Difference between revisions of "Test Triage and Intermittent Test techniques"
(Created page with "Attendance * Tyler @macetw * Emil * @RobPark ==Test Triage Techniques== Often tests appear without clear reason, many commits into a single build. Unreliable machines or fram...") |
|||
| (4 intermediate revisions by one other user not shown) | |||
| Line 6: | Line 6: | ||
==Test Triage Techniques== | ==Test Triage Techniques== | ||
Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond? | Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond? | ||
| + | |||
| + | Approaches: | ||
| + | * Not enough information in the log | ||
| + | * Limit smoke test time | ||
| + | |||
| + | 1 team might be dedicated to triaging | ||
| + | * Build sheriff | ||
| + | * Assign person responsible | ||
| + | * File ticket | ||
| + | |||
| + | Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce? | ||
| + | |||
| + | ==Intermittent Failures== | ||
| + | CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing. | ||
| + | |||
| + | Blame on load average, up-of-timeout-wait, but no real fix. | ||
| + | |||
| + | Approaches: | ||
| + | * Ticket made of while a build intermittently failed. | ||
| + | * Assign test to quarantine | ||
| + | ** Management approval required | ||
| + | ** Analysis of time-management | ||
| + | ** Stuff moves gradually out of quarantine | ||
| + | |||
| + | Sidebar: Google test on the toilet | ||
| + | ''mental note, for macetw, we need more code-literature in our bathroom'' | ||
| + | |||
| + | One approach: | ||
| + | * Remove from test running | ||
| + | ** Would affect code coverage numbers | ||
| + | ** Not all cultures | ||
| + | * If still present in code 90 days later, may choose delete test scenario | ||
| + | |||
| + | |||
| + | Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things | ||
Latest revision as of 10:54, 24 August 2013
Attendance
- Tyler @macetw
- Emil
- @RobPark
Test Triage Techniques
Often tests appear without clear reason, many commits into a single build. Unreliable machines or frameworks, how do we respond?
Approaches:
- Not enough information in the log
- Limit smoke test time
1 team might be dedicated to triaging
- Build sheriff
- Assign person responsible
- File ticket
Assumed: Does everyone care that a build is red? Is a build/test cycle easy enough to reproduce?
Intermittent Failures
CI tool (TeamCity, Jenkins plugin) will handle assignment of an issue, a particular failing test, and there would be an owner of a certain issue, even if not always-failing.
Blame on load average, up-of-timeout-wait, but no real fix.
Approaches:
- Ticket made of while a build intermittently failed.
- Assign test to quarantine
- Management approval required
- Analysis of time-management
- Stuff moves gradually out of quarantine
Sidebar: Google test on the toilet mental note, for macetw, we need more code-literature in our bathroom
One approach:
- Remove from test running
- Would affect code coverage numbers
- Not all cultures
- If still present in code 90 days later, may choose delete test scenario
Sidebar: If high defect rate, but also high code coverage, tests may not be testing the right kinds of things
