Software testing foundations - testing in the lifecycle

Testing in the Lifecycle
Software Testing Foundations
1 Principles 2 Lifecycle
4 Dynamic test
techniques
3 Static testing
5 Management 6 Tools
1
Contents
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
Lifecycle
1 2 3
ISEB Foundation Certificate Course
4 5 6
2
V-Model: test levels
Integration Testing
in the Small
Integration Testing
in the Large
System
Testing
Component
Testing

Acceptance
Testing
Code
Design
Specification
System
Specification
Project
Specification
Business
Requirements
3
Tests
Business
Requirements
Tests
Project
Specification
Tests
System
Specification
Tests
Design
Specification
Tests
Code
V-Model: late test design
Integration Testing
in the Small
Integration Testing

in the Large
System
Testing
Component
Testing
Acceptance
Testing
Design
Tests?
“We don’t have
time to design
tests early”
4
TestsTests
Business
Requirements
TestsTests
Project
Specification
TestsTests
System
Specification
TestsTests
Design
Specification
TestsTests
Code
V-Model: early test design
Integration Testing
in the Small

Integration Testing
in the Large
System
Testing
Component
Testing
Acceptance
Testing
Run
Tests
Design
Tests
5
Early test design
n test design finds faults
n faults found early are cheaper to fix
n most significant faults found first
n faults prevented, not built in
n no additional effort, re-schedule test design
n changing requirements caused by test design
Early test design helps to build quality,
Early test design helps to build quality,
stops fault multiplication
6
Experience report: Phase 1
Phase 1: Plan
2 mo 2 mo
dev test
test
150 faults

1st mo.
50 faults
users
not
happy
Quality
fraught, lots of dev overtime
Actual
"has to go in"
but didn't work
7
Experience report: Phase 2
Source: Simon Barlow & Alan Veitch, Scottish Widows, Feb 96
Phase 2: Plan
2 mo 6 wks
dev test
test
50 faults
1st mo.
0 faults
happy
users!
Quality
smooth, not much for dev to do
Actual
acc test: full
week (vs half day)
on time
8
VV&T

n Verification
•• the process of evaluating a system or component to the process of evaluating a system or component to
determine whether the products of the given determine whether the products of the given
development phase satisfy the conditions imposed development phase satisfy the conditions imposed
at the start of that phase [BS 7925at the start of that phase [BS 7925 1]1]
n Validation
•• determination of the correctness of the products of determination of the correctness of the products of
software development with respect to the user software development with respect to the user
needs and requirements [BS 7925needs and requirements [BS 7925 1]1]
n Testing
•• the process of exercising software to verify that it the process of exercising software to verify that it
satisfies specified requirements and to detect faultssatisfies specified requirements and to detect faults
9
Verification, Validation and Testing
Verification
Validation
Testing
Any
10
V-model exercise
11
The V Model - Exercise
DS
FD
Build
Components
Build
Units
TD
Build

System
Code
Build
Assemblage
VD
System
Test
Integration
Test
Review FD
Review TD
TUT
FUT
Review DS
Review VD
Assembly
Test
Exceptions:
Conversion Test
FOS: DN/Gldn
12
How would you test this spec?
n A computer program plays chess with one
user. It displays the board and the pieces on
the screen. Moves are made by dragging
pieces.
13
“Testing is expensive”
n Compared to what?
n What is the cost of NOT testing, or of faults

missed that should have been found in test?
Cost to fix faults escalates the later the fault is foundCost to fix faults escalates the later the fault is found
Poor quality software costs more to usePoor quality software costs more to use
•• users take more time to understand what to dousers take more time to understand what to do
•• users make more mistakes in using itusers make more mistakes in using it
•• morale suffers morale suffers
•• => lower productivity=> lower productivity
n Do you know what it costs your organisation?
14
What do software faults cost?
n Have you ever accidentally destroyed a PC?
knocked it off your desk?knocked it off your desk?
poured coffee into the hard disc drive?poured coffee into the hard disc drive?
dropped it out of a 2nd storey window?dropped it out of a 2nd storey window?
n How would you feel?
n How much would it cost?
15
Hypothetical Cost - 1
(Loaded Salary cost: £50/hr)
Fault Cost Developer User
£700 £50
- detect ( .5 hr) £25
- report ( .5 hr) £25
- receive & process (1 hr) £50
- assign & bkgnd (4 hrs) £200
- debug ( .5 hr) £25
- test fault fix ( .5 hr) £25
- regression test (8 hrs) £400
16
Hypothetical Cost - 2

Fault Cost Developer User
£700 £50
- update doc'n, CM (2 hrs) £100
- update code library (1 hr) £50
- inform users (1 hr) £50
- admin(10% = 2 hrs) £100
Total (20 hrs) £1000
17
Hypothetical Cost - 3
Fault Cost Developer User
£1000 £50
(suppose affects only 5 users)
- work x 2, 1 wk £4000
- fix data (1 day) £350
- pay for fix (3 days maint) £750
- regr test & sign off (2 days) £700
- update doc'n / inform (1 day) £350
- double check + 12% 5 wks £5000
- admin (+7.5%) £800
Totals £1000 £12000
18
Cost of fixing faults
Req UseDes Test
1
10
1000
100
19
How expensive for you?
n Do your own calculation

calculate cost of testingcalculate cost of testing
•• people’s time, machines, toolspeople’s time, machines, tools
calculate cost to fix faults found in testingcalculate cost to fix faults found in testing
calculate cost to fix faults missed by testingcalculate cost to fix faults missed by testing
n Estimate if no data available
your figures will be the best your company has!your figures will be the best your company has!
20
Contents
Lifecycle
1 2 3
ISEB Foundation Certificate Course
4 5 6
Models for testing, economics of testing
High level test planning
Component Testing
Integration testing in the small
System testing (non-functional and functional)
Integration testing in the large
Acceptance testing
Maintenance testing
21
(Before planning for a set of tests)
n set organisational test strategy
n identify people to be involved (sponsors,
testers, QA, development, support, et al.)
n examine the requirements or functional
specifications (test basis)
n set up the test organisation and infrastructure
n defining test deliverables & reporting
structure

See: Structured Testing, an introduction to TMap®, Pol & van Veenendaal, 1998
22
High level test planning
n What is the purpose of a high level test plan?
Who does it communicate to?Who does it communicate to?
Why is it a good idea to have one?Why is it a good idea to have one?
n What information should be in a high level
test plan?
What is your standard for contents of a test plan?What is your standard for contents of a test plan?
Have you ever forgotten something important?Have you ever forgotten something important?
What is not included in a test plan?What is not included in a test plan?
23
Test Plan 1
n 1 Test Plan Identifier
n 2 Introduction
software items and features to be testedsoftware items and features to be tested
references to project authorisation, project plan, QA references to project authorisation, project plan, QA
plan, CM plan, relevant policies & standardsplan, CM plan, relevant policies & standards
n 3 Test items
test items including version/revision leveltest items including version/revision level
how transmitted (net, disc, CD, etc.)how transmitted (net, disc, CD, etc.)
references to software documentationreferences to software documentation
Source: ANSI/IEEE Std 829-1998, Test Documentation
24
Test Plan 2
n 4 Features to be tested
identify test design specification / techniquesidentify test design specification / techniques
n 5 Features not to be tested
reasons for exclusionreasons for exclusion
25