3 principles of testing [compatibility mode]

1
Principles of Testing
1 Principles 2 Lifecycle
4 Dynamic test
techniques
3 Static testing
5 Management 6 Tools
Software Testing
ISEB Foundation Certificate Course
2
Contents
Why testing is necessary
Fundamental test process
Psychology of testing
Re-testing and regression testing
Expected results
Prioritisation of tests
Principles
1 2 3
4 5 6
ISEB Foundation Certificate Course
3
Testing terminology
n No generally accepted set of testing
definitions used world wide
n New standard BS 7925-1
Glossary of testing terms (emphasis on component Glossary of testing terms (emphasis on component
testing)testing)
most recentmost recent
developed by a working party of the BCS SIGISTdeveloped by a working party of the BCS SIGIST
adopted by the ISEBadopted by the ISEB

4
What is a “bug”?
n Error: a human action that produces an
incorrect result
n Fault: a manifestation of an error in software
also known as a defect or bug also known as a defect or bug
if executed, a fault may cause a failureif executed, a fault may cause a failure
n Failure: deviation of the software from its
expected delivery or service
(found defect)(found defect)
Failure is an event; fault is a state of
the software, caused by an error
5
Error - Fault - Failure
A person makes
an error
… that creates a
fault in the
software
… that can cause
a failure
in operation
6
Reliability versus faults
n Reliability: the probability that software will
not cause the failure of the system for a
specified time under specified conditions
Can a system be faultCan a system be fault free? (zero faults, right first free? (zero faults, right first
time)time)
Can a software system be reliable but still have Can a software system be reliable but still have

faults?faults?
Is a “faultIs a “fault free” software application always free” software application always
reliable?reliable?
7
Why do faults occur in software?
n software is written by human beings
who know something, but not everythingwho know something, but not everything
who have skills, but aren’t perfectwho have skills, but aren’t perfect
who do make mistakes (errors)who do make mistakes (errors)
n under increasing pressure to deliver to strict
deadlines
no time to check but assumptions may be wrongno time to check but assumptions may be wrong
systems may be incompletesystems may be incomplete
n if you have ever written software
8
What do software faults cost?
n huge sums
Ariane 5 ($7billion)Ariane 5 ($7billion)
Mariner space probe to Venus ($250m)Mariner space probe to Venus ($250m)
American Airlines ($50m)American Airlines ($50m)
n very little or nothing at all
minor inconvenienceminor inconvenience
no visible or physical detrimental impactno visible or physical detrimental impact
n software is not “linear”:
small input may have very large effectsmall input may have very large effect
9
Safety-critical systems
n software faults can cause death or injury
radiation treatment kills patients (Theracradiation treatment kills patients (Therac 25)25)
train driver killedtrain driver killed

aircraft crashes (Airbus & Korean Airlines)aircraft crashes (Airbus & Korean Airlines)
bank system overdraft letters cause suicidebank system overdraft letters cause suicide
10
So why is testing necessary?
because software is likely to have faultsbecause software is likely to have faults
to learn about the reliability of the softwareto learn about the reliability of the software
to fill the time between delivery of the software and to fill the time between delivery of the software and
the release datethe release date
to prove that the software has no faultsto prove that the software has no faults
because testing is included in the project planbecause testing is included in the project plan
because failures can be very expensivebecause failures can be very expensive
to avoid being sued by customersto avoid being sued by customers
to stay in businessto stay in business
11
Why not just "test everything"?
system has
20 screens
Average: 10 fields / screen
2 types input / field
(date as Jan 3 or 3/1)
(number as integer or decimal)
Around 100 possible values
Total for 'exhaustive' testing:
20 x 4 x 3 x 10 x 2 x 100 = 20 x 4 x 3 x 10 x 2 x 100 = 480,000 tests480,000 tests
If 1 second per test, 8000 mins, 133 hrs, 17.7 days
(not counting finger trouble, faults or retest)
Avr. 4 menus
3 options / menu
10 secs = 34 wks, 1 min = 4 yrs, 10 min = 40 yrs
12

Exhaustive testing?
n What is exhaustive testing?
when all the testers are exhaustedwhen all the testers are exhausted
when all the planned tests have been executedwhen all the planned tests have been executed
exercising all combinations of inputs and exercising all combinations of inputs and
preconditionspreconditions
n How much time will exhaustive testing take?
infinite timeinfinite time
not much timenot much time
impractical amount of timeimpractical amount of time
13
How much testing is enough?
it’s never enoughit’s never enough
when you have done what you plannedwhen you have done what you planned
when your customer/user is happywhen your customer/user is happy
when you have proved that the system works when you have proved that the system works
correctlycorrectly
when you are confident that the system works when you are confident that the system works
correctlycorrectly
it depends on the risks for your systemit depends on the risks for your system
14
How much testing?
n It depends on RISKRISK
riskrisk of missing important faultsof missing important faults
riskrisk of incurring failure costsof incurring failure costs
riskrisk of releasing untested or underof releasing untested or under tested softwaretested software
riskrisk of losing credibility and market shareof losing credibility and market share
riskrisk of missing a market windowof missing a market window
riskrisk of overof over testing, ineffective testingtesting, ineffective testing
15

what not to test (this time)what not to test (this time)
n use RISKRISK to
allocate the time available for testing by allocate the time available for testing by
prioritising testing prioritising testing
So little time, so much to test
n test time will always be limited
n use RISKRISK to determine:
what to test firstwhat to test first
what to test mostwhat to test most
how thoroughly to test each itemhow thoroughly to test each item
}
i.e. where to
place emphasis
16
Most important principle
Prioritise tests
so that,
whenever you stop testing,
you have done the best testing
in the time available.
17
Testing and quality
n testing measures software quality
n testing can find faults; when they are
removed, software quality (and possibly
reliability) is improved
n what does testing test?
system function, correctness of operationsystem function, correctness of operation
nonnon functional qualities: reliability, usability, functional qualities: reliability, usability,
maintainability, reusability, testability, etc.maintainability, reusability, testability, etc.

18
Other factors that influence testing
n contractual requirements
n legal requirements
n industry-specific requirements
e.g. pharmaceutical industry (FDA), compiler e.g. pharmaceutical industry (FDA), compiler
standard tests, safetystandard tests, safety critical or safetycritical or safety related such related such
as railroad switching, air traffic controlas railroad switching, air traffic control
It is difficult to determine
how much testing is enough
but it is not impossible
19
Contents
Why testing is necessary
Fundamental test process
Psychology of testing
Re-testing and regression testing
Expected results
Prioritisation of tests
ISEB Foundation Certificate Course
Principles
1 2 3
4 5 6
20
Test Planning - different levels
Test
Policy
Test
Strategy
Company level

High Level
Test Plan
High Level
Test Plan
Project level (IEEE 829)
(one for each project)
Detailed
Test Plan
Detailed
Test Plan
Detailed
Test Plan
Detailed
Test Plan
Test stage level (IEEE 829)
(one for each stage within a project,
e.g. Component, System, etc.)
21
The test process
specification execution recording
check
completion
Planning (detailed level)
22
Test planning
n how the test strategy and project test plan
apply to the software under test
n document any exceptions to the test strategy
e.g. only one test case design technique needed for e.g. only one test case design technique needed for
this functional area because it is less criticalthis functional area because it is less critical

n other software needed for the tests, such as
stubs and drivers, and environment details
n set test completion criteria
23
Test specification
specification execution recording
check
completion
Identify conditions
Design test cases
Build tests
Planning (detailed level)
24
A good test case
n effective
n exemplary
n evolvable
n economic
Finds faults
Represents others
Easy to maintain
Cheap to use
25
Test specification
n test specification can be broken down into three
distinct tasks:
1.1. identify:identify: determine ‘what’ is to be tested (identifydetermine ‘what’ is to be tested (identify
test conditions) and prioritisetest conditions) and prioritise
2.2. design:design: determine ‘how’ the ‘what’ is to be testeddetermine ‘how’ the ‘what’ is to be tested
(i.e. design test cases)(i.e. design test cases)

3.3. build:build: implement the tests (data, scripts, etc.)implement the tests (data, scripts, etc.)