ENTREPRENEURSHIP OECD framework for the evaluation of SME and entrepreneurship policies and progr

This Framework provides policy makers with a concrete, explicit, practical and accessible
guide to best practice evaluation methods for SME and entrepreneurship policies
and programmes, drawing upon examples from a wide range of OECD countries.
It examines the benefits of evaluation and how to address common issues that arise
when commissioning and undertaking SME and entrepreneurship evaluations. Key
evaluation principles are set out, including the “Six Steps to Heaven” approach, and
illustrated with examples of evaluations of national, regional and local programmes
that can be explored further by the reader. The publication focuses not only on the
evaluation of individual policies and programmes but also on bigger picture peer
review evaluations and assessment of the impact on SMEs and entrepreneurship
of mainstream programmes that do not have business development as their principal aim.

The full text of this book is available on line via this link:
www.sourceoecd.org/industrytrade/9789264040083
Those with access to all OECD books on line should use this link:
www.sourceoecd.org/9789264040083
SourceOECD is the OECD’s online library of books, periodicals and statistical databases.
For more information about this award-winning service and free trials, ask your librarian, or write to
us at

ISBN 978-92-64-04008-3
85 2007 04 1 P

�����������������������

852007041cov.indd 1

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes

OECD Framework for the Evaluation
of SME and Entrepreneurship Policies

and Programmes

OECD Framework
for the Evaluation of SME
and Entrepreneurship
Policies and Programmes

FIRST
IF
FIF TH
IRST F
H
T
R
OND F
U
ID SEC
IRD FO
H
H THR
T
D
T
R
N
U
D
O
O
S EC

IF TH F
EC O N
IRST F
FIRST
IRST S
OND F
FIF TH
IF TH F
D S EC
H
F
TH
IR
T
H
H
IF
R
T
F
T
U
T
FIRS
OURTH
D FO
FO U R
CO ND
IF TH F
D THIR

THIRD
N
IRD SE
D
IRST F
H
F
O
T
N
C
D
O
H
N
E
S EC
S EC O
FO U R T
ST S
THIRD
FIF TH
FIRST
TH FIR
OURTH
OURTH
FIF TH
TH FIF
H
R

IF TH F
IF TH F
T
F
F
U
R
T
T
N S ME
O
S
U
S
F
O
D FIR
N FIR
HIRD
LUATIO
A LU F
T
A
EC O N
V
V
S
D
LUATIO
E

E
A
D
N
S
V
Y
IR
Y
E
ICY
H TH
RSHIP
S EC O
OLIC
POLIC
IP POL
FO U R T
R E N EU
FIRST
SHIP P
R
EU R S H
FIF TH
IP
NTR EP
N
U
T
E

H
E
S
H
E
R
N
S
IR
S
P
N
F
R
IO
E
UR
RE
E NTR E
ALUAT
HIP D
IP SM
R E N EU
PR E N E
URSH
NTR EP
ICY E V
S ME
N EU R S
E POL

NTR EP
PR E N E
TI
R E PR E
S ME E
E NTR E
ATION
U
N
E
L
E NTR E
VALUA
N ENT
HIP SM
M
E
IO
S
A
S ME E
N
IO
S
T
R
Y
V
T
IO

U
A
A
E
IC
T
N
E
OL
VALU
REN
ALUA
E VALU
UATIO
OLICY
SME P
O
L
Y
LICY E
P
NTR EP
ICY E V
P
IP
A
E
O
L
H

IC
P
V
O
IP
S
N
L
IP
E
P
E
R
H
SM
SME
LUATIO
URS
LICY
R E N EU
IP PO
URSH
Y E VA
RSHIP
NTR EP
PR E N E
HIP PO
EU R S H
PR E N E
POLIC

E
S
R E N EU
E
N
N EN
ION E
E
R
P
R
T
E
R
IO
E
M
T
A
T
U
T
R
R
S
U
A
N
T
E

L
N
P
E
A
IP
N EN
EE
E VALU
TR E
REN
URSH
ICY E V
LUATIO
OLICY
PR E N E
E POL
ME EN
ION SM
NTR EP
S ME
Y E VA
SME P
IP SM
E NTR E
POLIC
ALUAT
S ME E
TION S
V

RSHIP
A
EU R S H
ATION
N
ATION
U
E
U
N
U
E
U
E
L
L
IO
Y
N
L
R
A
E
A
T
P
V
A
R
IC

V
A
V
L
RE
E
EP
U
Y E
E
T
O
R
L
N
Y
IC
T
Y
P
L
E
A
N
O
IC
V
E
ION
E P

YE
OLIC
SHIP
POL
ATION
ALUAT
IP SM
POLIC
E VALU
SHIP P
URSH
E N EU R
ICY E V
OLICY
PR E N E
E POL
E N EU R
URSHIP
T R E PR
R
E
SME P
N
IP SM
P
E NTR E
N
E
H
E

IP
E
S
H
E
R
R
S
R
R
ENT
N SM
R E N EU
R E N EU
NTR EP
NTR EP
N S ME
NTR EP
LUATIO
S ME E
TION E
ION E
Y E VA
LUATIO
VALUA
A
ALUAT
IC
E
V

V
L
E
Y
E
O
Y
IC
P
OL
OLIC
LICY
ION
SME P
URSHIP EURSHIP SME P
ALUAT
HIP PO
ICY E V
N
PR E N E
N EU R S
E POL
E
M
R
R E PR E
S
T
P
N

E NTR E
IP
E
E
H
N
NTR
N EU R S
LUATIO
R E PR E
S ME E
Y E VA
N ENT
POLIC
ATION
U
UATIO
L
L
A
A
V
V
E
YE
ICY
E POL
POLIC
HIP SM
E NTR E

PR E N E

URS

HIP SM
N EU R S
R E PR E
T
I
N
E
ATION
EU R S H
E VALU
E PR E N
R
OLICY
P
T
I
T
E
N
A
M
E
IP S
ME
E VALU

URSH
OLICY
TION S
PR E N E
SME P
E NTR E
VALUA
E
RSHIP
HIP PO
U
Y
E
S
N
IC
R
E
R E PR
E N EU
IP POL
R
H
P
N ENT
S
E
IO
R
T

R
ION
A
U
T
ALUAT
E VALU
E EN
RENE
ICY E V
OLICY
ION SM
NTR EP
E POL
T
SME P
E
M
A
S
E
IP
N
U
H
IP
L
M
S
H

R
E VA
ION S
LUATIO
N EU R S
R E N EU
S
OLICY
R E PR E
NTR EP
Y E VA
ALUAT
N ENT
ION E
RSHIP
SHIP P
POLIC
ICY E V
R
L
ALUAT
LUATIO
U
IP
R E N EU
V
A
O
E
E

H
V
E
P
E
N
Y
S
R
E
Y
R
NT P
IP
LIC
R
IC
U
E
O
L
H
P
E
P
N
O
S
E
P

N
P
E
IO
R
E
NTR
TR E
PR E
ALUAT
HIP SM
IP SM
R E N EU
URSH
ICY E V
S ME E
N EU R S
ME EN
E NTR E
E POL
NTR EP
PR E N E
R E PR E
NT
ATION
TION S
N S ME
U
A
L

E NTR E
HIP SM
N ENT
ION E
U
IO
S
A
T
S ME E
N
L
IO
T
R
A
V
T
U
A
IO
A
U
A
E
E
T
L
V
LUA

E VA
REN
VALU
LICY
ICY E
E VALU
O
L
Y E VA
Y
OLICY
NTR EP
LICY E
P
EV
O
IC
E
P
O
L
IC
P
P
Y
E
N
O
IP
L

P
E
M
IO
SM
SME
HIP S
POLIC
URSH
IP PO
ALUAT
RSHIP
RSHIP
N EU R S
ICY E V
RSHIP
PR E N E
EU R S H
R E N EU
E
U
N
P
R E N EU
UR
R E PR E
E POL
E
R
P

E
E
T
E
M
E
T
N
N
N
R
R
S
R
E
E
E
N
T
T
P
R
IP
E
PR
EN
N EN
TR E
ATION
URSH

NTR EP
LUATIO
E VALU
E NTR E
PR E N E
N S ME
TION E
ME EN
Y E VA
PO
OLICY
VALUA
E NTR E
N S ME
LUATIO
POLIC
TION S
SME P
A
IO
A
LICY E
ATION
V
T
U
IP
O
U
A

E
L
URSHIP
L
P
H
A
S
U
E
A
E
Y
V
R
L
V
N
IC
A
E
SM
E
EU
Y E
L
V
N
R
Y

IC
IP
E
E
O
L
P
N
H
R
O
IC
P
S
P
ME P
HIP
NTR E
N EU R
POL
LUATIO
NTR E
OLICY
Y E VA
SHIP S
R E PR E
TION E
S ME E
N EU R S
SHIP P

E
POLIC
N
R
N ENT
VALUA
E N EU R
R
E
E
R
U
IO
IO
P
P
M
T
P
Y
E
T
E
E
S
A
E
IC
R
A

N
U
L
IP
OL
LU
ENT
NTR
NTR
PR E
URSH
Y E VA
SME P
Y E VA
S ME E
POLIC
PR E N E
S ME E
E NTR E
POLIC
E NTR E

-:HSTCQE=UYUU]X:
11-Dec-2007 10:43:50 AM



OECD Framework
for the Evaluation of SME
and Entrepreneurship

Policies and Programmes


ORGANISATION FOR ECONOMIC CO-OPERATION
AND DEVELOPMENT
The OECD is a unique forum where the governments of 30 democracies work
together to address the economic, social and environmental challenges of globalisation.
The OECD is also at the forefront of efforts to understand and to help governments
respond to new developments and concerns, such as corporate governance, the
information economy and the challenges of an ageing population. The Organisation
provides a setting where governments can compare policy experiences, seek answers to
common problems, identify good practice and work to co-ordinate domestic and
international policies.
The OECD member countries are: Australia, Austria, Belgium, Canada, the
Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland,
Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand,
Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, Turkey,
the United Kingdom and the United States. The Commission of the European
Communities takes part in the work of the OECD.
OECD Publishing disseminates widely the results of the Organisation’s statistics
gathering and research on economic, social and environmental issues, as well as the
conventions, guidelines and standards agreed by its members.

This work is published on the responsibility of the Secretary-General of
the OECD. The opinions expressed and arguments employed herein do not
necessarily reflect the official views of the Organisation or of the governments
of its member countries.

Also available in French under the title:
Cadre de l’OCDE sur l’évaluation des politiques et des programmes à l’égard des PME

et de l’entrepreneuriat

Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda.

© OECD 2007
No reproduction, copy, transmission or translation of this publication may be made without written permission.
Applications should be sent to OECD Publishing or by fax 33 1 45 24 99 30. Permission to photocopy a
portion of this work should be addressed to the Centre français d’exploitation du droit de copie (CFC), 20, rue des
Grands-Augustins, 75006 Paris, France, fax 33 1 46 34 67 19, or (for US only) to Copyright Clearance
Center (CCC), 222 Rosewood Drive, Danvers, MA 01923, USA, fax 1 978 646 8600,


FOREWORD

Foreword

T

he OECD Working Party on Small and Medium-sized Enterprises and
Entrepreneurship (WPSMEE), in line with a recommendation of the 2004 Istanbul
Ministerial Declaration on Fostering the Growth of Innovation and Internationally
Competitive SMEs, has prepared this report aimed at strengthening the conceptual
framework for SME policy evaluation. This report seeks to be of direct practical
assistance to public administrators and politicians concerned with evidence on the
effectiveness of SME and entrepreneurship policies and programmes at a national and
local level.
The Framework was written by Dr. Jonathan Potter, Principal Administrator,
OECD Centre for Entrepreneurship, SMEs and Local Development, and Prof. David
Storey, Warwick Business School, UK, and prepared under the supervision of Mme
Marie-Florence Estimé, Deputy Director of the OECD Centre on Entrepreneurship,

SMEs and Local Development (CFE).
A Steering Group, co-chaired by Dr. Roger Wigglesworth, New Zealand and
Mr. George Bramley, United Kingdom, guided the preparation of the Framework. The
Co-Chairs along with the members of the Steering Group offered many valuable
comments during the drafting, revisions and review of the Framework: Mrs. Sue
Weston and Ms. Vicki Brown, Australia; Mrs. Laura Morin, and Ms. Kaili Levesque,
Canada; Ms. Annukka Lehtonen and Mr. Pertti Valtonen, Finland; Mr. Serge Boscher
and Mr. Jean-Hugues Pierson, France; Mr. Tamas Lesko and Dr. Ágnes Jánszky,
Hungary; Mr. Young-Tae Kim and Dr. Sung Cheon Kang, Korea; and Ms. Ana María
Lagares Pérez, Spain.
Sincere appreciation is extended to the Delegates of the OECD WPSMEE for their
numerous comments and inputs during the compilation of the Framework.
Thanks also go to Mr. Kevin Williams, Principal Administrator, OECD Council and
Executive Committee Secretariat, Mr. Hans Lundgren, Head of Section, Evaluation,
Development Co-operation Directorate, and Mrs. Mariarosa Lunati, Administrator, CFE/
SME and Entrepreneurship Division for their drafting suggestions and Ms. Brynn Deprey,
Mr. Jorge Gálvez Mérdez, Mr. Damian Garnys, and Ms. Elsie Lotthe for their operational
support.

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

3



TABLE OF CONTENTS

Table of Contents
Summary and Route Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9

Section 1. Evaluation Issues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Defining evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Why do an evaluation? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Typical objections to evaluation and responses . . . . . . . . . . . . . . . . . .
Key evaluation debates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Doing evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Some key principles for evaluation practice . . . . . . . . . . . . . . . . . . . . .

15
16
17
19
22
27
32

Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

34

Section 2. Evaluation of Individual National Programmes. . . . . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Evaluations of financial assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Enterprise culture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advice and assistance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37
38
39
42
43
47
48

Section 3. Evaluation of Regional and Local Programmes . . . . . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Advice, consultancy and financial assistance . . . . . . . . . . . . . . . . . . . .
Clusters and local innovation systems . . . . . . . . . . . . . . . . . . . . . . . . . .
Support to areas of geographical disadvantage . . . . . . . . . . . . . . . . . . .
Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

53
54
55
57
59
64

Section 4. The Role of Peer Review in Evaluation . . . . . . . . . . . . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The peer review methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OECD national SME reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
OECD regional and local entrepreneurship reviews . . . . . . . . . . . . . . .
OECD evaluation guidance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

67

68
68
70
71
72

Section 5. Reviewing the Aggregate Impact of Public Policies . . . . . . . . . .
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of mainstream policies on SMEs. . . . . . . . . . . . . . . . . . . . . . . . .
Capturing the total policy package . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

75
76
77
91

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

5


TABLE OF CONTENTS

Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

94

Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

95
97

Appendix A. Six Steps to Heaven: Methods for Assessing
the Impact of SME Policy . . . . . . . . . . . . . . . . . . . . . . . . . . .
Appendix B. The OECD Istanbul Position . . . . . . . . . . . . . . . . . . . . . . . .
Appendix C. Examples of Evaluation Guidance . . . . . . . . . . . . . . . . . . .
Appendix D. Assessing the Quality of an Evaluation . . . . . . . . . . . . . . .
Appendix E. Framework Condition Indicators: Entrepreneurship
Conditions in Denmark in 2005 . . . . . . . . . . . . . . . . . . . . .
Appendix F. Summary of the Evaluation of State Aid to SMEs
in the Member States, European Economic Area
and the Candidate Countries . . . . . . . . . . . . . . . . . . . . . . .

106
103
109
111
113

120

List of tables
1.1.
1.2.
2.1.
2.2.
2.3.
2.4.
2.5.

2.6.
2.7.
2.8.
2.9.
2.10.
2.11.
2.12.
2.13.
2.14.
2.15.
2.16.
2.17.
2.18.
2.19.
2.20.
2.21.
2.22.
2.23.
2.24.

6

Qualitative compared with quantitative evaluation . . . . . . . . . . . .
The choice of internal and external evaluators . . . . . . . . . . . . . . . .
SME and entrepreneurship policy areas covered . . . . . . . . . . . . . . .
Loan guarantee scheme, Japan . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Loan guarantee scheme, Canada . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Assistance to new enterprises started by young people, Italy . . . .
Grant assistance and small firm performance, Ireland . . . . . . . . . .
Public subsidies to business angels: EIS and VCT, UK . . . . . . . . . . .

Public subsidies to business angels: EIS, UK . . . . . . . . . . . . . . . . . . .
Assisting young disadvantaged people to start up businesses, UK . .
Graduates into business, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Investment readiness, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of marketing advice, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of business advice, Belgium . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of advisory support, Bangladesh . . . . . . . . . . . . . . . . . . . . . .
Bank customers receiving business advice, UK . . . . . . . . . . . . . . . .
Assistance and advice for mature SMEs, UK . . . . . . . . . . . . . . . . . . .
Use and impact of business advice, UK . . . . . . . . . . . . . . . . . . . . . . .
Evaluating entrepreneurial assistance programs, US . . . . . . . . . . .
Encouraging partnerships amongst SMEs, Sweden . . . . . . . . . . . . .
Technology assistance to small firms, US . . . . . . . . . . . . . . . . . . . . .
The SBIR program, US . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The UK SMART programme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of science parks, Greece . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Impact of science parks, Sweden . . . . . . . . . . . . . . . . . . . . . . . . . . . .
University/SME links, New Zealand . . . . . . . . . . . . . . . . . . . . . . . . . .

23
25
38
39
40
40
40
41
41
42
43

43
44
44
44
45
45
46
46
47
48
48
49
49
50
50

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


TABLE OF CONTENTS

2.25.
2.26.
3.1.
3.2.
3.3.
3.4.
3.5.
3.6.
3.7.

3.8.
3.9.
3.10.
3.11.
3.12.
3.13.
3.14.
3.15.
3.16.
3.17.
3.18.
5.1.
5.2.
5.3.
5.4.
5.5.
B.1.
D.1.

Impact of management training on SMEs, UK . . . . . . . . . . . . . . . . .
51
Small firms training loans, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51
Regional/local policy areas covered . . . . . . . . . . . . . . . . . . . . . . . . . .
55
Subsidised consulting, Belgium, Wallonia. . . . . . . . . . . . . . . . . . . . .
56
Business advisory services, UK, South West England . . . . . . . . . . .
56
Enhancing the capability of the SME owner through use

of consultants, UK, Scotland . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
Export information and advice, Canada, Quebec . . . . . . . . . . . . . . .
57
Enterprise partnerships for exporting, Sweden, Örebo . . . . . . . . . .
58
Small business grants, UK, North East England . . . . . . . . . . . . . . . .
58
Regional development agency grants, Ireland, Shannon . . . . . . . .
59
Local innovation system policy, EU regions . . . . . . . . . . . . . . . . . . .
60
Business networking, UK, North East England . . . . . . . . . . . . . . . . .
61
Enterprise Zone evaluation, US, Indiana . . . . . . . . . . . . . . . . . . . . . .
61
Enterprise Zone evaluation, US, Five States . . . . . . . . . . . . . . . . . . .
62
Enterprise Zone evaluation, UK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
62
Evaluation of enterprise support in disadvantaged areas, UK . . . .
63
Regional policy evaluation, UK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
63
Regional policy evaluation, Italy . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
64
Rural policy evaluation, Canada, Quebec. . . . . . . . . . . . . . . . . . . . . .
64
Rural enterprise support, United Kingdom, Northumberland . . . .
65

The indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
83
Ease of Doing Business ranking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
84
Starting a business in 1999, 2004 and 2006 . . . . . . . . . . . . . . . . . . . .
85
Average conversion rates young businesses/nascent
entrepreneurs, 2000-2004 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
92
Selecting policy areas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
92
Six Steps to Heaven: Methods for assessing the impact
of SME policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
Grid for a synthetic assessment of the quality
of evaluation work. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

Figure
1.1.

New Zealand Trade and Enterprise (NZTE) Growth Range
Programme Logic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

31

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

7

ISBN 978-92-64-04008-3
OECD Framework for the Evaluation of SME and Entrepreneurship
Policies and Programmes
© OECD 2007

Summary and Route Map

T

his Framework document provides a forum for the international exchange
of knowledge on best practice evaluation of Small and Medium-sized
Enterprise (SME) and Entrepreneurship policy. Its target readership is public
administrators and policy-makers concerned with the formulation,
development and implementation of SME policy, together with professionals
concerned with evaluation of such policies. It seeks to be concrete, explicit,
practical and accessible, drawing upon examples from a wide range of OECD
countries. Almost all the evaluations documented are publicly available
online. It is also intended that the text will assist SME policy makers in nonmember countries.
In line with the OECD Istanbul Position, which underlines the need to
strengthen the culture of evaluation of SME and entrepreneurship policies
(Appendix A), this document has four objectives:

●

To increase the awareness of politicians and public officials of the benefits
from having an evaluation culture.

●

To disseminate examples of good micro evaluation practice at national and

sub national levels.

●

To highlight key evaluation debates: Who does evaluations? What
procedures and methods should be used? When to do the evaluations?
What about the dissemination of findings? Should all policies be
disseminated in the same way?

●

To make a clear distinction between policies that operate at the micro level,
i.e. SME and entrepreneurship specific policies, and those that operate at
the macro level, i.e. mainstream policies that nonetheless influence SMEs
and entrepreneurship.

To achieve this end, the Framework is divided in three main parts. The
first deals with evaluations of micro-entrepreneurship and SME policies
formulated and delivered at the national level. The second deals with
entrepreneurship and SME policies delivered at the local/regional level. The
third section is rather different. It reviews approaches to establishing the
aggregate impact of a range of public policies that strongly influence
entrepreneurship and SME performance, yet are rarely the responsibility of

9


SUMMARY AND ROUTE MAP

the main department of government responsible for SMEs. Prior to that, the

Framework reviews good practice in evaluation more generally.
It should be noted that the Framework does not seek to be a handbook or
manual that sets out the steps that need to be taken to complete an
evaluation. A substantial body of such handbooks and manuals exist and
selected examples are provided in Appendix C. Rather the focus of the
Framework is on discussing the difficult issues that arise in evaluating SME
and entrepreneurship policies and programmes, particularly with respect to
quantitative impact evaluation, and providing examples of evaluation
approaches that have been used to address these issues. The Framework
should therefore be read in conjunction with, rather than in place of, other
evaluation guidance in this field.
This summary provides a route-map for the reader, highlighting its key
conclusions. It then moves on to setting out the key conclusions from each of
the three parts. Finally it sets out a proposal for continuous improvement in
the evaluation of SME policy.

So, why do evaluation?
●

To establish the impact of policies and programmes.

●

To make informed decisions about the allocation of funds.

●

To show the taxpayer and business community whether the programme is
a cost-effective use of public funds.

●

To stimulate informed debate.

●

To achieve continued improvements in the design and administration of
programmes.

When and how should programme evaluation be done?

10

●

Evaluation has to be integral to the policy process. Hence there is merit in
undertaking prospective evaluations – as policy options are being
formulated; formative evaluations as the policy is in operation; and
summative evaluations once a clear policy impact can be judged. The
summative evaluation findings have to feed back into current policy
making.

●

For summative evaluations we favour a dual approach. The first is to
establish the impact of established large scale programmes by using
quantitative, statistical methods using “control groups” that score highly on
the “Six Steps to Heaven” metric.

●

These can be valuably complemented with qualitative approaches such as
case studies and peer reviews for more detail on how policy works and how

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


SUMMARY AND ROUTE MAP

it may be adjusted. Qualitative approaches are also useful for smaller scale
programmes for which the costs of quantitative evaluation may be too high.

And by whom?
●

Evaluation undertaken by specialists is essential for reliable impact evaluation.
Sometimes the necessary independence can only be delivered by “outsiders”
but independent evaluation units within government can also perform this
role.

The bedrock of good evaluation comprises:
●

The programme has to have clearly specified objectives from which it is
possible to determine whether or not it succeeded.

●

The evaluation has to be set in progress and data collection begun as, or
even before, the programme is implemented.

●

The evaluation has to be able to lead to policy change.

The evaluation of national programmes
●

This section of the Framework provides examples of evaluations that have
been undertaken on the following policy areas: Financial Assistance;
Enterprise Culture; Advice and Assistance; Technology; and Management
Training.

●

It concludes that, whilst there are examples of high quality evaluations, this
is not the norm.

●

Broadly, lower quality evaluations seem to produce more “favourable”
outcomes for the project because they attribute observed change to the policy
when this may not be justified.

The evaluation of local and regional programmes
●

At the regional and local level less costly and less sophisticated approaches
are often adopted because the programmes are often smaller and because
evaluation structures in terms of information bases, professional

evaluation capabilities and understanding of evaluation methods by users
may be weaker.

●

The work of the OECD Local Economic and Employment Development
(LEED) Programme with city and regional governments and development
agencies has shown that a critical issue for policy development is increasing
understanding of the real policy needs of the region or locality and
assessing the alternative options for intervention given the specific local
context.

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

11


SUMMARY AND ROUTE MAP

Peer review: a tool for evaluation
●

Whilst evaluation of programme impacts is still required, broader “peer
reviews” are also useful in providing “big picture” assessments of the full
range of entrepreneurship and SME policies including in selected regions.

Reviewing the aggregate impact on SMEs and entrepreneurship
of public policies
●

Although explicit and targeted SME and entrepreneurship policies influence
the creation of new firms and the development of SMEs, so also do other
government policies which do not have such a focus. They are also rarely
the responsibility of the main SME department of government. These
policies include control of interest rate and tax policies, social policies such
as the setting of unemployment benefits, the cost and time of starting a
new business and the role of immigration and emigration.

●

These policies represent substantial expenditures in many countries and
our review shows they impact powerfully on entrepreneurship and SME
development. However, control, or influence, over that total expenditure is
rarely exercised by the department of government responsible for SME
policy. Instead, other departments or organisations of government often
have considerably larger budgets, but may have different priorities to that of
the main SME department.

●

The challenge for SME and entrepreneurship policy makers is to identify
these macro policies and their links to enterprise. It is then to seek to
ensure that they work in a way which is congruent with the objectives of
enterprise support.

●

Evaluation approaches need to be developed that permit policy makers with
SME and entrepreneurship responsibilities to be able to engage more fully in
cross-government discussions on priority setting.

Future work
The current document is not to be regarded as the definitive or “final”
statement on how SME policy and its constituent parts should be evaluated.
In our judgement there remain too few examples of top quality evaluations. We
also have too little knowledge about the impact which these evaluations have had
upon the formulation of policy and the impact which policy changes have upon
SMEs and the economy more widely for this to be “the last word”. We therefore
propose that this document should evolve over time to reflect the direct interests

12

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


SUMMARY AND ROUTE MAP

of policy makers, SMEs and the taxpayer. A future text could benefit from the
following:
●

More examples of high quality evaluations that can be shared between
countries; and

●

More evidence, probably in a case study format, of the links between
evaluations undertaken and policy changes. An example here might be the
review by OECD of SME policy in Mexico and the changes that subsequently
occurred in that country.

In short, what we are able to provide in this current text are some generic
approaches to evaluation and some examples of evaluations undertaken,
some of which are better than others in terms of their technical merit.

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

13



ISBN 978-92-64-04008-3
OECD Framework for the Evaluation of SME and Entrepreneurship
Policies and Programmes
© OECD 2007

Section 1

Evaluation Issues

15


1.

EVALUATION ISSUES

Defining evaluation
In their review of policy evaluation in innovation and technology,
Papaconstantinou and Polt (1997) provide a very helpful definition of

evaluation. They say: “Evaluation refers to a process that seeks to determine
as systematically and objectively as possible the relevance, efficiency and
effectiveness of an activity in terms of its objectives, including the analysis of
the implementation and administrative management of such activity”.
Several words or phrases in this definition merit strong emphasis. The
first key-word is “process”. This emphasises that evaluation is not a “once-off”
activity, undertaken once a particular programme has been completed.
Instead it is an integral element of a process of improved policy or service
delivery.
A second key phrase in the definition of evaluation is “as systematically
and objectively as possible”. Given that evaluation traditionally takes place “at
the end of the line”1 there are likely to be strong entrenched interests in place
once a programme has been in existence for a number of years. These
entrenched interests include the direct beneficiaries of the programme, such
as the businesses receiving funds, but they will also include those who are
responsible for initiating and administering these programmes. All else held
equal, it is to be expected that all these groups will choose the programme to
continue or expand. The task of the evaluator, however, is to “systematically
and objectively” assess the merits of the programme. In this task, the evaluator
may well conflict with those committed to the programme. Only through the
use of objective techniques, discussed later in the paper, can the evaluator
demonstrate their independence to those delivering programmes.
The third key phrase in the definition is “the relevance, efficiency and effect
of an activity in terms of its objective”. The implicit assumption in this statement
is that the policy has clear objectives and that these are stated in sufficiently
clear terms for them to be used by the evaluator. In practice, this is by no
means always the case. As will be shown later, a key role for evaluators is often
to formalise for the first time the objectives of programmes, often after such
programmes have been in operation for many years.
This definition, and the OECD Istanbul paper, 2 emphasised that

evaluation has an integral role to play in the policy process. Evaluation cannot
be left “at the end of the line”. Instead, it has to be a key element of initial

16

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


1.

EVALUATION ISSUES

policy formulation. Once the policy is operational, all organisations and
individuals responsible for delivery have to be aware that evaluation is to take
place. Once the evaluation has been undertaken, and sometimes as it is taking
place, it should be used as the basis for dialogue with policy makers, with the
objective of delivering better policy. The outcome of the evaluation can then
become an input into a debate on the appropriate ways for governments and
SMEs to interact.

Why do an evaluation?
Whilst some countries have a long established tradition of undertaking
evaluation, others do not. For those seeking to champion a culture of
evaluation, the following arguments summarise the case in favour. We then
also take the arguments that are often used against evaluation and address
them.

To establish the impact of policies and programmes against their
objectives
The principal reason for doing evaluation is to establish whether or not

policy has contributed to correcting or ameliorating the problem it set out to
resolve. This is often thought of in terms of tackling market failures that
reduce economic efficiency, such as inadequate availability of finance, skills,
advice and technologies, but may also encompass a desire to improve equity
among groups of people or places, for example by supporting entrepreneurship
among unemployed youth or entrepreneurship in poor localities. Evaluation
of these impacts is facilitated by a clear statement of measurable outcomes
right at the start of the policy/programme design and the collection of relevant
data throughout its life.

To make informed decisions about the allocation of funds
Governments manage a portfolio of policies and programmes each with
its own rationale and justification. Evaluation assists managers to assess the
relative effectiveness of these policies and programmes and to make
judgements about where to place their efforts in order to obtain the greatest
benefits for given costs. Evaluation evidence can help to identify where
government can make the biggest difference to its objectives and targets.

To show the tax payer and business community whether
the programme is a cost effective use of public funds
The scale of tax-payers’ funding for entrepreneurship and small business
policies clearly varies from one country to another. It also varies according to
precisely what is incorporated into the definition. Nevertheless the amounts

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

17


1.

EVALUATION ISSUES

are usually substantial. For example, EIM (2004) reports that approximately
six billion Euros was spent annually by EU Member States on state aid to small
and medium sized enterprises.3 However, even this may be a considerable
underestimate. One EU Member State – the United Kingdom – in a
comprehensive review of tax-payers’ funding directed towards SMEs, reported
that 2.5 billion GBP of public money was spent on direct support to SMEs in
England alone, PACEC [2005] quoted by [National Audit Office (2006)]. A third
example is a programme in the United States – the Small Business Innovation
Research Program. Cooper (2003) reports this programme made annual awards
of USD 1.1 billion in the 1997-1999 calendar years.4
These examples illustrate that, probably for most developed countries,
public funding of SMEs is substantial, even if it is extremely difficult to
quantify in aggregate and may still be relatively modest in terms of tax-payers’
support to large enterprises. Given these substantial sums of public money, it
is reasonable for tax-payers to be reassured that their funding is being spent
in an appropriate manner. It is reasonable for tax-payers to demand evidence
that public programmes are spending funds in accordance with their stated
objectives. This role is normally played by public auditors. A second role, but
one not normally played by auditors, is to assess whether the public funds are
achieving the objectives set out by politicians. This is the function of
evaluators.

To stimulate democratic debate
In democracies, it is reasonable for the electorate to question the
decisions made by governments. In order to facilitate that debate, it is
appropriate for organisations to be able to have access to evidence on the
impact of policies. In this regard, SME and entrepreneurship policies are no

different from other areas of government expenditure. For this reason, the
results of evaluations enhance and inform public debate.
This debate only takes place when the results of evaluations enter the
public domain. This emphasises not only the importance of undertaking
evaluations, but also of their findings being disseminated.

To achieve continued improvement in the design and administration
of programmes
Politicians and public servants administering SME and entrepreneurship
programmes should be seeking continuous improvements and there is of
course a need to ensure adaptation to changing conditions. Evaluation is a key
tool for learning about how well policies and programmes are delivering, what
problems may be emerging, what elements work well and less well and what
could be done better in the future. For example, policy makers may seek to

18

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


1.

EVALUATION ISSUES

deliver policies to different groups, for example by directing more resources
towards enterprises established by the socially disadvantaged or by those
likely to employ others, or those in high technology. They may seek to deliver
policies using different organisational forms, to stimulate the take-up of
policies or to deliver them in a more cost effective manner. All these changes
of focus can emerge from undertaking appropriate evaluations. Alternatively,

existing policies can be delivered more effectively as a result of accumulated
evaluation experience.

Typical objections to evaluation and responses
The discussion above focussed on the positive aspects of evaluation.
However, one of the barriers to spreading an evaluation practice is a resistance
to evaluation amongst a range of politicians, policy makers and practitioners.
Here we discuss some of the most common objections to evaluation and the
degree to which they stand up to critical assessment. Our judgement is that
although the objections have some weight, on balance, they do not amount to
a solid case for rejecting evaluation and hence sacrificing the benefits cited
above.

But evaluation is expensive and bureaucratic
Evaluation is not costless. Costs include the payment of consultants/
evaluators, the collection of data and the time taken from those delivering
programmes to inform the evaluation. The United Kingdom statistical office,
for example, requires the time of recipients of the programme in providing
their opinions and information about the programme to be costed (i.e. the cost
of the respondents time must be explicitly included in the cost of the
evaluation). Data may also have to be collected from both clients of the
programme, and a “control group” of non clients.
However, the resources committed to evaluation are normally very
modest in comparison with the total size of the programme. For example, the
review by Sheikh and Steiber [2002], “Evaluating Actions and Measures
Promoting Female Entrepreneurship” identified an appropriate budget of
between 2% and 5% for the purposes of evaluation. This may be appropriate
for small programmes but for programmes in larger countries a figure of
between 0.5% and 1% of annual expenditure would be more usual.
Given the opportunity which evaluation provides for using resources

more efficiently, and for the design of new programmes, these seem to be very
modest costs indeed.

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

19


1.

EVALUATION ISSUES

But evaluation does not always lead to policy improvements
Evaluations of programmes can fail to lead to policy change for several
reasons. It may be because those responsible for programme management are
hostile to the concept of evaluation. It can also happen where evaluators fail
to engage programme managers, or where they fail to understand the details
of the programme. Evaluators themselves may fail to express their findings in
a language that is easily understandable to policy makers and those
responsible for policy delivery.
Although there are instances where evaluations have not led to
improvement, this is not a sufficient justification for being reluctant to
undertake any form of evaluation. To minimise the potential problems,
programme managers have to be persuaded that the quality of programme
delivery can be enhanced through evaluations and the consultants have to
“reach out” to programme managers to engage them wherever possible.
But ultimately evaluation takes place for the benefit of the tax-payer, and
not for the provider[s] of the programme. Those programmes that are shown
to be demonstrably ineffective have to be closed, and this has to be recognised
by programme managers.

In practice, if evaluation is to lead to change, a balance must be struck
between, on the one hand, ensuring the independence of the evaluator whilst,
on the other, engaging support of those involved with programme delivery.

And risks diverting attention away from programme delivery
It is the case that there are cultural differences between evaluators and
deliverers of programmes. The former are often analytical individuals, often
with an academic background, whereas the latter consider themselves
practical individuals focused upon delivering services to their clients. Because
they are so close to their clients they view themselves as the best judge of the
effectiveness of the programme. They have difficulty seeing what value a
“detached” consultant can provide in terms of programme improvement. For
this reason, programme deliverers often resent the time taken in completing
forms and collecting data which are, however, vital to the success of an
evaluation. Programme managers and deliverers understandably can also feel
threatened by an evaluation, especially when they know they do not fully
understand the techniques used by the evaluators, but fear the evaluators do
not fully understand the programme.
For an evaluation to be a success however, these cultural differences have
to be managed. The most effective way of achieving this, as identified above,
is to demonstrate that the interests of both the evaluators and the programme
managers/deliverers can be more closely aligned by both parties focussing on
areas for programme improvement. This can be most effectively achieved by

20

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


1.

EVALUATION ISSUES

engaging those delivering the policy through ensuring the issues of concern
are addressed in the evaluation and by being adequate opportunity to
comment upon, and offer their interpretation of, provisional findings.
It is, of course, a simplification to imply that the programme managers
most hostile to evaluation are those fearing negative feedback. Nevertheless
senior policy-makers need to be aware that evaluation, whilst it is in the
taxpayers interests, may provoke considerable hostility from programme
deliverers. The latter have to be engaged but not the ultimate voice.

But evaluation is only for advanced countries
It is the case that programme evaluation is more frequently undertaken
in advanced, rather than in developing economies. In part this may be because
it is more difficult to find sufficient numbers of individuals with the type of
analytical skills necessary to conduct good quality evaluations in developing
economies. Major donor organisations, such as the World Bank, can therefore
play a role in both undertaking evaluations themselves and in training others
to perform these tasks.
Nevertheless it is not only the most developed countries that undertake
evaluation. In its review of state aid to SMEs, EIM [2004] surveyed EU Member
States, European Economic Area and candidate countries. A total of
29 countries were identified. Only Ireland, the Netherlands and Slovakia
performed state aid evaluations on all schemes, implying that evaluation is
not simply characteristic of the more wealthy countries. EIM specifically noted
that the State Aid Act obliges the Slovak Government to evaluate all state aid
using statistical analysis of aid recipients and control groups. They also noted
that the analyses are performed on macro and micro levels. Full details of this
important survey are provided in Appendix F.

Mexico has also recently committed to undertaking SME evaluation. It
believes this will “improve support systems” and identify areas of opportunity,
thus granting certainty to the population on the efficient use of resources.
These examples illustrate that it is not necessarily the most economically
developed countries which are committed to undertaking evaluation.

But there is no history of undertaking evaluation
In countries without a tradition of evaluation it can be difficult to make
this transition. Nevertheless, it is clear that the electorates in many countries
are becoming more sophisticated, in part because of access to the media and
the internet. Countries where evaluations do not take place are likely, in the
future, to be asked why it is that such policy assessments take place
elsewhere. The, perhaps unjustified, inference is that evaluations do not take
place because there is something to hide. It is not sufficient to imply that

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

21


1.

EVALUATION ISSUES

policies are being delivered efficiently because there is no information to the
contrary.

Key evaluation debates
This section reviews four key evaluation debates. The first is the
appropriate technique for evaluating SME and entrepreneurship policies. The

second is the appropriate level of sophistication of the quantitative evaluation
approaches. The third is whether evaluation should be undertaken by
“insiders or outsiders”. The fourth is whether the same evaluation techniques
should be used for all programmes.

The choice of technique
There are two basic options in undertaking summative evaluations5 – the
quantitative and qualitative approaches. Quantitative evaluation involves
assessment of the impact of programmes through a comparison of outcomes
between the group in receipt of aid and some form of “control group”, for
example a similar group of enterprises that have not benefited from policy or
the same enterprises before and after receipt of policy support. Such data may
be collected either directly from the firms themselves or from official data.
Qualitative evaluation or approaches are much more likely to rely upon the
opinions of programme stakeholders including managers and beneficiaries
about the functioning and impact of the programme through techniques
including surveys, case studies and peer reviews. Both approaches will rely
upon a careful scrutiny of programme documentation.
Table 1.1 reviews the advantages and disadvantages of the quantitative
and qualitative approaches.
The principal advantage of qualitative evaluation is the additional
information that it can provide beyond that associated with quantitative
evaluations. Qualitative evaluation normally involves face-to-face discussions
with those in receipt of aid, those responsible for delivering programmes and
other stakeholders. These conversations help not only to obtain information
from stakeholders that can lead to a deeper understanding of the mechanisms
by which policy impact is achieved and how policy might be adjusted but also
to engage stakeholders in policy learning processes. The approach can also
pick up a wide range of other information of interest to policy makers, going
beyond impact to issues such as client satisfaction, policy appropriateness,

sustainability and conflict with other policies.
However, qualitative evaluation has the major disadvantage that it is not
good at providing reliable estimates of policy impact for a number of reasons.
First, surveys of a sample of stakeholders run the risk of being unrepresentative
of programme participants. Increasing the numbers however either adds

22

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007


1.

EVALUATION ISSUES

Table 1.1. Qualitative compared with quantitative evaluation
Qualitative evaluation methodologies

Quantitative evaluation methodologies

Advantages

Disadvantages

Advantages

Disadvantages

Engages participants in
policy learning

Respondents and
Clear answers on impact
interviewers may be biased
or poorly informed

Can vary the scale and
hence cost

Rarely provides a clear
answer

If well done will get close to Lacks information on
true impact
context and mechanisms
behind policy impacts

Deeper understanding of
processes leading to
impacts

Tends to “describe” rather
than “evaluate”

Can be independently
verified

Cost of data collection and
technical demands

Absence of pure control
groups

Should be easy to interpret Risks including “unrepresentative” groups

Possible false impression
of precision

Can assess against a wide
range of evaluation criteria

No opportunity for
independent verification

Narrow focus on
effectiveness and efficiency

Picks up unintended
consequences

Hard to judge efficiency and
effectiveness

Difficult to use on indirect
interventions that seek to
influence the business
environment

Better understanding of
policy options and

alternatives

Hard to establish cause and
effect

considerably to budgets or reduces the quality or depth of the interviews.
Second, despite the best efforts of interviewers, there remains a strong risk of
interviewer bias. Thirdly, the outcome of qualitative evaluation is more often
to describe a process rather than to evaluate an outcome. Fourthly, there is no
opportunity for independent verification. Finally, programme participants
may be asked questions that are virtually impossible to answer. The classic
example is “What impact do you think this programme had on your
business?” Implicitly the respondent is required to hold every other influence
on their business constant and estimate how a programme which probably
took place some years previously has influenced their business in the
intervening period. Even if some programme participants were able to
undertake such mental gymnastics others clearly are not and there is no way
of distinguishing between the answers of the two groups.
The principal disadvantages of the quantitative approach concern its
technical difficulties and the relatively narrow nature of the results it offers,
which focus primarily on issues of effectiveness and efficiency. In terms of the
technical issues, effective quantitative evaluation requires extensive data
collection on the performance of policy-targeted and control group firms.
More importantly, however, in SME and entrepreneurship policy evaluation
situations, there may sometimes be no natural, uncontaminated control
group. Whilst good quantitative analysis seeks to match as closely as

OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007

23