Statistical benchmarking what it can add to CQC

Institute for Transport Studies
FACULTY OF ENVIRONMENT

Statistical Benchmarking:
What it can add to CQC
Phill Wheat
Senior Research Fellow
15th October 2013


What is Good?
• Consider trying to determine if an authority is maintaining it’s
highway network efficiently
Different answers from
different measures

Cost per
highway km

Cost per
head

Which to
use?

• Some authorities will perform well against some criteria and
badly against others
• Lack of a single measure which gives a handle on potential
cost savings that could be made

What is Good?
• Even if there is only one measure, it maybe the case that
larger authorities have an implicit (scale) advantage over
smaller authorities
• Also other factors which influence (characterise) cost:
Customer perception and output quality
An authority has high cost, high quality
Another has low cost, low quality
The question we really want to know is whether either
can reduce cost and still maintain quality


What is Good?

Cost

.

B

.

A

O

.

C

Cost frontier (drawn
for a given level of
quality)
Notice not a straight line…
Economies of Scale (unit
costs fall with the size of
the authority)

Highway
Length


What is Good?

Cost

.

.

C

B
D

.

A

O

.

.

E

Cost frontier (quality
= high)
Cost frontier (quality
= low)

The bottom line here is that authorities A, B, C, D
and E are all achieving the minimum cost possible
for their size and quality, even though their unit
costs are quite different

Highway
Length


What is Good?
• Authorities X and Y are above the frontier and so are inefficient
• By adopting best practice Authorities X and Y can reduce costs
without sacrificing output
X

Cost
Y

.

A

.

.

.
.
. X’

C

B

Cost frontier (drawn
for a given level of
quality)

Cost efficiency (TE)
= Minimum cost/
Actual cost
TEA=OX’/OX
0<=TE<=1

O

Highway
Length

TE shows the
proportion of actual
cost which is needed if
the authority adopted
best practice (all other
things equal)


Approach
• Need to model the explicit relationship between cost and the
various cost drivers
– Include quality, physical outputs and citizen satisfaction

• Allows us to see the trade-off that exists between cost and
quality and citizen satisfaction
• Do this by exploiting the rich dataset collected
– Use of regression analysis to model minimum costs and then
compute the efficiency gap for each authority
– Gives clear indication as to the cost of incrementing quality and
citizen satisfaction holding everything else equal

Log(min cost) = a0 + a1 . Log (cost driver 1) + a2 . Log (cost driver 2)
+…
… + ak . Log (cost driver k)


The cost categories that we
consider and cost drivers
Cost Category

Cost Drivers

Highway Pavement
Maintenance (Reactive +
Structural Maintenance)

Total Highway Network Length 90
Average Highway Condition
CSM Condition of Road
Surfaces

Street Lighting Spend

Number of lighting columns
% of units operational
CSM Street Lighting

107

Winter maintenance
spend

Length of Precautionary
Salting Routes
Number of Salting Runs
CSM Winter Maintenance

135

1

an authority for a given year

Number of
complete
observations1


Overall Model Evolutions
• Highways model is fairly good in terms of the shape of the
frontier
• The street lighting model is okay but has some limits
• The winter service model is more of a concern… missing
key variables?


Summary of cost savings by
model
Average
potential saving
Model Average Score Minimum Score
by LA
Highway

71%

33% £

4,669,803

Lighting

72%

33%

£

900,738

Winter_Op

86%

53%

£

318,688

• The Efficiency Score gives the minimum % of actual cost
required to produce the existing level of output and quality i.e.
after eliminating cost inefficiency.
• Average Scores seem plausible
• Minimum Scores of 33% are not uncommon from this type of
analysis. Probably a good reason (other than pure inefficiency)
for this LA to be different – Important: all we really measure is a
gap after controlling for a set of cost drivers

What we DO get from this
analysis
• An estimate of the minimum cost for each authority tailored to its
own characteristics, quality and citizen satisfaction
• A tool to conduct what if analysis e.g. How do (minimum) costs
change if:
– Authorities merged highway functions and increased network size for a
given operation,
– an Authority were to change quality (e.g. to improve average condition by
1%)
– Authority was prepared to allow public satisfaction to reduce

• A potential cost saving for each LA (if they closed the gap)
• Identification of the best performing authorities which is useful to
direct more process oriented analysis (to establish why there is a
gap)


The estimated cost frontier:
highway maintenance model
Unit Cost at:
5000km of highway
length = £2350 per km
20000km of highway
length = £2035 per km
ECONOMIES OF SCALE
Implication: A large authority
can be expected to have unit
costs 15% cheaper than a

small authority even when
both are efficient (producing
at minimum possible cost)


The cost of incrementing quality:
highway maintenance model


What we DO get from this
analysis
• An estimate of the minimum cost for each authority tailored to its
own characteristics, quality and citizen satisfaction
• A tool to conduct what if analysis e.g. How do (minimum) costs
change if:
– Authorities merged highway functions and increased network size for a
given operation,
– an Authority were to change quality (e.g. to improve average condition by
1%)
– Authority was prepared to allow public satisfaction to reduce

• A potential cost saving for each LA (if they closed the gap)
• Identification of the best performing authorities which is useful to
direct more process oriented analysis (to establish why there is a
gap)


What we DO NOT get from
this analysis
• An understanding of WHY there is a gap between actual

observed cost and minimum cost
– Analysis is useful to identify potential saving magnitude and direct to
which LAs do look like they are performing close to minimum cost
– But supplementary work is required to establish and disseminate
best practice

• The ‘gap’ is what is left over once we control for the effects
of cost drivers
– We can only control for the cost drivers (variables) that we can
collect
– Thus there maybe other reasons (outside of an LA’s control) which
my explain the gap
• The gap is likely to be a maximum possible saving rather than an
absolute


Going forward
• We have made a good start in the pilot study
• Clear ways to take this forward (more years of data, more
cost drivers, more cost categories)
• But needs to be taken forward along side other analyses
– Process benchmarking is important too
– Authorities need the why as well as the what!


Institute for Transport Studies
Phill Wheat
Senior Research Fellow
Institute for Transport Studies
University of Leeds

We offer taught courses, bespoke training and research
consultancy across a range of transportation policy,
regulation and economics areas