Lecture Critical Appraisal of Systematic Reviews

Critical Appraisal of
Systematic Reviews

Douglas Newberry


Systematic Reviews
—
or How to make a Monkey
out of EBM without hardly
trying!


Systematic Reviews:
Objectives:
• Appraise a systematic review for validity
• Discuss Meta Analysis / use Odds Ratios
• Obtain Number Needed to Treat (NNT)
from Odds Ratios
• Consider clinical implications of a
Systematic Review {including when to
bin it instead!}


We can see further than our
forbearers because we stand on
the shoulders of Giants
{and have better spectacles}
• these ideas are cribbed unashamedly
from friends, books & previous courses

Systematic Reviews:
What are your Objectives:
What do you want to cover?
Please interject with helpful questions!


Did I really want a
systematic review?
(but please do not pretend)
• admit your ignorance — expert review or
consensus guidelines > broad introduction,
cover many areas (class C evidence).
• if the question is important > formulate it!
• Systematic review > narrow but rigorous
focus.


Systematic Reviews — Where
do I start:
• Start with your 4 (or 3) part clinical
question!
• Is a systematic review a sensible
approach?
• Does THIS systematic review address
MY question?
• Is it a systematic review at all?


Is it a systematic review? does it:

• define a four part (answerable) clinical
question?
• combine Randomized Controlled Trials
(RCT’s)?
• describe PRE-DEFINED search methods?
• PRE-DEFINED inclusion criteria?
• PRE-DEFINED methodological exclusion
criteria?


Sceptical View?
Take it with a grain of salt:
• transparent declaration of funding of work?
• Drug Company sponsorship of Reviews vs.
Methodological quality>Cochrane review!
• who employs the authors?
• open discussion of existing controversy &
commercial gain?
• Don’t waste salt on your food, keep it for
your reading!


Meta analysis —
combine what with what?
• Low Molecular Weight Heparin (LMWH)
in hip surgery — begin before or after the
operation?
• meta analysis of placebo controlled
RCT’s of heparin in hip surgery >>
• pre-op & post-op LMWH vs. placebo

• post-op LMWH Vs placebo
• pre-operative >> less intra-op bleeding??


Can we believe it ?
• bias free search & inclusion criteria?
• appraisal of methodology of primary
studies?
• consistent results from all primary
studies?
– if not, are the differences sensibly
explained?
• are the conclusions supported by the data?


If we believe it — does it apply to
our patient?
• Is our patient (or population) so different
from those in the primary studies that the
results may not apply?
• consider differences in:
– time — many things change.
– culture — both treatments and values of
outcomes can be different
– stage of illness or prevalence can effect


We believe it ! but
—>> does it matter?
• Is the benefit worthwhile to our patient?

• Ask the patient about cultural values.
• Think about Relative Risk Reduction vs.
Absolute Risk to our patient.
• Potential benefit is the Absolute risk
avoided in our patient = Absolute Risk
Reduction (ARR)!


Absolute Risk—> The risk our
patient is facing!
• How likely is our patient to die (or have
the outcome of interest) without
intervention? = Control Event Rate (CER)
• consider this individual patient’s risk
factors to estimate Patient Expected Event
Rate = PEER.
• Absolute Risk usually increases with age.
• Improvement measured as Absolute Risk
Reduction (ARR)


Relative Risk Reduction:
• Usually reported in studies.
• Ratio of the improvement of outcome
over outcome without intervention (Rx):
• {Control Event Rate (CER) —
Experimental Event Rate (EER)} / CER
• i.e. {CER-EER}/CER
• often independent of prevalence!
• often similar at different ages!

Our patient wants an absolute
Risk Reduction (ARR):
• is a 40% reduction in Cardiac Risk worth
taking pills daily for 10 years?? >vote!
• if I have a 30% risk of MI or death {30
out of 100 people like me will suffer MI
or death} in next 10 years > 40% RRR >>
only 18 out of 100 will have MI or death.
ARR = 12 out of 100! >>I like that!
• BUT if I have a 1% risk in 10 years, 40%
less is a 0.6% risk >> hardly different!


Number Needed to Treat (NNT)
(very trendy but tricky):
• only defined for specific prevalencePatient’s Expected Event Rate=PEER!
• only defined for a specific intervention!
• only defined for a specific outcome!
– eg. Pravastatin™ 40 mg nocte x10 years,
in a 65 year old male, ex-smoker with
high BP and Diabetes, to reduce MI or
Death.
• NNT is the inverse of Absolute Risk


Number Needed to Treat (NNT)
for previous example:
• 12 fewer MI or death in 10 years per 100

persons treated: ARR=12/100
• NNT = 1/(12/100)=100/12= 8.3
• But the same Relative Risk Reduction
(RRR) of 40% with a low prevalence:
• 0.4 fewer MI/death per 100 treated,
ARR=0.4/100.
• NNT = 1/(0.4/100) = 100/0.4 = 250!


Why Odds Ratios? > compare
results of different studies.
• consider 2x2 table:
• RRR is (a-b/a) — but you can only go in
rows within same study!
• Odds ratio is (a/c)/(b/d) = ad / bc — the
individual ratios are in columns, and
therefore are independent of the prevalence
which is different in different studies.
• must use odds ratios to combine RCT’s


Odds Ratio (OR) to NNT — is the
improvement worth the trouble?
• 1>OR>0, lower the OR = better the
treatment (Rx) >> lower NNT.
• for any OR, NNT is lowest when
PEER=0.5
• estimate the PEER (patient’s risk)
• apply the OR to get patient's NNT.

Convert PEER & OR to NNT:
Control
Event
Rate
(CER)
{apply
PEER 
here}

Odds Ratio (OR)
CER
0.9
0.7
0.5
0.1 110
36
21
0.5

38

11

6

0.9

101

27

12


Formula used in the table:

NNT= 1­ {PEER * (1­OR)}
(1­PEER)*(PEER)*(1­OR)


Table induced nausea!
• lower OR >> lower NNT
• Patient needs to be at risk (non-trivial
PEER) in order for risk reduction to be
worth the effort.
• for any OR, NNT lowest when PEER=0.5
• more effective treatment > lower NNT
• BUT are your patient’s values satisfied by
the intervention and its sequelae?


Subgroup analysis: Sceptical unless:
• the subgroups make biological and clinical
sense?
• the differences are both clinically &
statistically significant?
• was a-priori hypothesis (before this data)?
• other evidence supports these sub-groups?
• few (OK) or many (nix) sub-group

analyses?


Any Questions?