improved algorithm for adaboost with svm base classifiers

IMPROVED
ALGORITHM
FOR
ADABOOST
WITH
SvM
BASE
CLASSIFIERS
Xiaodan
WANG,
Chongming
WU,
Chunying
ZHENG,
Wei
WANG
Department
of
Computer
Engineering,
Air
Force
Engineering
University
afeu_wgyahoo.com.cn
Abstract
the
margin[5],
which
could
enhance

AdaBoost's
The
relation
between
the
performance
of
AdaBoost
generalization
capability.
and
the
performance
of
base
classifiers
was
analyzed,
Support
vector
machine[6]
was
developed
from
the
and
the
approach
of
improving

the
classification
theory
of
structural
risk
minimization.
By
using
a
kernel
performance
of
AdaBoostSVM
was
studied
There
is
trick
to
map
the
training
samples
from
an
input
space
to
a

inconsistency
existed
between
the
accuracy
and
diversity
high
dimensional
feature
space,
SVM
finds
an
optimal
of
base
classifiers,
and
the
inconsistency
affect
separating
hyperplane
in
the
feature
space
and
uses

a
generalization
performance
of
the
algorithm.
A
new
regularization
parameter,
C,
to
balance
its
model
variable
c-AdaBoostSVM
was
proposed
by
adjusting
the
complexity
and
training
error.
kernelfunction
parameter
of
the

base
classifier
based
on
How
about
the
generalization
performance
of
using
the
distribution
of
training
samples.
The
proposed
SVM
as
the
base
learner
of
AdaBoost?
Does
this
algorithm
improves
the

classification
performance
by
AdaBoost
have
some
advantages
over
the
existing
ones?
making
a
balance
between
the
accuracy
and
diversity
of
Also,
compared
with
using
a
single
SVM,
what
is
the

base
classifiers.
Experimental
results
indicate
the
benefit
of
using
this
AdaBoost
which
is
a
combination
of
effectiveness
of
the
proposed
algorithm.
multiple
SVMs?
These
are
the
attractive
research
issues
in

recent
years[7]
[8][9]
[
1O0].
Keywords:
Support
Vector
Machine;
AdaBoost.
After
analyzing
the
relation
between
the
performance
of
AdaBoost
and
the
performance
of
base
classifiers,
the
approach
of
improving
the

classification
performance
of
1.
INTRODUCTION
AdaBoost
with
SVM
base
classifiers
was
studied
in
this
paper.
A
new
variable
a-AdaBoostSVM
was
proposed
by
Boosting
is
the
machine-learning
method
that
working
adjusting

the
kernel
function
parameter
of
the
base
in
Valiant's
PAC
(probably
approximately
correct)
classifier
based
on
the
distribution
of
training
samples.
learning
model[1].
A
"weak"
learning
algorithm
that
The
proposed

algorithm
improves
the
classification
performs
just
slightly
better
than
random
guessing
in
the
performance
by
making
a
balance
between
the
accuracy
PAC
model
can
be
"boosted"
into
an
arbitrarily
accurate

and
diversity
of
base
classifiers.
Experimental
results
for
"strong"
learning
algorithm.
Schapire[2]
came
up
with
the
benchmark
dataset
indicate
the
effectiveness
of
the
the
first
provable
polynomial-time
boosting
algorithm.
proposed

algorithm.
Freund[3]
developed
a
much
more
efficient
boosting
algorithm
which,
although
optimal
in
a
certain
sense,
nevertheless
suffered
from
certain
practical
drawbacks.
2.
ADABOOST
The
AdaBoost
algorithm,
introduced
by
Freund

and
Schapire[4],
solved
many
of
the
practical
difficulties
of
Given
a
set
of
training
samples,
{(x1,y1), ,
the
earlier
boosting
algorithms,
and
can
be
easily
used
in
(x
,y,)}
,
where

each
training
sample
xi
belongs
to
solving
practical
problems.
AdaBoost[4]
creates
a
.
.
~~some
domain
or
instance
space
X,
and
each
class
label
collection
of
weak
learners
by
maintaining

a
set
of
.
.
weights
over
training
samples
and
adjusting
these
ylsi
s
i
label
set
Y
x
ase
X,ayinY={-1,+1}t
weights
after
each
weak
learning
cycle
adaptively:
the
AdaBoost

calls
a
given
weak
or
base
learning
algorithm
weights
of
the
samples
which
are
misclassified
by
current
repeatedly
in
a
series
of
rounds
t
=
1,
,T
.
One
of

the
weak
learner will
be
increased
while
the
weights
of
the
main
ideas
of
the
algorithm
is
to
maintain
a
distribution
samples
which
are
correctly
classified
will
be
decreased.
or
set

of
weights
over
the
training
set.
The
weight
of
this
The
success
of
AdaBoost
can
be
explained
as
enlarging
distribution
on
training
example
xi
on
round
t
is
Proc.
5th

IEEE
Int.
Conf.
on
Cognitive
Informatics
(ICCI'06)
Y.Y.
Yao,
Z.Z.
Shi,
Y.
Wang,
and
W.
Kinsner
(Eds.)94
1
-4244-0475-4/06/$20.OO
@2006
IEEE94
denoted
w,
(i),
i.e.
w,
(i)
is
the
weight

of
sample
xi
at
3.
Do
for
t
=
I
,T
the
iteration
round
t
.
Initially,
all
weights
are
set
(1)
Train
the
base
classifier
Ct
on
the
weighted

equally,
but
on
each
round,
the
weights
of
incorrectly
training
sample
set;
Alternatively,
a
subset
of
classified
examples
are
increased
so
that
the
base
learner
the
training
examples
can
be

sampled
is
forced
to
focus
on
the
hard
examples
in
the
training
according
to
wt,
and
these
resampled
set.
examples
are
used
to
train
the
base
learner
The
base
learner's

job
is
to
find
a
base
classifier
Ct,
C1.
The
decision
function
of
C
is
ht.
ht
is
the
decision
function
of
base
classifier
Ct,
and
(2)
Calculate
the
training

error
e
of
C:
ht(xi)
gives
the
class label
(+1
or
-1)
of
the
training
n
sample
xi,
In
the
simplest
case,
the
range
of
each
ht
is
8t
=
W,

(i),
Y1
.
'1
(xi)
binary,
i.e.,
restricted
to
{-1,
+1
};
the
base
learner's
job
(3)
Set
weight
of
base
classifier
C,:
then
is
to
minimize
the
error:
£,

=
Pri-
[h,(xi)
.
y]=
w,(i)
(1)
'=
-n(
t)
t
~
~ ~
ih()y2
e
Notice
that
the
error
is
measured
with
respect
to
the
(4)
Update
training
samples'
weights:

distribution
w,
on
which
the
base
learner
was
trained.
In
Wt+i
w,
(i)
exp{-at,yh,
(xi)}
practice,
the
base
learner
may
be
an
algorithm
that
can
Zt
use
the
weights
wt

on
the
training
examples.
wt
(i)
Fe
if
yi
=ht
(xi)
Alternatively,
when
this
is
not
possible,
a
subset
of
the
Z
X
Lea
if
y;
.
h,
(x1)
training

examples
can
be
sampled
according
to
wt,
and
these
(unweighted)
resampled
examples
can
be used
to
Where
Z,
is
a
normalization
factor,
and
E
w,+1
(i)
=
1
train
the
base

learner.
Both
resampling
and
reweighting
can
be
used
to
train
AdaBoost.
4.
Output:
the
final
classifier:
Once
the
base
classifier
ht
has
been
received,
H(x)
=
sign[T
a,h
(x)]
AdaBoost

chooses
a
parameter
at
that
intuitively
t=1
measures
the
importance
that
is
assigned
to
ht
.
For
The
most
basic
theoretical
property
of
AdaBoost
binary
ht
.
asinteorignaldecriptinofAconcerns
its
ability

to
reduce
the
training
error,
i.e.,
the
bivenary
hre,dasn
theporigina
dcption
oA
o
fraction
of
mistakes
on
the
training
set.
Let
the
error
et
given
by
Freund
and
Schapire[4],
typically

set
a
1-8t
of
ht
be
et
=1/2-
yt,
Yt
measures
how
much
better
2t
=
-In(
'
(2)
than
random
(which
has
an
error
rate
of
1/2)
are
h,

's
Note
that
at
>
0
if
et
<1/2
,
and
that
a,
gets
larger
classifications.
Freund
and
Schapire[4]
prove
that
the
training
error
(the
fraction
of
mistakes
on
the

training
set)
as
et
gets
smaller.
of
the
final
hypothesis
H
is
at
most:
The
distribution
w,
is
then
updated
using
the
rule
T T
T
t
f
I
~~~~J7[
2

Ve,(1-e,)]
II-
4
yt2
.
exp(-2>,yt)
(3)
shown
in
the
algorithm.
The
effect
of
this
rule
is
to
t=,
t=,
t=1
increase
the
weight
of
examples
misclassified
by
ht,
and

Thus,
if
each
base
hypothesis
is
slightly
better
than
to
decrease
the
weight
of
correctly
classified
examples.
random
so
that
Yt
2
y
for
some
y
>
0,
then
the

training
Thus,
the
weight
tends
to
concentrate
on
"hard"
error
drops
exponentially
fast.
examples.
Previous boosting
algorithms
required
that
such
a
The
final
or
combined
classifier
H
is
a
weighted
lower

bound
y be
known
a
priori
before
boosting
begins.
majority
vote
of
the
T
base
classifiers
where
at
is
the
In
practice,
knowledge
of
such
a
bound
is
very
difficult
weight

that
assigned
to
ht
.
to
obtain.
AdaBoost,
on
the
other
hand,
is
adaptive
in
that
The
algorithm
for
AdaBoost
is
given
below:
it
adapts
to
the
error
rates
of

the
individual
base
1.
Input:
a
set
of
training
samples
with
labels
hypotheses.
This
is
the
basis
of
its
name
-
"Ada"
is
short
D=
{(xl,y1),
.,(xYn)}
,
xi
,

Y
=
1-1,+1.
for
"adaptive".
Base
learner
algorithm,
the
number
of
cycles
T.
2.
Initialize:
the
weight
of
samples:
w1
(i)
=
1
/
n,
for
alli
=l,*,
n
949

3.
AN
IMPROVED
ALGORITHM
FOR
In
order
to
avoid
the
problem
resulting
from
using
a
ADABOOST
WITH
SVM
BASE
single
and
fixed
a
to
all
RBFSVM
base
classifiers,
and
CLASSIFIERS

get
good
classification
performance,
it
is
necessary
to
find
suitable
a
for
each
base
classifier
RBFSVM.
Because
if
a
roughly
suitable
C
is
given
and
the
variance
Diversity
is
known

to
be
an
important
factor
affecting
of
the
training
samples
is
used
as
the
Gaussian
width
a
of
the
generalization
performance
of
ensemble
methods
RBFSVM,
the
SVM
may
get
comparable

good
[11][12],
which
means
that
the
errors
made
by
different
classification
performance,
we
will
use
the
variance
of
base
classifiers
are
uncorrelated.
If
each
base
classifier
is
the
training
samples

for
each
base
classifier
as
the
moderately
accurate
and
these
base
classifiers
disagree
Gaussian
width
a
of
RBFSVM
in
this
paper,
this
will
with
each
other,
the
uncorrelated
errors
of

these
base
generate
a
set
of
moderately
accurate
RBFSVM
classifiers
will
be
removed
by
the
voting
process
so
as
to
classifiers
for
AdaBoost,
and
an
improved
algorithm
will
achieve
good

ensemble
results[13].
This
also
applies
to
be
obtained,
we
call
it
the
variable
o-
AdaBoost.
AdaBoostRBFSVM.
Studies
that
using
SVM
as
the
base
learner
of
In
the
proposed
AdaBoostSVM,
the

obtained
SVM
AdaBoost
have
been
reported[7][8][9][10].
These
studies
base
learners
are
mostly
moderately
accurate,
which
give
showed
the
good
generalization
performance
of
chances
to
obtain
more
un-correlated
base
learners.
AdaBoost.

Through
adjusting
the
a
value
according
to
the
variance
For
AdaBoost,
it
is
known
that
there
exists
a
dilemma
of
the
training
samples,
a
set
of
SVM
base
learners
with

between
base
learner's
accuracy
and
diversity[14],
which
different
learning
abilities
is
obtained.
The
proposed
means
that
the
more
accurate
two
base
learners
become,
variable
G-AdaBoostRBFSVM
is
hoped
to
achieve
the

less
they
can
disagree
with
each
other.
Therefore,
higher
generalization
performance
than
AdaBoostSVM
how
to
select
SVM
base
learners
for
AdaBoost?
Select
which
using
a
single
and
fixed
a
to

all
RBFSVM
base
accurate
but
not
diverse
base
learners?
Or
select
diverse
classifiers.
In
the
proposed
algorithm,
without
loss
of
but
not
too
accurate
ones?
generality,
re-sampling
technique
is
used.

Suppose
we
can
keep
an
accuracy
and
diversity
The
algorithm
for
variable
a
-AdaBoostRBFSVM:
balance
among
different
base
classifiers,
the
superior
1.
Input:
a
set
of
training
samples
with
labels

result
of
AdaBoost
will
be
gotten,
but
there
is
no
D=
{(x,
,
yI
),
-
,
(x,
yJ
)},
xi
E
X,
y1
E
Y
=
{-1,+1}.
effective
way

to
get
a
desirable
result
for
AdaBoost.
We
will
then
analyze
the
instance
of
using
RBFSVM
as
the
Bs
hlerer
the
number
of
cycles
T.
base
classifier
of
AdaBoost.
2.

Initialize:
the
weight
of
samples:
w1
(i)
=
1/n,
for
The
problem
of
model
selection
is
very
important
for
all
i
=
1, ,
n
SVM,
the
classification
performance
of
SVM

is
affected
3.
Do
for
t
=
1
'T
by
its
parameters.
For
RBFSVM,
they
are
the
Gaussian
(1)
A
subset
of
the
training
examples
can
be
width
a,
and

the
regularization
parameter
C.
The
variation
of
either
of
them
leads
to
the
change
of
sampled
accrdn
tonwtitand
these
classification
performance.
However,
as
reported
in
[7]
resampled
examples
constitute
the

new
although
RBFSVM
cannot
learn
well
when
a
very
low
training
data
set
d1,
dt
will
be
used
to
train
value
of
C
is
used,
its
performance
largely
depends
on

the
base
classifier
Ct
.
The
decision
function
the
a
value
if
a
roughly
suitable
C
is
given.
of
C
is
h
How
to
set
the
a
value
for
the

base
learners
when
t
using
the
RBFSVM
as
base
learner
for
AdaBoost?
(2)
Calculate
the
variance
a
of
dt:
Problems
are
encountered
when
applying
a
single
and
a=sqrt(mean(var(dt
))).
fixed

a
to
all
RBFSVM
base
learners.
In
detail[10],
an
(3)
Using
dt
as
the
training
sample
set,
a
as
over-large
a
often
results
in
too
weak
RBFSVM.
Its
classification
accuracy

is
often
less
than
5000
and
cannot
G
t
meet
the
requirement
on
a
base
learner
given
in
the
RBFSVM
with
Gaussian
width
(,
and
AdaBoost;
On
the
other
hand,

a
smaller
a
often
makes
ht
is
the
decision
function
of
Ct;
the
RBFSVM
stronger
and
boosting
them
may
become
(4)
Calculate
the
training
error
et
of
Ct:
inefficient
because

the
errors
of
these
base
learners
are
n
highly
correlated.
Furthermore,
too
small
a
can
even
Ct
=wt
(i),
Yi
#ht(xi);
make
RBFSVM
overfit
the
training
samples
and
they
also

cannot
be
used
as
base
learners.
Hence,
finding
a
suitable
a
for
AdaBoost
with
SVM
base
learners
becomes
a
problem[10].
950
(5)
Set
weight
of
base
classifier
Ct
number
of

training
samples,
and
axis
Y
gives
the
correct
1
1-
classification
rates.
In
Fig.1,
Ada-SVM
stands
for
at
=-ln(
t);
AdaBoostSVM,
and
improved
Ada-SVM
stands
for
2
Et
variable
a-AdaBoostRBF

SVM.
(6)
Update
training
samples'
weights:
90
-
w,
(i)
exp
{-ar,yih,
(xi
)}
wt+1
W
8
Where
Z,
is
a
normalization
factor,
and
5/+mprvdAaV
427
w,+1
(i)
=
1;

*
4
i
rn
10
0
2i
0O
2ai
300
3;iO
400
4ai
ai0Z
t+
n
~~~~~~~~~~~~~~~~~~NuberfThaig
Saples
4.Otu:tefia
lsiir
Fig.
1.
Performance
comparison
for
Westontoynonliner
H(x)
=
seign[
ah

(xh)].
t
61
4.
EXPERIMENTS
AND
RESU
LTS
.S
t
Ada~~~~~~~~~~~~~~~~~~~~~~~~-
SVM
Impove
Ada-SV
To
evaluate
the
performance
of
the
variable
ci-
79
~Ad-V
AdaBoostRBFSVM,
and
make
a
experimental
7

J100
1i
comparison
between
AdaBoostSVM
which
using
a
single
Fig.2
Performbance
compailes
Fig.
2.
Performance
comparison
for
Wetnooiner
and
fixed
a
to
all
RBFSVM
base
classifiers
and
our
improved
algorithm,

experiments
for
the
Westontoynon-
For
the
Wine
data
set,
the
training
and
testing
samples
liner
data
set
and
the
Wine
data
set[8]
were
conducted,
were
also
chosen
randomly
from
the

given
datasets,
and
the
results
of
the
classification
experiments
are
50,80,100,130,150
are
the
numbers
of
training
samples
given.
used
in
the
experiments,
and
79
is
the
number
of
testing
The

SVM
we
used
is
from
Steve
Gunn
SVM
Toolbox.
samples
used
in
the
experiments.
The
Westontoynonliner
data
set
consists
of
1000
samples
For
SVM
and
AdaBoostSVM,
set
the
Gaussian
width

of
2
classes,
each
sample
having
52
attributes.
The
Wine
ai
of
RBFSVM
to
2,
6,
and
12,
the
average
correct
data
set
consists
of
178
samples
of
3
classes,

each
sample
classification
rates
for
randomly
chosen
testing
data
sets
having
13
attributes,
class
1
is
used
as
the
positive
class
are
calculated.
and
the
other
two
classes
belong
to

the
negative
class
in
For
variable
ci-AdaBoostRBFSVM
and
AdaBoost-
the
classification
experiments.
SVM,
1/2-1/8
of
the
training
samples
are
used
to
train
the
The
training
of
SVMs
for
the
variable

ci-
base
classifiers,
and
the
average
correct
classification
AdaBoostRBFSVM,
AdaBoostSVM
and
SVM
are
under
rates
for
3
randomly
chosen
testing
data
sets
are
the
same
parameter
when
comparing
the
performance

of
calculated.
the
algorithms,
C=
1000.
For
SVM
and
AdaBoostSVM,
Fig.2
gives
the
results
of
performance
comparison
for
set
the
Gaussian
width
ai
of
RBFSVM
to
12.
Let
T
be

the
the
Wine
data
set,
axis
X
indicates
the
number
of
training
number
of
base
classifiers
and
T=10
in
the
experiments,
samples,
and
axis
Y
gives
the
correct
classification
rates.

For
the
Westontoynonliner
data
set,
the
training
and
In
Fig.2,
Ada-SVM
stands
for
AdaBoostSVM,
and
testing
samples
are
chosen
randomly
from
the
given
improved
Ada-SVM
stands
for
variable
ci-AdaBoost-
datasets,

50,150,200,300,500
are
the
numbers
of
training
RB3FSVM.
samples
used
in
the
experiments,
and
128
is
the
number
From
Fig.
1
and
Fig.2
we
can
know
that
comparing
of
testing
samples

used
in
the
experiments.
AdaBoostSVM
with
a
single
SVM,
they
have
almost
the
For
variable
ci-AdaBoostRBFSVM
and
AdaBoost-
same
classification
performances,
but
our
improved
SVM,
1/2-1/10
of
the
training
samples

are
used
to
train
AdaBoostRBFSVM
improves
the
average
correct
the
base
classifiers,
and
the
average
correct
classification
classification
rates
obviously.
rates
for
3
randomly
chosen
testing
data
sets
are
For

the
Wine
data
set,
the
distribution
of
the
training
calculated.
samples
is
unbalanced,
because
there
are
59
training
Fig.
1
gives
the
results
of
performance
comparison
for
samples
in
the

positive
class
and
119
training
samples
in
the
Westontoynonliner
data
set,
axis
X
indicates
the
the
negative
class.
From
Fig.2
we
can
know
that
the
951
variable
a-AdaBoostRBFSVM
is
more

efficient
for
[4]
Y.
Freund,
R.
E.
Schapire,
"A
decision-theoretic
unbalanced
data
set.
generalization
of
online
learning
and
an
application
to
Compared
with
using
a
single
SVM,
the
benefit
of

boosting",
Journal
of
Computer
and
System
Sciences,
using
the
improved
AdaBoostRBFSVM
is
its
advantage
vol.
55,no.
1,
pp.119-139,
August
1997.
of
model
selection;
and
compared
with
using
AdaBoost
[5]
R.

E.
Schapire,
Y.
Singer,
P.
Bartlett,
and
W.
Lee,
of
a
single
and
fixed
a
to
all
RBFSVM
base
classifiers,
it
"Boosting
the
margin:
A
new
explanation
for
the
has

better
generation
performance.
effectiveness
of
voting
methods,"
The
Annals
of
Statistics,
vol.
26,
no.
5,
pp.
1651-1686,
1998.
[6]
Vladimir
Vapnik,
Statistical
Learning
Theory,
John
5.
CONCLUSIONS
Wiley
and
Sons

Inc.,
New
York,
1998.
[7]
G.
Valentini,
T.
G.
Dietterich,
"Bias-variance
AdaBoost
is
a
general
method
for
improving
the
analysis
of
support
vector
machines
for
the
accuracy
of
any
given

learning
algorithm.
After
development
of
svm-based
ensemble
methods",
analyzing
the
relation
between
the
performance
of
Journal
of
Machine
Learning
Research,
vol.
5,
pp.
AdaBoost
and
the
performance
of
base
classifiers,

the
725-775,
2004.
approach
of
improving
the
classification
performance
of
[8]
Dmitry
Pavlov,
Jianchang
Mao,
"Scaling-up
Support
AdaBoostSVM
was
studied
in
this
paper.
Vector
Machines
Using
Boosting
Algorithm",
In
There

is
inconsistency
existed
between
the
accuracy
Proceedings
of
ICPR
2000.
and
diversity
of
base
classifiers,
and
the
inconsistency
[9]
Hyun-Chul
Kim,
Shaoning
Pang,
Hong-Mo
Je,
affect
generalization
performance
of
the

algorithm.
How
Daijin
Kim,
and
Sung
Yang
Bang,
"Constructing
to
deal
with
the
dilemma
between
base
classifier's
support
vector
machine
ensemble,"
Pattern
accuracy
and
diversity
of
AdaBoost
is
very
important

for
Recognition,
vol.
36,
no.
12,
pp.
2757-2767,
Dec
improving
the
performance
of
AdaBoost.
A
new
variable
2003.
a-AdaBoostSVM
was
proposed
by
adjusting
the
kernel
[10]Xuchun
Li,
Lei
Wang,
Eric

Sung,
"A
Study
of
function
parameter
of
the
base
classifier
based
on
the
AdaBoost
with
SVM
Based
Weak
Learners",
In
distribution
of
training
samples.
The
proposed
algorithm
Proceedings
of
IJCNN

2005.
improves
the
classification
performance
by
making
a
[11]
P.
Melville,
R.
J.
Mooney,
"Creating
diversity
in
balance
between
the
accuracy
and
diversity
of
base
ensembles
using
articial
data",
Information

Fusion,
classifiers.
Experimental
results
for
the
benchmark
vol.
6,
no.
1,pp.
99-111,
Mar
2005.
dataset
indicate
the
effectiveness
of
the
proposed
[12]
I.
K.
Ludmila,
J.
W.
Christopher,
"Measures
of

algorithm.
The
experimental
results
also
indicate
that
the
diversity
in
classifier
ensembles
and
their
relationship
proposed
algorithm
is
more
efficient
for
unbalanced
data
with
the
ensemble
accuracy",
Machine
Learning,
vol.

set.
51,
no.
2,pp.
181-207,
May
2003.
[13]
H.
W.
Shin
and
S.
Y.
Sohn,
"Selected
tree
classifier
combination
based
on
both
accuracy
and
error
diversity,"
Pattern
Recognition,
vol.
38,

pp.
191-197,
Acknowledgements
2005.
[14]
Thomas
G.
Dietterich,
"An
experimental
This
work
is
supported
by
Natural
Science
Basic
comparison
of
three
methods
for
constructing
Research
Plan
in
Shaanxi
Province
of

China
under
Grant
ensembles
of
decision
trees:
Bagging,
boosting,
and
2004F36,
and
partially
supported
by
NSFC
under
Grant
randomization,"
Machine
Learning,
vol.
40,
no.
2,
50505051.
pp.
139-157,
Aug
2000.

References
[1]
L.
G.
Valiant,
"A
theory
of
the
learnable",
Communications
of
the
ACM,
vol.
27,
no.
11,
pp.1134-1142,
November
1984.
[2]
R.
E.
Schapire,
"The
strength
of
weak
learnability",

Machine
Learning,
vol.
5,
no.
2,
pp.
197-
227,
1990.
[3]
Yoav
Freund,
"Boosting
a
weak
learning
algorithm
by
majority",
Information
and
Computation,
vol.
121,
no.
2,
pp.256-285,
1995.
952