Slide tria tuệ nhân tạo learning decision trees

Introduction to
Artificial Intelligence
Chapter4:Learning
(1)LearningDecisionTrees
NguyễnHảiMinh,Ph.D


CuuDuongThanCong.com

/>

Outline
q FormofLearning
q LearningfromDecisionTrees
q Summary

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

2
/>

LearningAgents–Whylearning?
1.  Unknownenvironments
•  i.e.,arobotdesignedtonavigatemazesmustlearnthelayoutof
eachnewmazeitencounters.

2.  Environmentchangesovertime
• i.e.,Anagentdesignedtopredicttomorrow’sstockmarket

pricesmustlearntoadaptwhenconditionschangefrom
boomtobust.

3.  Noideahowtoprogramasolution
• i.e.,thetasktorecognizingthefacesoffamilymembers.

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

3
/>

Learningelement
q Designofalearningelementisaffectedby

o  Whichcomponentsistobeimproved
o  Whatpriorknowledgetheagentalreadyhas
o  Whatrepresentationisusedforthecomponents
o  Whatfeedbackisavailabletolearnthesecomponents

q Typeoffeedback:

o  Supervisedlearning:correctanswersforeach
example
o  Unsupervisedlearning:correctanswersnotgiven
o  Reinforcementlearning:occasionalrewards

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

4
/>

SupervisedLearning
q Simplestform:learnafunctionfromexamples
q Problem:givenatrainingsetofNexampleinputoutputpairs
(x1,y1),(x2,y2),…,(xN,yN)

Whereeachyjwasgeneratedbyanunknownfunctiony=f(x)

àFindahypothesishsuchthath≈f
q Tomeasuretheaccuracyofahypothesiswegiveit
atestsetofexamplesthataredifferentwiththe
trainingset.

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

5
/>

SupervisedLearning
Consistentlinearfit

Consistent7thorder
polynomialfit

Inconsistentlinearfit.
Consistent6thorder
polynomialfit.

Consistentsinusoidal
fit

• Constructhsothatitagreeswithf.
• Thehypothesishisconsistentifitagreeswithfonall
observations.
• Ockham’srazor:Selectthesimplestconsistenthypothesis.
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

6
/>

Learningproblems
h(x)=thepredictedoutputvaluefortheinputx.
q Discretevaluedfunction⇒classiCication
q Continuousvaluedfunction⇒regression

7/25/18

NguyễnHảiMinh@FIT-HCMUS

CuuDuongThanCong.com

7
/>

ClassiCication
q Isthisnumber9?
o  2classes:Yes/No
q Willyoupassorfailtheexam?
o  2classes:Fail/Pass

q Isthisanapple,anorangeora
tomato?
o  3classes:Apple/Orange/
Tomato

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

8
/>

Regression
q Estimatingthepriceofahouse

7/25/18

NguyễnHảiMinh@FIT-HCMUS

CuuDuongThanCong.com

9
/>

AclassiCicationproblemexample
Predictingwhetheracertainperson
willwaittohaveaseatinarestaurant.

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

10
/>

LearningDecisiontrees
q  “Divideandconquer”:Splitdata
intosmallerandsmallersubsets

x1 > α ?

q  Splitsusuallyonasinglevariable

x2 > β ?
yes

7/25/18

no

yes

no

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

x2 > γ ?
no

yes

11
/>

Thewait@restaurantdecisiontree
Thisisourtruefunction.
Canwelearnthistreefromexamples?

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

12
/>

Inductivelearningofdecisiontree

q Simplest:Constructadecisiontreewithoneleafforevery
example=memorybasedlearning.
Notverygoodgeneralization.
q Advanced:Splitoneachvariablesothatthepurityofeach
splitincreases(i.e.eitheronlyyesoronlyno)
q Puritymeasured,e.g,withentropy
o  EntropyisameasureoftheuncertaintyofarandomvariableVwith
onevaluevk

o  vk:1classinV(yes/noinbinaryclassiCication)
o  P(vk):theproportionofthenumberofelementsinclassvktothe
numberofelementsinV
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

13
/>

Entropy
q Entropyisameasureoftheuncertaintyofa
randomvariablewithonlyonevalue
Theentropyismaximalwhen
allpossibilitiesareequally
likely.

Thegoalofthedecisiontree
istodecreasetheentropyin
eachnode.

Entropyiszeroinapure”yes”
node(orpure”no”node).
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

14
/>

Decisiontreelearningexample
Problem:decidewhethertowaitforatableatarestaurant,
basedonthefollowingattributes:
1.  Alternate:isthereanalternativerestaurantnearby?
2.  Bar:isthereacomfortablebarareatowaitin?
3.  Fri/Sat:istodayFridayorSaturday?
4.  Hungry:arewehungry?
5.  Patrons:numberofpeopleintherestaurant(None,
Some,Full)
6.  Price:pricerange($,$$,$$$)
7.  Raining:isitrainingoutside?
8.  Reservation:havewemadeareservation?
9.  Type:kindofrestaurant(French,Italian,Thai,
Burger)
10. WaitEstimate:estimatedwaitingtime(0-10,
10-30,30-60,>60)
7/25/18

NguyễnHảiMinh@FIT-HCMUS

CuuDuongThanCong.com

15
/>

Decisiontreelearningexample

( 12)log (612)− (612)log (612) = 01.30

Entropy
H(S) = − 6
7/25/18

2

2

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

T=True,F=False

6True,
6False
16

/>

Decisiontreelearningexample
Alternate?

Yes

No

3T,3F

3T,3F

q CalculateAverageEntropyofattributeAlternate:
AEAlternate=P(Alt=T)xH(Alt=T)+P(Alt=F)xH(Alt=F)



AE Alternate



=

6⎡ 3
⎤ + 6 ⎡− 3 log 3 − 3 log 3 ⎤ = 1
3
3
3
−
log
−
log
2
2

2
2
6
6
6
6 ⎦ 12 ⎣
6
6
6
6⎦
12 ⎣

( )

( ) ( )

( )

( )

( ) ( )

q InformationGained(differenceinentropyfrombeforeto
afterthesetSissplitonattributeAlternate)
IG(Alternate,S)=H(S)–AEAlternate=1–1=0
7/25/18

NguyễnHảiMinh@FIT-HCMUS

CuuDuongThanCong.com

/>
( )

17


Decisiontreelearningexample
Bar?
Yes

No

3T,3F

AE Bar =

3T,3F

6⎡ 3
⎤ + 6 ⎡− 3 log 3 − 3 log 3 ⎤ = 1
3
3
3
−
log
−
log
2
2

2
2
6
6
6
6 ⎦ 12 ⎣
6
6
6
6⎦
12 ⎣

( )

( ) ( )

( )

( )

( ) ( )

( )

IG(Bar,S)=H(S)–AEBar=1–1=0

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

18
/>

Decisiontreelearningexample
Sat/Fri?
Yes

No

2T,3F

4T,3F

5⎡ 2
⎤+
3
3
2
−
log
−
log
2
2
5
5
5
5⎦
12 ⎣

7⎡ 4
4 − 3 log 2 3 ⎤ = 0.979
−
log
2
7
7
7
7⎦
12 ⎣
AE Sat/Fri? =

( )

( ) ( ) ( )
( ) ( ) ( )

( )

IG(Sat/Fri,S)=H(S)–AESat/Fri=1–0.979=0.021
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

19
/>

Decisiontreelearningexample
Hungry?

Yes

No

5T,2F

AE Hungry =

1T,4F

7⎡ 5
⎤ + 5 ⎡− 1 log 1 − 4 log 4 ⎤ = 0.804
5
2
2
−
log
−
log
2
2
2
2
7
7
7
7 ⎦ 12 ⎣
5
5
5

5⎦
12 ⎣

( )

( ) ( )

( )

( )

( ) ( )

( )

IG(Hungry,S)=H(S)–AEHungry=1–0.804=0.196

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

20
/>

Decisiontreelearningexample
Raining?
Yes

No

2T,2F

AE Raining =

4T,4F

4
8
− 2 log2 2 − 2 log2 2 +
− 4 log2 4 − 4 log2 4 = 01.30
4
4
4
4 12
8
8
8
8
12

[ ( ) ( ) ( ) ( )]

[ ( ) ( ) ( ) ( )]

IG(Raining,S)=H(S)–AERaining=1–1=0

7/25/18

NguyễnHảiMinh@FIT-HCMUS

CuuDuongThanCong.com

21
/>

Decisiontreelearningexample
ReservaUon?
Yes

No

3T,2F

AE Reservation =

3T,4F

5⎡ 3
3 − 2 log 2 2 ⎤ + 7 ⎡− 3 log 2 3 − 4 log 2 4 ⎤ = 0.979
−
log
2
5
5
5
5 ⎦ 12 ⎣
7
7
7
7⎦

12 ⎣

( )

( ) ( )

( )

( )

( ) ( )

( )

IG(Reservation,S)=H(S)–AEReservation=1–0.979=0.021

7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

22
/>

Decisiontreelearningexample
Patrons?
None

Full
Some

2F

2T,4F
4T

2⎡ 0
⎤ + 4 ⎡− 4 log 4 − 0 log 0 ⎤
0
2
2
−
log
−
log
2
2
2
2
2
2
2
2 ⎦ 12 ⎣
4
4
4
4⎦
12 ⎣
6
+ ⎡⎣− 2 log 2 2 − 4 log 2 4 ⎤⎦ = 0.541

6
6
6
6
12

( ) ( ) ( ) ( )
( ) ( ) ( ) ( )

AE Patrons =

( )

( ) ( )

( )

IG(Patrons,S)=H(S)–AEPatrons=1–0.541=0.459
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

23
/>

Decisiontreelearningexample
Price
$

$$$
$$

3T,3F

1T,3F
2T

6⎡ 3
2⎡ 2
⎤
⎤
3
3
3
0
0
2
AE Price = ⎣−
log 2
−
log 2
+
−
log
−
log
2
2
6

6
6
6 ⎦ 12 ⎣
2
2
2
2⎦
12
4
+ ⎡⎣− 1 log 2 1 − 3 log 2 3 ⎤⎦ = 0.770
4
4
4
4
12

( ) ( ) ( ) ( )
( ) ( ) ( ) ( )

( )

( ) ( )

( )

IG(Price,S)=H(S)–AEPrice=1–0.770=0.23
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

24
/>

Decisiontreelearningexample
Type
French

1T,1F

Burger

Italian
Thai

1T,1F

2T,2F

2T,2F

2⎡ 1
2⎡ 1
⎤
⎤
1
1
1
1
1

1
AE Type = ⎣−
log 2
−
log 2
+
−
log
−
log
2
2
2
2
2
2 ⎦ 12 ⎣
2
2
2
2⎦
12
4
4
+ ⎡⎣− 2 log 2 2 − 2 log 2 2 ⎤⎦ + ⎡⎣− 2 log 2 2 − 2 log 2 2 ⎤⎦ = 1
4
4
4
4
4
4

4
4
12
12

( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
( ) ( ) ( ) ( )
IG(Type,S)=H(S)–AEAlternate=1–1=0
7/25/18

NguyễnHảiMinh@FIT-HCMUS
CuuDuongThanCong.com

25
/>