Deep learning for natural language processing CCF ADL 20160529

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

❳✐♣❡♥❣ ◗✐✉
①♣q✐✉❅❢✉❞❛♥✳❡❞✉✳❝♥
❤tt♣✿✴✴♥❧♣✳❢✉❞❛♥✳❡❞✉✳❝♥
❤tt♣✿✴✴♥❧♣✳❢✉❞❛♥✳❡❞✉✳❝♥
❋✉❞❛♥ ❯♥✐✈❡rs✐t②

✷✵✶✻✴✺✴✷✾✱ ❈❈❋ ❆❉▲✱ ❇❡✐❥✐♥❣

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶ ✴ ✶✸✶


❚❛❜❧❡ ♦❢ ❈♦♥t❡♥ts
✶

✷

✸

✹

❇❛s✐❝ ❈♦♥❝❡♣ts
❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡
▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣
❉❡❡♣ ▲❡❛r♥✐♥❣
◆❡✉r❛❧ ▼♦❞❡❧s ❢♦r ❘❡♣r❡s❡♥t❛t✐♦♥ ▲❡❛r♥✐♥❣

●❡♥❡r❛❧ ❆r❝❤✐t❡❝t✉r❡
❈♦♥✈♦❧✉t✐♦♥❛❧ ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rr❡♥t ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rs✐✈❡ ◆❡✉r❛❧ ◆❡t✇♦r❦
❆tt❡♥t✐♦♥ ▼♦❞❡❧
❆♣♣❧✐❝❛t✐♦♥s
◗✉❡st✐♦♥ ❆♥s✇❡r✐♥❣
▼❛❝❤✐♥❡ ❚r❛♥s❧❛t✐♦♥
❚❡①t ▼❛t❝❤✐♥❣
❈❤❛❧❧❡♥❣❡s ✫ ❖♣❡♥ Pr♦❜❧❡♠s

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✷ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❚❛❜❧❡ ♦❢ ❈♦♥t❡♥ts
✶

✷

✸

✹

❇❛s✐❝ ❈♦♥❝❡♣ts
❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡
▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣
❉❡❡♣ ▲❡❛r♥✐♥❣
◆❡✉r❛❧ ▼♦❞❡❧s ❢♦r ❘❡♣r❡s❡♥t❛t✐♦♥ ▲❡❛r♥✐♥❣
●❡♥❡r❛❧ ❆r❝❤✐t❡❝t✉r❡
❈♦♥✈♦❧✉t✐♦♥❛❧ ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rr❡♥t ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rs✐✈❡ ◆❡✉r❛❧ ◆❡t✇♦r❦
❆tt❡♥t✐♦♥ ▼♦❞❡❧
❆♣♣❧✐❝❛t✐♦♥s
◗✉❡st✐♦♥ ❆♥s✇❡r✐♥❣
▼❛❝❤✐♥❡ ❚r❛♥s❧❛t✐♦♥
❚❡①t ▼❛t❝❤✐♥❣
❈❤❛❧❧❡♥❣❡s ✫ ❖♣❡♥ Pr♦❜❧❡♠s

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✸ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❇❡❣✐♥ ✇✐t❤ ✏❆■✑

❍✉♠❛♥✿ ▼❡♠♦r②✱ ❈♦♠♣✉t❛t✐♦♥

❈♦♠♣✉t❡r✿ ▲❡❛r♥✐♥❣✱ ❚❤✐♥❦✐♥❣✱ ❈r❡❛t✐✈✐t②

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✹ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❚✉r✐♥❣ ❚❡st

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✺ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❉❡✜♥✐t✐♦♥ ❢r♦♠ ❲✐❦✐♣❡❞✐❛
❆rt✐✜❝✐❛❧ ✐♥t❡❧❧✐❣❡♥❝❡ ✭❆■✮ ✐s t❤❡ ✐♥t❡❧❧✐❣❡♥❝❡ ❡①❤✐❜✐t❡❞ ❜② ♠❛❝❤✐♥❡s✳
❈♦❧❧♦q✉✐❛❧❧②✱ t❤❡ t❡r♠ ✏❛rt✐✜❝✐❛❧ ✐♥t❡❧❧✐❣❡♥❝❡✑ ✐s ❧✐❦❡❧② t♦ ❜❡ ❛♣♣❧✐❡❞ ✇❤❡♥ ❛
♠❛❝❤✐♥❡ ✉s❡s ❝✉tt✐♥❣✲❡❞❣❡ t❡❝❤♥✐q✉❡s t♦ ❝♦♠♣❡t❡♥t❧② ♣❡r❢♦r♠ ♦r ♠✐♠✐❝
✏❝♦❣♥✐t✐✈❡✑ ❢✉♥❝t✐♦♥s t❤❛t ✇❡ ✐♥t✉✐t✐✈❡❧② ❛ss♦❝✐❛t❡ ✇✐t❤ ❤✉♠❛♥ ♠✐♥❞s✱ s✉❝❤
❛s ✏❧❡❛r♥✐♥❣✑ ❛♥❞ ✏♣r♦❜❧❡♠ s♦❧✈✐♥❣✑✳
❘❡s❡❛r❝❤ ❚♦♣✐❝s
❑♥♦✇❧❡❞❣❡ ❘❡♣r❡s❡♥t❛t✐♦♥
▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣
◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣
❈♦♠♣✉t❡r ❱✐s✐♦♥
···
❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✻ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❈❤❛❧❧❡♥❣❡✿ ❙❡♠❛t✐❝ ●❛♣

❚❡①t ✐♥ ❍✉♠❛♥

床前明月光，
疑是地上霜。
举头望明月，

低头思故乡。

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✼ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❈❤❛❧❧❡♥❣❡✿ ❙❡♠❛t✐❝ ●❛♣

❚❡①t ✐♥ ❈♦♠♣✉t❡r
✶✶✶✵✵✶✵✶✶✵✶✶✶✵✶✵✶✵✵✵✶✵✶✵✶✶✶✵✵✶✵✶✶✵✵✵✶✵✵✶✶✵✵✵✶✶✵✶✶✶✶✵✵✶✶✵✶✵✵✶✶✵✵✵✶✵✵✵✶
✶✶✵✶✶✶✵✵✶✶✵✶✵✵✶✶✶✵✵✶✵✵✵✶✵✵✵✶✶✶✵✵✶✵✶✶✵✵✵✵✶✵✶✶✵✵✵✶✵✵✶✶✶✶✵✶✶✶✶✶✵✶✶✶✶✵✵✶✵✵✵✶✶✵✵
✶✶✶✵✵✶✶✶✶✵✵✶✵✶✶✵✶✵✵✶✵✵✵✶✶✶✶✵✵✶✶✵✶✵✵✶✶✵✵✵✶✵✶✵✶✶✶✶✶✶✶✵✵✶✵✶✶✵✵✶✶✶✵✵✶✵✶✶✵
✵✵✵✶✶✶✵✵✶✵✵✶✵✶✶✶✵✵✵✶✵✵✵✶✵✶✵✶✶✶✵✶✵✵✶✶✵✵✶✶✶✵✵✶✵✵✶✶✶✵✵✶✶✶✵✵✵✶✶✶✵✵✵✵✵✵✵✶✵✵✵✵✵✶✵
✵✵✶✵✵✵✵✵✵✵✶✵✵✵✵✵✵✵✶✵✵✵✵✵✵✵✵✵✶✵✶✵✶✶✶✵✵✶✵✵✶✵✶✶✶✵✵✵✶✵✶✶✶✶✶✵✶✶✶✵✵✶✵✶✶✵✶✵✵✶
✵✵✶✵✶✶✵✶✵✵✶✶✶✵✵✶✶✵✶✵✵✶✶✶✵✵✶✵✵✶✶✵✶✶✶✶✶✵✵✶✶✵✶✵✵✶✶✵✵✵✶✵✵✵✶✶✶✵✶✶✶✵✵✶✶✵✶✵✵✶✶✶✵✵
✶✵✵✵✶✵✵✵✶✶✶✵✶✶✶✶✶✵✶✶✶✶✵✵✶✵✵✵✶✶✵✵✵✵✵✵✶✵✶✵✶✶✶✵✵✶✵✵✶✵✶✶✶✶✵✶✶✵✵✵✶✶✶✵✶✶✶✵✵✶
✵✶✶✵✶✵✵✶✵✵✶✵✶✶✵✶✵✵✶✶✶✵✵✶✶✵✶✵✵✵✵✵✵✵✶✵✵✶✶✶✵✶✶✶✶✵✵✶✶✵✶✵✵✶✵✶✵✶✶✵✵✵✵✶✵✶✶✶✶✵✵✶✵✵✶✵
✶✶✶✵✵✶✶✵✶✵✵✵✵✶✶✶✶✵✵✵✶✶✶✵✵✵✵✵✵✵✶✵✵✵✵✵✶✵

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✽ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡

❈❤❛❧❧❡♥❣❡✿ ❙❡♠❛t✐❝ ●❛♣

❋✐❣✉r❡✿ ●✉❡r♥✐❝❛ ✭P✐❝❛ss♦✮

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✾ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

❚❛❜❧❡ ♦❢ ❈♦♥t❡♥ts
✶

✷

✸

✹

❇❛s✐❝ ❈♦♥❝❡♣ts
❆rt✐✜❝✐❛❧ ■♥t❡❧❧✐❣❡♥❝❡
▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣
❉❡❡♣ ▲❡❛r♥✐♥❣
◆❡✉r❛❧ ▼♦❞❡❧s ❢♦r ❘❡♣r❡s❡♥t❛t✐♦♥ ▲❡❛r♥✐♥❣
●❡♥❡r❛❧ ❆r❝❤✐t❡❝t✉r❡
❈♦♥✈♦❧✉t✐♦♥❛❧ ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rr❡♥t ◆❡✉r❛❧ ◆❡t✇♦r❦
❘❡❝✉rs✐✈❡ ◆❡✉r❛❧ ◆❡t✇♦r❦
❆tt❡♥t✐♦♥ ▼♦❞❡❧
❆♣♣❧✐❝❛t✐♦♥s
◗✉❡st✐♦♥ ❆♥s✇❡r✐♥❣
▼❛❝❤✐♥❡ ❚r❛♥s❧❛t✐♦♥
❚❡①t ▼❛t❝❤✐♥❣
❈❤❛❧❧❡♥❣❡s ✫ ❖♣❡♥ Pr♦❜❧❡♠s

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✵ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

■♥♣✉t✿ x

▼♦❞❡❧

❚r❛✐♥✐♥❣ ❉❛t❛✿ (x, y )

▲❡❛r♥✐♥❣ ❆❧❣♦r✐t❤♠

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

❖✉t♣✉t✿ y

✶✶ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

❇❛s✐❝ ❈♦♥❝❡♣ts ♦❢ ▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

■♥♣✉t ❉❛t❛✿ (xi , yi )✱✶ ≤ i ≤ m
▼♦❞❡❧✿
▲✐♥❡❛r ▼♦❞❡❧✿ y = f (x) = w T x + b
●❡♥❡r❛❧✐③❡❞ ▲✐♥❡❛r ▼♦❞❡❧✿ y = f (x) = w T φ(x) + b
◆♦♥✲❧✐♥❡❛r ▼♦❞❡❧✿ ◆❡✉r❛❧ ◆❡t✇♦r❦
❈r✐t❡r✐♦♥✿
▲♦ss ❋✉♥❝t✐♦♥✿

L(y , f (x)) → ❖♣t✐♠✐③❛t✐♦♥
❊♠♣✐r✐❝❛❧ ❘✐s❦✿
m
Q(θ) = m✶ · i=✶ L(yi , f (xi , θ)) → ▼✐♥✐♠✐③❛t✐♦♥
❘❡❣✉❧❛r✐③❛t✐♦♥✿ θ ✷
❖❜❥❡❝t✐✈❡ ❋✉♥❝t✐♦♥✿ Q(θ) + λ θ ✷

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✷ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦ss ❋✉♥❝t✐♦♥

●✐✈❡♥ ❛♥ t❡st s❛♠♣❧❡ (x, y )✱ t❤❡ ♣r❡❞✐❝t❡❞ ❧❛❜❡❧ ✐s f (x, θ)
✵✲✶ ▲♦ss

L(y , f (x, θ)) =

✵ ✐❢ y = f (x, θ)
✶ ✐❢ y = f (x, θ)

✭✶✮

= I (y = f (x, θ)),

✭✷✮

❤❡r❡ I ✐s ✐♥❞✐❝❛t♦r ❢✉♥❝t✐♦♥✳
◗✉❛❞r❛t✐❝ ▲♦ss
L(y , yˆ) = (y − f (x, θ))✷

✭✸✮

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✸ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦ss ❋✉♥❝t✐♦♥

❈r♦ss✲❡♥tr♦♣② ▲♦ss ❲❡ r❡❣❛r❞ fi (x, θ) ❛s t❤❡ ❝♦♥❞✐t✐♦♥❛❧ ♣r♦❜❛❜✐❧✐t② ♦❢ ❝❧❛ss
i✳
C

fi (x, θ) ∈ [✵, ✶],

fi (x, θ) = ✶

✭✹✮

i=✶

fy (x, θ) ✐s t❤❡ ❧✐❦❡❧✐❤♦♦❞ ❢✉♥❝t✐♦♥ ♦❢ y ✳ ◆❡❣❛t✐✈❡ ▲♦❣ ▲✐❦❡❧✐❤♦♦❞ ❢✉♥❝t✐♦♥ ✐s
L(y , f (x, θ)) = − ❧♦❣ fy (x, θ).

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✭✺✮

✶✹ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦ss ❋✉♥❝t✐♦♥

❲❡ ✉s❡ ♦♥❡✲❤♦t ✈❡❝t♦r ② t♦ r❡♣r❡s❡♥t ❝❧❛ss c ✐♥ ✇❤✐❝❤ yc = ✶ ❛♥❞ ♦t❤❡r
❡❧❡♠❡♥ts ❛r❡ ✵✳
◆❡❣❛t✐✈❡ ▲♦❣ ▲✐❦❡❧✐❤♦♦❞ ❢✉♥❝t✐♦♥ ❝❛♥ ❜❡ r❡✇r✐tt❡♥ ❛s
C

yi ❧♦❣ fi (x, θ).

L(y , f (x, θ)) = −

✭✻✮

i=✶

yi ✐s ❞✐str✐❜✉t✐♦♥ ♦❢ ❣♦❧❞ ❧❛❜❡❧s✳ ❚❤✉s✱ ❊q ✻ ✐s ❈r♦ss ❊♥tr♦♣② ▲♦ss ❢✉♥❝t✐♦♥✳

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✺ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦ss ❋✉♥❝t✐♦♥

❍✐♥❣❡ ▲♦ss ❋♦r ❜✐♥❛r② ❝❧❛ss✐✜❝❛t✐♦♥✱ y ❛♥❞ f (x, θ) ❛r❡ ✐♥ {−✶, +✶}✳ ❍✐♥❣❡
▲♦ss ✐s

L(y , f (x, θ)) = ♠❛① (✵, ✶ − yf (x, θ))

✭✼✮

= |✶ − yf (x, θ)|+ .

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✭✽✮

✶✻ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦ss ❋✉♥❝t✐♦♥

❋♦r ❜✐♥❛r② ❝❧❛ss✐✜❝❛t✐♦♥✱ y ❛♥❞ f (x, θ) ❛r❡ ✐♥ {−✶, +✶}✳
z = yf (x, θ)✳

❤tt♣✿✴✴✇✇✇✳❝s✳❝♠✉✳❡❞✉✴ ②❛♥❞♦♥❣❧✴❧♦ss✳❤t♠❧
❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✼ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

P❛r❛♠❡t❡r ▲❡❛r♥✐♥❣

■♥ ▼▲✱ ♦✉r ♦❜❥❡❝t✐✈❡ ✐s t♦ ❧❡❛r♥ t❤❡ ♣❛r❛♠❡t❡r θ t♦ ♠✐♥✐♠✐③❡ t❤❡ ❧♦ss
❢✉♥❝t✐♦♥✳

θ∗ = ❛r❣ ♠✐♥ R(θt )

✭✾✮

θ

✶
= ❛r❣ ♠✐♥
N
θ

N

L y (i) , f (x (i) , θ) .

✭✶✵✮

i=✶

●r❛❞✐❡♥t ❉❡s❝❡♥t✿

❛t+✶ = ❛t − λ
= ❛t − λ

∂R(θ)

∂θt
N
i=✶

∂R θt ; x (i) , y (i)
,
∂θ

✭✶✶✮
✭✶✷✮

λ ✐s ❛❧s♦ ❝❛❧❧❡❞ ▲❡❛r♥✐♥❣ ❘❛t❡ ✐♥ ▼▲✳
❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✶✽ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

❙t♦❝❤❛st✐❝ ●r❛❞✐❡♥t ❉❡s❝❡♥t ✭❙●❉✮

❛t+✶ = ❛t − λ

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

∂R θt ; x (t) , y (t)

,
∂θ

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✭✶✸✮

✶✾ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

❚✇♦ ❚r✐❝❦s ♦❢ ❙●❉

❊❛r❧②✲❙t♦♣
❙❤✉✤❡

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✷✵ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲✐♥❡❛r ❈❧❛ss✐✜❝❛t✐♦♥

❋♦r ❜✐♥❛r② ❝❧❛ss✐✜❝❛t✐♦♥ y ∈ {✵, ✶}✱ t❤❡ ❝❧❛ss✐✜❡r ✐s

yˆ =

✶
✵

✇❚ ① > ✵
= I (✇❚ ① > ✵),
❚
✐❢ ✇ ① ≤ ✵
✐❢

✭✶✹✮

✇❚
①=

✵

x✶

✇
✇ ✶✇

x✷

❋✐❣✉r❡✿ ❇✐♥❛r② ▲✐♥❡❛r ❈❧❛ss✐✜❝❛t✐♦♥

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✷✶ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦❣✐st✐❝ ❘❡❣r❡ss✐♦♥

❍♦✇ t♦ ❧❡❛r♥ t❤❡ ♣❛r❛♠❡t❡r ✇✿ P❡r❝❡♣tr♦♥✱ ▲♦❣✐st✐❝ ❘❡❣r❡ss✐♦♥✱ ❡t❝✳
❚❤❡ ♣♦st❡r✐♦r ♣r♦❜❛❜✐❧✐t② ♦❢ y = ✶ ✐s

P(y = ✶|①) = σ(✇❚ ①) =

✶
,
✶ + ❡①♣(−✇❚ ①)

✭✶✺✮

✇❤❡r❡✱ σ(·) ✐s ❧♦❣✐st✐❝ ❢✉♥❝t✐♦♥✳
❚❤❡ ♣♦st❡r✐♦r ♣r♦❜❛❜✐❧✐t② ♦❢ y = ✵ ✐s P(y = ✵|①) = ✶ − P(y = ✶|①)✳

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✷✷ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▲♦❣✐st✐❝ ❘❡❣r❡ss✐♦♥

●✐✈❡♥ n s❛♠♣❧❡s (x (i) , y (i) ), ✶ ≤ i ≤ N ✱ ✇❡ ✉s❡ t❤❡ ❝r♦ss✲❡♥tr♦♣② ❧♦ss
❢✉♥❝t✐♦♥✳
N

y (i) ❧♦❣ σ(✇❚ ①(i) ) + (✶ − y (i) ) ❧♦❣ ✶ − σ(✇❚ ①(i) )

J (✇ ) = −

✭✶✻✮

i=✶

❚❤❡ ❣r❛❞✐❡♥t ♦❢ J (✇) ✐s
∂J (✇)
=
∂✇

N
(i)

①

· σ(✇❚ ①(i) ) − y (i)

✭✶✼✮

i=✶

■♥✐t✐❛❧✐③❡ ✇✵ = ✵✱ ❛♥❞ ✉♣❞❛t❡

✇t+✶ = ✇t + λ

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

∂J (✇)
,
∂✇

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✭✶✽✮

✷✸ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

▼✉❧t✐❝❧❛ss ❈❧❛ss✐✜❝❛t✐♦♥

●❡♥❡r❛❧❧②✱ y = {✶, · · · , C }✱ ✇❡ ❞❡✜♥❡ C ❞✐s❝r✐♠✐♥❛♥t ❢✉♥❝t✐♦♥s

fc (①) = ✇c❚ ①,

c = ✶, · · · , C ,

✇❤❡r❡ ✇c ✐s ✇❡✐❣❤t ✈❡❝t♦r ♦❢ ❝❧❛ss c ✳
❚❤✉s✱
C
yˆ = ❛r❣ ♠❛① ✇c❚ ①
c=✶

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✭✶✾✮

✭✷✵✮

✷✹ ✴ ✶✸✶


❇❛s✐❝ ❈♦♥❝❡♣ts

▼❛❝❤✐♥❡ ▲❡❛r♥✐♥❣

❙♦❢t♠❛① ❘❡❣r❡ss✐♦♥

❙♦❢t♠❛① r❡❣r❡ss✐♦♥ ✐s ❛ ❣❡♥❡r❛❧✐③❛t✐♦♥ ♦❢ ❧♦❣✐st✐❝ r❡❣r❡ss✐♦♥ t♦ ♠✉❧t✐✲❝❧❛ss
❝❧❛ss✐✜❝❛t✐♦♥ ♣r♦❜❧❡♠s✳
❲✐t❤ s♦❢t♠❛①✱ t❤❡ ♣♦st❡r✐♦r ♣r♦❜❛❜✐❧✐t② ♦❢ y = c ✐s

P(y = c|①) = s♦❢t♠❛①(✇c❚ ①) =

✇ ①)
.
✇i ①)

❡①♣( c
C
i=✶ ❡①♣(

✭✷✶✮

❚♦ r❡♣r❡s❡♥t ❝❧❛ss c ❜② ♦♥❡✲❤♦t ✈❡❝t♦r

② = [I (✶ = c), I (✷ = c), · · · , I (C = c)]❚ ,

✭✷✷✮

✇❤❡r❡ I () ✐s ✐♥❞✐❝t♦r ❢✉♥❝t✐♦♥✳

❳✐♣❡♥❣ ◗✐✉ ✭❋✉❞❛♥ ❯♥✐✈❡rs✐t②✮

❉❡❡♣ ▲❡❛r♥✐♥❣ ❢♦r ◆❛t✉r❛❧ ▲❛♥❣✉❛❣❡ Pr♦❝❡ss✐♥❣

✷✺ ✴ ✶✸✶