PPt6 - Hopfield pdf
Bạn đang xem bản rút gọn của tài liệu. Xem và tải ngay bản đầy đủ của tài liệu tại đây (719.44 KB, 53 trang )
1
!!"
Recurrent Network có các hidden neuron: phần tử làm trễ z
-1
được dùng
- Đầu ra của Neural được feedback về tất cả các Neural
z
-1
z
-1
z
-1
2
!!"
#$$%&#$$'
Input: Pattern (thưng c nhiu hoc xung cp)
Output: Corresponding pattern (hon hảo/xét m$t c%ch tương đi
l ko c nhiu)
Process
–
$()*+, /&'012345)-6
()
–
7-(-8--9414*(-0:
1;-
–
<=1>?(-0?0/
3
!!"
@ABC$%
Input: Pattern (thưng c nhiu hoc xung cp)
Output: Corresponding pattern (hon hảo/xét m$t c%ch tương đi
l ko c nhiu)
Process
–
$()*+, /&'012345)-6
()
–
7-(-8--9414*(-0:
1;-
–
<=1>?(-0?0/
4
!!"
70(@A$%
!"#$
% !& '"()$
*+
,
-./01 23
* !4
&5#& ' !526
5
!!"
70(@A$%&'
% !7."()$
*'8 !#)
% !7."()$
*'8 !#)
6
!!"
DDE#
9
7:.;7'
<<:'' '6 4. = >4
4''!'6 ?' .246
' 6> 2 4''@
%??=AA
! '=42 2
.>'2 6 4 6 48'6
2 = .?
'?!
?44 =@@@
@. !
5 >526
B.must store ! .' ? ''
2 '' '
'
7
!!"
77)
C
D
E 2 2 ''4 4'. >'
D
! '= 4'
2
D
F!6 42=! ' 2 4
! '
D
G. ?42;
!.'
HH
HH
?
?
!.7'
?
?
8
!!"
FDDE#
I
72 ' . . 4'
'
∑
=
∝
P
p
pjpkjk
iiw
1
pjpkjk
iiw ∝∆
%?? J'4 '' ! '2=
.
*:2 2
44=
.
''6=.'' ! .' ''
∑
=
≡
P
p
pjpkjk
ii
P
w
1
1
:.#! .7'
pjpkjk
oiw ∝∆
%
∑
=
∝
P
p
pjpkjk
oiw
1
%
∑
=
≡
P
p
pjpkjk
oi
P
w
1
1
%
#2
#2
9
!!"
)$%
Auto-Association Network
Fully-connected (clique) with symmetric weights
State of node = f(inputs)
Weight values based on Hebbian principle
Performance: Must iterate a bit to converge on a pattern, but
generally
much less computation than in back-propagation networks.
A
42 6
))(sgn()1(
1
∑
=
=+
n
j
pjkjpk
txwtx
K '
10
!!"
)$%
The Hopfield network implements a so-called associative
(also called content addressable) memory.
A collection of patterns called fundamental memories is
stored in the NN by means of weights.
Each neuron represents an attribute (dimension,
feature) of the input.
The weight of the link between two neurons measures
the correlation between the two corresponding
attributes over the fundamental memories. If the weight
is high then the corresponding attributes are often equal
over the fundamental memories.
John Hopfield (Physicist, at Caltech in 1982, proposed
the model, now at Princeton)
11
!!"
G)HA
D
G &2
D
J6
D
? 6
.
D
B?64
2 4
%4'
72 .
12
!!"
)$$-H
5'''4? 62
'44?
L
L
.
E> 2'4%4'4
2 '
"
" "
?4>=>=>
E>
? ' 4
'6 4 6'6' .
13
!!"
I)$$
Input vectors values are in {-1,1} (or {0,1}).
The number of neurons is equal to the input dimension.
Every neuron has a link from every other neuron
(recurrent architecture) except itself (no self-feedback).
The neuron state at time n is its output value.
The network state at time n is the vector of neurons
states.
The activation function used to update a neuron state is
the sign function except if the input of the activation
function is 0 then the new output (state) of the neuron
is equal to the old one.
Weights are symmetric:
jiij
ww =
14
!!"
%%)-%-J
N: input dimension.
M: number of patterns (called fundamental memories)
are used to compute the weights.
i-th component of the fundamental memory.
State of neuron i at time n.
i
f
µ
µ
)(nx
i
15
!!"
1.Storage. Mf
1
=f
2
=N=f
M
4
2 '4 2 '22B.4
ji
.42OB
'24!f
μ
P=LQ6
2=6 . 4>
=
≠
=
∑
=
ij
i jff
w
M
ji
M
ji
0
1
,,
1
µ
µµ
16
!!"
#
Each stored pattern (fundamental memory)
establishes a correlation between pairs of neurons:
neurons tend to be of the same sign or opposite sign
according to their value in the pattern.
If wij is large, this expresses an expectation that
neurons i and j are positively correlated. If it is small
(negative) this indicates a negative correlation.
will thus be large for a state x equal to a
fundamental memory (since wij will be positive if
the product xi xj > 0 and negative if xi xj < 0).
The negative of the sum will thus be small.
9
∑
ji
jiij
xxw
,
17
!!"
G
C
4. Outputting. M>
fixed
4>stable state, that is such
that x(n+1)=x(n)=2 4B'.64
3. Iteration Until Convergence. R '24 >
6'6 2'6 2 .'
' >2 .
Njnxwsignnx
N
i
ijij
, 2, ,1 )()1(
1
=
=+
∑
=
fixed
xy =
2. Initialisation. M>
probe
!?
B '.2 '?6.
>
j
4O 2#= x
probe,j
O
'24?!x
probe
, , Njxx
jprobej
1 )0(
,
==
18
!!"
GHK
I
D
D
F''6'8
D
:.# !.'
'' 4.
!J 22?
K?G .4''J
! '
72S! ''.
?'>#)L
S>#)
'..#)
19
!!"
G)G
A
==
==
==
==
==
==
==
==
?
?
L
L
.
?'
20
!!"
G)
Consider the two fundamental memories (-1 -1 -1)
( 1 1 1).
Find weights of Hopfield network
? !
#*L*S#
#
#
21
!!"
Properties of Hopfield Nets
Distributed Representations
52 '4
K442244 4 !
22
!!"
Properties of Hopfield Nets
Distributed Representations
Local Asynchronous Control
E 2 ' '? ' ''6
! ' ?'42 ? ! = 6'642
6
23
!!"
Properties of Hopfield Nets
Distributed Representations
Local Asynchronous Control
Content-addressable memory ' associative
memory
2?4 ?
?!2'6?646. 4=
'' 2 ''64'
24
!!"
Properties of Hopfield Nets
Distributed Representations
Local Asynchronous Control
Content-addressable memory ' associative
memory
Fault Tolerant
B''4 ?'64 44
2 .2 6
: all22 . ?=
'.'4222
25
!!"
@))HD
It illustrates the behavior of the discrete Hopfield
network as a content addressable memory.
n = 120 neurons (⇒n
2
- n = 12,280 weights).
The network is trained to retrieve 8 black and white
patterns (see next slide). Each pattern contains 120
pixels. The inputs of the net assume value +1 for
black pixels and -1 for white pixels.
Implemented in MATLAB, available at:
/>run demo: ‘hop_demo’.
