Linear algebra (2)

Review of Linear Algebra

Review of Linear Algebra
CS224W: Social and Information Network Analysis
Fall 2015
Nihit Desai, Sameep Bagadia
Based on Fall 2014 slides by David Hallac


Review of Linear Algebra

Matrices and Vectors
Matrix: A rectangular array of numbers, e.g., A ∈ Rm×n :


a11 a12 . . . a1n
 a21 a22 . . . a2n 


A= .
..
.. 
 ..
.
. 
am1 am2 . . . amn
Vector: A matrix consisting of only one column (default) or
one row, e.g., x ∈ Rn


x1

 x2 


x = . 
 .. 
xn


Review of Linear Algebra

Matrix Multiplication

If A ∈ Rm×n , B ∈ Rn×p , then their product AB ∈ Rm×p
Number of columns of A must equal number of rows of B

We can compute the product C = AB using this formula:
Cij =

n
X
k=1

Aik Bkj


Review of Linear Algebra

Matrix Multiplication
(AB)(x) = A(B(x))

Review of Linear Algebra

Properties of Matrix Multiplication

Associative: (AB)C = A(BC)
Distributive: A(B + C) = AB + AC
Non-commutative: AB 6= BA
They don’t even have to be the same size!


Review of Linear Algebra

Linear Transformation and Matrices
A linear transformation T is a function from Rn to Rm that
satisfies two properties:
1

For all x, y ∈ Rn ,
T (x + y) = T (x) + T (y)

2

For all x ∈ Rn and all a ∈ R (scalar)
T (ax) = aT (x)

Every linear transformation can be represented by a matrix.
Every matrix is a linear transformation.

Review of Linear Algebra

Transform Example
Let


M=


1 0.3
.
0 1

If we apply M to every point on the Mona Lisa, we get the
following:


Review of Linear Algebra

Matrix Transpose
Transpose: A ∈ Rm×n , then AT ∈ Rn×m : (AT )ij = Aji .
For example, if


1 2 3
A=
4 5 6
then




1 4
AT = 2 5
3 6

Properties:
(AT )T = A
(AB)T = B T AT
(A + B)T = AT + B T


Review of Linear Algebra

Matrix Inverse

If A ∈ Rn×n and invertible, then the inverse of A, denoted
A−1 is the matrix that: AA−1 = A−1 A = I
Properties:
(A−1 )−1 = A
(AB)−1 = B −1 A−1
(A−1 )T = (AT )−1


Review of Linear Algebra

Identity Matrix
Identity matrix: I = In ∈ Rn×n :
(
1 i=j,
Iij =

0 otherwise.
∀A ∈ Rm×n : AIn = Im A = A


Review of Linear Algebra

Diagonal Matrix

Diagonal matrix: D = diag(d1 , d2 , . . . , dn ):
(
di j=i,
Dij =
0 otherwise.


Review of Linear Algebra

Other Special Matrices

Symmetric matrices: A ∈ Rn×n is symmetric if A = AT .
Orthogonal matrices: U ∈ Rn×n is orthogonal if
UU T = I = U T U
Every column is orthogonal to every other column (dot
product = 0)
The inverse of an orthogonal matrix is its transpose


Review of Linear Algebra

Linear combinations and Span

Given a set of vectors S = {x1 , . . . , xn } where xi ∈ Rn , a
linear combination
of this set of vectors is an expression of
P
the form: ni=1 αi xi where αi ∈ R
Span(S) is the set of all linear combinations of the
elements of S


Review of Linear Algebra

Linear Independence and Rank

A set of vectors S =P
{x1 , . . . , xn } is linearly independent if
the following holds: ni=1 αi xi = 0 only if
α1 = α2 . . . = αn = 0
Rank: A ∈ Rm×n , then rank(A) is the maximum number of
linearly independent columns (or equivalently, rows)
Properties:
rank (A) ≤ min{m, n}
rank (A) = rank (AT )
rank (AB) ≤ min{rank (A), rank(B)}
rank (A + B) ≤ rank (A) + rank (B)


Review of Linear Algebra

Example of Linear Dependence

These three vectors are linearly dependent because they all lie
in the same plane. The matrix with these three vectors as rows
has rank 2.


Review of Linear Algebra

Eigenvalues and Eigenvectors
Given A ∈ Rn×n , λ ∈ C is an eigenvalue of A with the
corresponding eigenvector x ∈ Cn (x 6= 0) if:
Ax = λx
For example, if


2 1
A=
1 2





3
then the vector
is an eigenvector with eigenvalue 1,
−3
because


Review of Linear Algebra

Solving for Eigenvalues/Eigenvectors
Characteristic Polynomial: If Ax = λx then
(A − λI)x = 0
so (A − λI) is singular (not full rank), so
det(A − λI) = 0.
Thus the eigenvalues are exactly the n possibly complex
roots of the degree n polynomial equation det(A − λI) = 0.
This is known as the characteristic polynomial.
Once we solve for all λ’s, we can plug in to find each
corresponding eigenvector.


Review of Linear Algebra

Eigenvalue/Eigenvector Properties

Usually eigenvectors are normalized to unit length.
If A is symmetric, then all the eigenvalues are real
P
tr (A) = ni=1 λi
Q
det(A) = ni=1 λi


Review of Linear Algebra

Matrix Eigendecomposition
A ∈ Rn×n , λ1 , . . . , λn the eigenvalues, and x1 , . . . , xn the
eigenvectors. P = [x1 |x2 | . . . |xn ], D = diag(λ1 , . . . , λn ), then:

Review of Linear Algebra

Matrix Eigendecomposition

Therefore, A = PDP −1 .
In addition:
A2 = (PDP −1 )(PDP −1 ) = PD(P −1 P)DP −1 = PD 2 P −1
By induction, An = PD n P −1 .