Gmns-Based Tensor Decomposition.pdf

VIETNAM NATIONAL UNIVERISTY, HANOI
UNIVERSITY OF ENGINEERING AND TECHNOLOGY

LE TRUNG THANH

GMNS-BASED TENSOR DECOMPOSITION

MASTER THESIS: COMMUNICATIONS ENGINEERING

Hanoi, 11/2018


VIETNAM NATIONAL UNIVERISTY, HANOI
UNIVERSITY OF ENGINEERING AND TECHNOLOGY

LE TRUNG THANH

GMNS-BASED TENSOR DECOMPOSITION

Program: Communications Engineering
Major: Electronics and Communications Engineering
Code: 8510302.02

MASTER THESIS: COMMUNICATIONS ENGINEERING

SUPERVISOR: Assoc. Prof. NGUYEN LINH TRUNG

Hanoi – 11/2018


Authorship

“I hereby declare that the work contained in this thesis is of my own and has not been
previously submitted for a degree or diploma at this or any other higher education
institution. To the best of my knowledge and belief, the thesis contains no materials
previously or written by another person except where due reference or acknowledgement
is made”.
Signature: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

i


Supervisor’s approval
“I hereby approve that the thesis in its current form is ready for committee examination as a requirement for the Degree of Master in Electronics and Communications
Engineering at the University of Engineering and Technology”.
Signature: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ii


Acknowledgments
This thesis would not have been possible without the guidance and the help of several
individuals who contributed and extended their valuable assistance in the preparation
and completion of this study.
I am deeply thankful to my family, who have been sacrificing their whole life for
me and always supporting me throughout my education process.
I would like to express my sincere gratitude to my supervisor, Prof. Nguyen Linh
Trung who introduced me to the interesting research problem of tensor analysis that
combines multilinear algebra and signal processing. Under his guidance, I have learned
many useful things from him such as passion, patience and academic integrity. I am
lucky to have him as my supervisor. To me, he is the best supervisor who a student can
ask for. Many thanks to Dr. Nguyen Viet Dung for his support, valuable comments

on my work, as well as his professional experience in academic life. My main results in
this thesis are inspired directly from his GMNN algorithm for subspace estimation.
I am also thankful to all members of the Signals and Systems Laboratory and my
co-authors, Mr. Truong Minh Chinh, Mrs. Nguyen Thi Anh Dao, Mr. Nguyen Thanh
Trung, Dr. Nguyen Thi Hong Thinh, Dr. Le Vu Ha and Prof. Karim Abed-Meraim for
all their enthusiastic guidance and encouragement during the study and preparation
for my thesis.
Finally, I would like to express my great appreciation to all professors of the Faculty
of Electronics and Telecommunications for their kind teaching during the two years of
my study.
The work presented in this thesis is based on the research and development conducted in Signals and Systems Laboratory (SSL) at University of Engineering and
Technology within Vietnam National University, Hanoi (UET-VNU) and is funded by
Vietnam National Foundation for Science and Technology Development (NAFOSTED)
under grant number 102.02-2015.32.

iii


The work has been presented in the following publication:
[1] Le Trung Thanh, Nguyen Viet-Dung, Nguyen Linh-Trung and Karim AbedMeraim. “Three-Way Tensor Decompositions: A Generalized Minimum Noise Subspace Based Approach.” REV Journal on Electronics and Communications, vol. 8,
no. 1-2, 2018.
Publications in conjunction with my thesis but not included:
[2] Le Trung Thanh, Viet-Dung Nguyen, Nguyen Linh-Trung and Karim AbedMeraim. “Robust Subspace Tracking with Missing Data and Outliers via ADMM
”, inThe 44th International Conference on Acoustics, Speech and Signal Processing
(ICASSP), Brighton-UK, 2019. IEEE. [Submitted]
[3] Le Trung Thanh, Nguyen Thi Anh Dao, Viet-Dung Nguyen, Nguyen LinhTrung, and Karim Abed-Meraim. “Multi-channel EEG epileptic spike detection by a
new method of tensor decomposition”. IOP Journal of Neural Engineering, Oct 2018.
[under revision]
[4] Nguyen Thi Anh Dao, Le Trung Thanh, Nguyen Linh-Trung, Le Vu Ha.
“Nonne-gative Tucker Decomposition for EEG Epileptic Spike Detection”, in 2018

NAFOS-TED Conference on Information and Computer Science (NICS), Ho Chi Minh,
2018, pp.196-201. IEEE.

iv


Table of Contents

List of Figures

vii

Abbreviations

ix

Abstract

x

1 Introduction

1

1.1

Tensor Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . .

2

1.2

Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

1.3

Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

1.4

Thesis organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4

2 Preliminaries

5

2.1

Tensor Notations and Definitions . . . . . . . . . . . . . . . . . . . . .

5

2.2

PARAFAC based on Alternating Least-Squares . . . . . . . . . . . . .

7

2.3

Principal Subspace Analysis based on GMNS . . . . . . . . . . . . . . .

10

3 Proposed Modified and Randomized GMNS based PSA Algorithms 12
3.1

Modified GMNS-based Algorithm . . . . . . . . . . . . . . . . . . . . .

12

3.2

Randomized GMNS-based Algorithm . . . . . . . . . . . . . . . . . . .

15

3.3

Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . .

19

4 Proposed GMNS-based Tensor Decomposition

21

4.1

Proposed GMNS-based PARAFAC . . . . . . . . . . . . . . . . . . . .

21

4.2

Proposed GMNS-based HOSVD . . . . . . . . . . . . . . . . . . . . . .

25

5 Results and Discussions
5.1

29

GMNS-based PSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
v

29


5.2

5.3

5.1.1

Effect of the number of sources, p . . . . . . . . . . . . . . . . .

31

5.1.2

Effect of the number of DSP units, k . . . . . . . . . . . . . . .

32

5.1.3

Effect of number of sensors, n, and time observations, m . . . .

34

5.1.4

Effect of the relationship between the number of sensors, sources
and the number of DSP units . . . . . . . . . . . . . . . . . . .

35

GMNS-based PARAFAC . . . . . . . . . . . . . . . . . . . . . . . . . .

36

5.2.1

Effect of Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . .

37

5.2.2

Effect of the number of sub-tensors, k . . . . . . . . . . . . . . .

38

5.2.3

Effect of tensor rank, R

. . . . . . . . . . . . . . . . . . . . . .

39

GMNS-based HOSVD . . . . . . . . . . . . . . . . . . . . . . . . . . .

40

5.3.1

Application 1: Best low-rank tensor approximation . . . . . . .

40

5.3.2

Application 2: Tensor-based principal subspace estimation . . .

42

5.3.3

Application 3: Tensor based dimensionality reduction . . . . . .

46

6 Conclusions

47

References

47

vi


List of Figures

4.1

Higher-order singular value decomposition. . . . . . . . . . . . . . . . .

5.1

Effect of number of sources, p, on performance of PSA algorithms; n =
200, m = 500, k = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.2

. . . . . . . . . . . . . . . . . . . . . . . . .

36

Effect of noise on performance of PARAFAC algorithms; tensor size
= 50 × 50 × 60, rank R = 5. . . . . . . . . . . . . . . . . . . . . . . . .

5.9

35

Performance of the randomized GMNS algorithm on data matrices with
k.p > n, k = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.8

34

Effect of data matrix size, (n, m), on runtime of GMNS-based PSA algorithms; p = 20, k = 5. . . . . . . . . . . . . . . . . . . . . . . . . . .

5.7

33

Effect of matrix size, (m, n), on performance of PSA algorithms; p = 2,

k = 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.6

32

Effect of number of DSP units, k, on performance of PSA algorithms;
n = 240, m = 600, p = 20.

5.5

31

Performance of the proposed GMNS algorithms for PSA versus the number of DSP units k, SEP vs. SNR with n = 240, m = 600 and p = 2. . .

5.4

30

Performance of the proposed GMNS algorithms for PSA versus the number of sources p, with n = 200, m = 500 and k = 2. . . . . . . . . . . .

5.3

25

37

Effect of number of sub-tensors on performance of GMNS-based PARAFAC
algorithm; tensor rank R = 5. . . . . . . . . . . . . . . . . . . . . . . .

38

5.10 Effect of number of sub-tensors on performance of GMNS-based PARAFAC
algorithm; tensor size = 50 × 50 × 60, rank R = 5. . . . . . . . . . . . .

39

5.11 Effect of tensor rank, R, on performance of GMNS-based PARAFAC
algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

vii

40


5.12 Performance of Tucker decomposition algorithms on random tensors, X1
and X2 , associated with a core tensor G1 size of 5 × 5 × 5. . . . . . . .

42

5.13 Performance of Tucker decomposition algorithms on real tensor obtained
from Coil20 database [5]; X of size 128×128×648 associated with tensor
core G2 of size 64 × 64 × 100. . . . . . . . . . . . . . . . . . . . . . . .

43

5.14 HOSVD for PSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

44

5.15 Image compression using SVD and different Tucker decomposition algorithms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

viii

45


Abbreviations

Abbreviation

Definition

EEG

Electroencephalogram

GMNS

Generalized minimum noise subspace

MSA

Minor Subspace Analysis

SVD

Singular Value Decomposition

HOSVD

Higher-order SVD

PCA

Principal Component Analysis

PSA

Principal Subspace Analysis

PARAFAC

Parallel Factor Analysis

ix


Abstract

Tensor decomposition has recently become a popular method of multi-dimensional data
analysis in various applications. The main interest in tensor decomposition is for dimensionality reduction, approximation or subspace purposes. However, the emergence of
“big data” now gives rise to increased computational complexity for performing tensor
decomposition. In this thesis, motivated by the advantages of the generalized minimum
noise subspace (GMNS) method, recently proposed for array processing, we proposed
two algorithms for principal subspace analysis (PSA) and two algorithms for tensor decomposition using parallel factor analysis (PARAFAC) and higher-order singular value
decomposition (HOSVD). The proposed decompositions can preserve several desired
properties of PARAFAC and HOSVD while substantially reducing the computational
complexity. Performance comparisons of PSA and tensor decompositions between using our proposed methods and the state-of-the-art methods are provided via numerical
studies. Experimental results indicated that the proposed methods are of practical

values.

Index Terms: Generalized minimum noise subspace, Principal subspace analysis, Tensor decomposition, Parallel factor analysis, Tucker decomposition, High-order singular
value decomposition.

x


Chapter 1
Introduction

Over the last two decades, the number of large-scale datasets have been increasingly
collected in various fields and can be smartly mined to discover new valuable information, helping us to obtain deeper understanding of the hidden values [6]. Many
examples are seen in physical, biological, social, health and engineering science applications, wherein large-scale multi-dimensional, multi-relational and multi-model data
are generated. Therefore, data analysis techniques using tensor decomposition now
attract a great deal of attention from researchers and engineers.
A tensor is a multi-dimensional array and often considered as a generalization
of a matrix. As a result, tensor representation gives a natural description of multidimensional data and hence tensor decomposition becomes a useful tool to analyze
high-dimensional data. Moreover, tensor decomposition brings new opportunities for
uncovering hidden and new values in the data. As a result, tensor decomposition has
been used in various applications. For example, in neuroscience, brain signals are inherently multi-way data in general, and spatio-temporal in particular, due to the fact that
they can be monitored through different brain regions at different times. In particular,
an electroencephalography (EEG) dataset can be represented by a three-way tensor
with three dimensions of time, frequency and electrode, or even by multi-way tensors
when extra dimensions such as condition, subject and group are also considered. Tensor decomposition can be used to detect abnormal brain activities such as epileptic

1


seizures [7], to extract features of Alzheimer’s disease [8] or other EEG applications,

as reviewed in [9].

1.1

Tensor Decompositions

Two widely used decompositions for tensors are parallel factor analysis (PARAFAC)
(also referred to as canonical polyadic decomposition) and Tucker decomposition. PARAFAC
decomposes a given tensor into a sum of rank-1 tensors. Tucker decomposition decomposes a given tensor into a core tensor associated with a set of matrices (called factors)
which are used to multiply along each mode (way to model a tensor along a particular
dimension).
In the literature of tensors, many algorithms have been proposed for tensor decomposition. We can categorize them into three main approaches, respectively based on
divide-and-conquer, compression, and optimization. The first approach aims to divide
a given tensor into a finite number of sub-tensors, then estimate factors of the subtensors and finally combine them together into true factors. The central idea behind
the second approach is to reduce the size of a given tensor until it becomes manageable
before computing a specific decomposition of the compressed tensor, which retains the
main information of the original tensor. In the third approach, tensor decomposition is
cast into optimization and is then solved using standard optimization tools. We refer
the reader to surveys in [10–12] for further details on the different approaches.

2


1.2

Objectives

In this thesis, we focus on the divide-and-conquer approach for PARAFAC and highorder singular value decomposition (HOSVD) of three-way tensors. HOSVD is a specific orthogonal form of Tucker decomposition. Examples of three-way tensors are numerous. (Image-row × image-column × time) tensors are used in video surveillance, human action recognition and real-time tracking [13–15]. (Spatial-row × spatial-column
× wavelength) tensors are used for target detection and classification in hyperspectral image applications [16, 17]. (Origin × destination × time) tensors are used in
transportation networks to discover the spatio-temporal traffic structure [18]. (Time

× frequency × electrode) tensors are used in EEG analysis [7].
Recently, generalized minimum noise subspace (GMNS) was proposed by Nguyen et
al. in [19] as a good technique for subspace analysis. This method is highly beneficial
in practice because it not only substantially reduces the computational complexity in
finding bases for these subspaces, but also provides high estimation accuracy. Several
efficient algorithms for principal subspace analysis (PSA), minor subspace analysis
(MSA), PCA utilizing the GMNS were proposed and shown to be applicable in various
applications. This motivates us to propose in this thesis new implementations for
tensor decomposition based on GMNS.

1.3

Contributions

The main contributions of this thesis are summarized as follows. First, by expressing
the right singular vectors obtained from singular value decomposition (SVD) in terms

3


of principal subspace, we derive a modified GMNS algorithm for PSA with running time
faster than the original GMNS, while still retaining the subspace estimation accuracy.
Second, we introduce a randomized GMNS algorithm for PSA that can deal with
several matrices by performing the randomized SVD.
Third, we propose two algorithms for PARAFAC and HOSVD based on GMNS.
The algorithms are highly beneficial and easy to implement in practice, thanks to its
parallelized scheme with a low computational complexity. Several applications are
studied to illustrate the effectiveness of the proposed algorithms.

1.4

Thesis organization

The structure of the thesis is organized as follows. Chapter 2 provides some background
for our study, including two kinds of algorithms for PSA and tensor decomposition.
Chapter 3 presents modified and randomized GMNS algorithms for PSA. Chapter 4
presents the GMNS-based algorithms for PARAFAC and HOSVD. Finally, Chapter 5
show experimental results. Chapter 6 gives conclusions on the developed algorithms.

4


Chapter 2
Preliminaries

In this chapter, we describe a brief review of tensors, related mathematical operators in
multilinear algebra (e.g., tensor additions and multiplications). In addition, a divideand-conquer algorithm for PARAFAC called alternating least-square (ALS) is also
provided that is considered as fundamental of our proposed method. Moreover, it is of
interest to first explain the central idea of the method before showing how GMNS can
be used for tensor decomposition.

2.1

Tensor Notations and Definitions

Follow notations and definitions presented in [1], the mathematical symbols used in
this thesis is summarized in the Table 2.1. We use lowercase letters (e.g., a), boldface
lowercase letters (e.g., a), boldface capital letters (e.g., A) and bold calligraphic letters
(e.g., A) to denote scalars, vectors, matrices and tensors respectively. For operators
on a n-order tensor A, A(k) denotes the mode-k unfolding of A, k ≤ n. The k-mode

product of A with a matrix U is denoted by A ×k U. The Frobenius norm of A is
denoted by kAkF , meanwhile hA, Bi denotes the inner product of A and a same-sized
tensor B. Specifically, definitions of these operators on A ∈ RI1 ×I2 ···×In used in this
thesis are summarized as follows:
The mode-k unfolding A(k) of A is a matrix in vector space RIk ×(I1 ...Ik−1 Ik+1 ...In ) , in

5


Table 2.1: Mathematical Symbols
a, a, A, A

scalar, vector, matrix and tensor

AT

the transpose of A

AT

the pseudo-inverse of A

A(k)

the mode-k unfolding of A

kAkF

the Frobenius norm of A

a◦b

the outer product of a and b

A⊗B

the Kronecker product of A and B

A ×k U

the k-mode product of the tensor A with a matrix U

hA, Bi

the inner product of A and B

which each element of A(k) is defined by

A(k) (ik , i1 . . . ik−1 ik+1 . . . in ) = A(i1 , i2 , . . . , in ).

where (ık , i1 . . . ik−1 ik+1 . . . in ) denotes the row and column of the matrix A(k) .
The k-mode product of A with a matrix U ∈ Rrk ×Ik yields a new tensor B ∈
RI1 ×···×Ik−1 ×rk ×Ik+1 ···×In such that
B = A ×k U ⇔ B(k) = UA(k) .
As a result, we derive a desired property for the k-mode product as follows

A ×k U ×l V = A ×l V ×k U for k 6= l,
A ×k U ×k V = A ×k (VU).
The inner product of two n-order tensors A, B ∈ RI1 ×I2 ···×In is defined by
hA, Bi =

I1
X
i1 =1

···

In
X

A(i1 , i2 , . . . , in )B(i1 , i2 , . . . , in ).

in =1

6


The Frobenius norm of a tensor A ∈ RI1 ×I2 ···×In is defined by the inner product of
A with itself

kAkF =

p
hA, Ai.

For operators on a matrix A ∈ RI1 ×I2 , AT and AT denote the transpose and
the pseudo-inverse of A respectively. The Kronecker product of A with a matrix
B ∈ RJ1 ×J2 , denoted by A ⊗ B, yields a matrix C ∈ RI1 J1 ×I2 J2 defined by




 a1,1 B . . . a1,I2 B 


 .

.
.
.
.
.
.
C=A⊗B=
.
.
.






aI1 ,1 B . . . aI1 ,I2 B
For operators on a vector a ∈ RI1 ×1 , the outer product of a and vector b ∈ RI2 ×1 ,
denoted by a ◦ b, yields a matrix C ∈ RI1 ×I2 defined by
T





C = a ◦ b = ab = b1 a b2 a . . . bI2 a .

2.2

PARAFAC based on Alternating Least-Squares

Several divide-and-conquer based algorithms have been proposed for PARAFAC such
as [20–25]. The central idea of the approach is to divide a tensor X into k parallel subtensors Xi , then estimate the factors (loading matrices) of the sub-tensors, and then
combine them together into the factors of X . In this section, we would like to describe
the algorithm proposed by Nguyen et al. in [23], namely parallel ALS-based PARAFAC
summarized in Algorithm 1, which has motivated us to develop new algorithms in this
thesis.
7


Algorithm 1: Parallel ALS-based PARAFAC [23]

1
2
3
4
5
6
7
8
9

Input: Tensor X ∈ RI×J×K , target rank p, k DSP units

Output: Factors A ∈ RI×p , B ∈ RJ×p , C ∈ RK×p
function
Divide X into k sub-tensors X1 , X2 , . . . , Xk
Compute A1 , B1 , C1 of X1 using ALS
Compute factors of sub-tensors: // updates can be done in parallel
for i = 2 → k do
Compute Ai , Bi and Ci of Xi using ALS
Rotate Ai , Bi and Ci
Update A, B, C

// (2.4)
// (2.5)

return A, B, C

Without loss of generality, we assume that a tensor X is divided into k sub-tensors
X1 , X2 , . . . , Xk , by splitting the loading matrix C into C1 , C2 , . . . , Ck so that the corresponding matrix presentation of the sub-tensor Xi can be determined by
Xi = (Ci
A)BT .

(2.1)

Here, Xi is considered as a tensor composed of frontal slices of X , while Xi is to present
the sub-matrix of its matrix representation X of X .
Exploiting the fact that the two factors A and B are unique when decomposing
the sub-tensors, thanks to the uniqueness of PARAFAC (see [11, Section IV] and [12,
Section III]), gives

Xi = Ii ×1 A ×2 B ×3 Ci .

(2.2)

As a result, we here need to look for an updated rule to concatenate the matrices Ci
into the matrix C, while A and B can be obtained directly from PARAFAC of X1 .
In particular, the algorithm can be described as follows. First, by performing

8