Carrier recovery for high speed coherent optical communication systems based on digital signal processing

CARRIER PHASE RECOVERY FOR HIGH
SPEED COHERENT OPTICAL
COMMUNICATION SYSTEMS BASED ON
DIGITAL SIGNAL PROCESSING
XU ZHUORAN
(B.Eng.), National University of Singapore, Singapore
A THESIS SUBMITTED FOR THE DEGREE OF
DOCTOR OF PHILOSOPHY
DEPARTMENT OF
ELECTRICAL AND COMPUTER ENGINEERING
NATIONAL UNIVERSITY OF SINGAPORE
2014
ii
DECLARATION
I hereby declare that this thesis is my original work and it has been
written by me in its entirety.
I have duly acknowledged all the sources of information which have
been used in the thesis.
This thesis has also not been submitted for any degree in any uni-
versity previously.
XU ZHUORAN
January 28, 2015
iii
iv
Acknowledgement
First of all, I would like to express my deepest gratitude and most sincere ap-
preciation to my supervisors Dr. Changyuan Yu and Prof. Pooi-Yuen Kam for
their valuable guidance and kind support throughout my Ph.D study. They en-
lightened me by sharing their knowledge and experience in research. This thesis
would not been possible without their help and encouragement.
I very appreciate for my thesis committee’s effort and time to review my

thesis.
Besides, I wish to thank my seniors Dr. Zhang Shaoliang, Dr. Zhang
Banghong and Dr. Shao Xuguang with whom I gain a lot of knowledge in
research. My thanks also go to my colleagues in NUS for providing friendly
environment and exchanging insightful ideas.
Last but not least, I would like to thank my parents and relatives who are
always there to support me. I am indebted for their love.
v
ACKNOWLEDGEMENT
vi
Contents
Acknowledgement v
Summary xi
List of Tables xv
List of Figures xxi
List of Abbreviations xxv
1 Introduction 1
1.1 Review of Coherent Optical Communication . . . . . . . . . . . 2
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Contribution of the Thesis . . . . . . . . . . . . . . . . . . . . 9
1.4 Outline of the Thesis . . . . . . . . . . . . . . . . . . . . . . . 11
2 Background 15
2.1 Advanced Modulation Formats . . . . . . . . . . . . . . . . . . 16
2.1.1 Signal Model . . . . . . . . . . . . . . . . . . . . . . . 16
2.1.2 Generation Methods . . . . . . . . . . . . . . . . . . . 17
vii
CONTENTS
2.2 Transmission Links . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.1 Linear Channel Impairments . . . . . . . . . . . . . . . 21
2.2.2 Fiber Nonlinearity . . . . . . . . . . . . . . . . . . . . 24

2.3 Coherent Receiver . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3.1 Coherent Detection . . . . . . . . . . . . . . . . . . . . 27
2.3.2 Signal-to-Noise Ratio . . . . . . . . . . . . . . . . . . 30
2.3.3 Laser Phase Noise . . . . . . . . . . . . . . . . . . . . 32
2.3.4 DSP Algorithms in Coherent Receivers . . . . . . . . . 34
2.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
3 Trellis-Based Maximum Likelihood Sequence Detection 41
3.1 The Principle of Maximum Likelihood Sequence Detection . . . 42
3.1.1 Signal Model . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.2 Conditional PDF of Carrier Phase . . . . . . . . . . . . 45
3.1.3 Decision Metric of Viterbi Algorithm . . . . . . . . . . 47
3.2 The Performance of MLSD in MPSK and M-QAM . . . . . . . 51
3.2.1 Performance of MLSD in Linear Phase Noise Channel . 51
3.2.2 Performance of MLSD in Long Haul Transmission System 64
3.3 Analysis of Phase Estimation Error . . . . . . . . . . . . . . . . 70
3.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
4 Adaptive MLSD Algorithm 77
4.1 The Principle of Adaptive MLSD . . . . . . . . . . . . . . . . . 78
4.2 Simulation Performance of Adaptive MLSD . . . . . . . . . . . 83
4.2.1 Performance of Adaptive MLSD in Linear Phase Noise
Channel . . . . . . . . . . . . . . . . . . . . . . . . . . 83
viii
CONTENTS
4.2.2 Performance of Adaptive MLSD in Long Haul System . 88
4.3 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
4.3.2 Experimental results . . . . . . . . . . . . . . . . . . . 95
4.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5 Pilot Assissted MLSD Algorithm 103
5.1 The Principle of PA MLSD . . . . . . . . . . . . . . . . . . . . 104

5.2 Simulation Study of PA MLSD . . . . . . . . . . . . . . . . . . 107
5.3 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
5.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
6 Conclusion and Future Work 119
6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
6.2.1 Joint Estimation . . . . . . . . . . . . . . . . . . . . . 121
6.2.2 Sequence Detection in OFDM . . . . . . . . . . . . . . 122
6.2.3 Multi-channel System . . . . . . . . . . . . . . . . . . 122
References 139
Publication List 141
ix
CONTENTS
x
Summary
The exponentially growing demand of data traffic motivates the research of more
spectrally efficient systems to better utilize the limited bandwidth of optical
fibers. With the recent availability of high-speed analog-to-digital converters
(ADCs), full information of electric field can be preserved, such as amplitude,
phase and polarization. Advanced modulation formats together with coherent
detection are shown to be the most promising solution to increase the data rate
without increasing required bandwidth. However, one of the challenges in co-
herent optical systems is to recover the carrier phase, which is perturbed by
laser phase noise and nonlinear phase noise. In this thesis, we will study carrier
recovery algorithms to compensate for phase noise impairments for coherent
optical systems.
Firstly, a trellis-based maximum likelihood sequence detection algorithm is
proposed and the implementable decision metric function is theoretically de-
rived. The conventional sequence detection is hard to be implemented since the
computational complexity and memory requirement of the receiver increases

exponentially with the data sequence length. Therefore, a Viterbi type scheme
is applied here to treat the sequence detection as a problem to search for the
correct path through the trellis. The survivor is chosen at each state among all
xi
SUMMARY
the incoming paths based on the decision metric function, which is the posterior
probability given the received symbols.
Compared with conventional symbol-by-symbol (SBS) algorithms, the pro-
posed MLSD has superior performance. In a channel with memory, the carrier
phase of consecutive symbols are correlated, which gives an advantage for se-
quence detection. It is shown that the laser linewidth tolerance is tremendously
increased through Monte Carlo simulations. However, the window length of the
phase reference can influence the system performance. If the window length
is chosen to be long to average out the additive noise and accumulated nonlin-
ear phase noise, the estimation accuracy of the linear phase noise, which is fast
varying, will be compromised. Optimal window length is theoretically anal-
ysed. However, the optimal performance can only be obtained when the system
parameters, such as phase noise variance, SNR and transmission distance, are
available to the coherent receiver.
To solve this problem, the adaptive MLSD algorithm is proposed to auto-
matically deliver the optimal performance by self-adjusting the effective win-
dow length. A first-order adaptive filter is introduced to calculate the phase
reference. The filter gain coefficient can be viewed as a forgetting factor and is
continuously updated based on received symbols. It is shown that the adaptive
MLSD outperforms the non-adaptive version especially for MPSK signals. For
QAM modulation format, although adaptive MLSD experiences constellation
penalty when the phase noise variance is very small, it approaches the optimum
performance when the phase noise becomes significant. A back-to-back 100-
Gbit/s polarization division multiplexing DQPSK experiment is demonstrated
using our proposed carrier recovery algorithm. Significant improvements are

xii
SUMMARY
observed compared with conventional Mth power and DA ML PE schemes.
Most carrier phase estimation algorithms suffer from cycle slips which lead
to error propagation. To solve this problem, differential encoding/decoding
(DE) techniques are usually applied, though there will be a DE-induced penalty
since one symbol decision error will lead to two symbol errors. Here we propose
a pilot-assisted (PA) MLSD algorithm to eliminate the cycle slips by inserting
a short pilot between adjacent data blocks. The phase reference is re-calculated
based on the known pilot symbols after each data block. Significant improve-
ments are observed over symbol-by-symbol PA schemes. Experiments of 10-
GBaud/s 8QAM signals are demonstrated to verify our results.
xiii
SUMMARY
xiv
List of Tables
3.1 Complexity comparison among MLSD, DA ML PE and Mth
power . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
3.2 Simulation parameters in long-haul transmission system . . . . 65
4.1 Complexity comparison among adaptive MLSD, MLSD and DA
ML PE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
5.1 Complexity comparison among MLSD, PA DAML PE and PA
MLSD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
xv
LIST OF TABLES
xvi
List of Figures
1.1 Photo of our testbed for 100G experiments. . . . . . . . . . . . 11
2.1 Block diagram of coherent optical communication system. . . . 16
2.2 The transfer curves of an MZM. . . . . . . . . . . . . . . . . . 19

2.3 The structure of I/Q modulator and QPSK constellation. . . . . . 19
2.4 Received signal constellation of QPSK signals transmitted over
25×100km fiber at launched power of 0dBm . . . . . . . . . . 26
2.5 Structure of a single-polarization coherent receiver. . . . . . . . 28
2.6 Received constellation for 16QAM signal with SNR per bit of
15dB and LLW=500kHz. . . . . . . . . . . . . . . . . . . . . . 33
2.7 Received constellation for 16QAM signal with SNR per bit of
15dB and LLW=5MHz. . . . . . . . . . . . . . . . . . . . . . . 34
2.8 Block diagram of the offline DSP blocks. CDC: chromatic dis-
persion compensation; FOC: frequency offset compensation; CPE:
carrier phase estimation. . . . . . . . . . . . . . . . . . . . . . 35
3.1 Uncoded QPSK trellis of data sequence of length K . . . . . . . 44
3.2 BER performance comparison 40-Gbit/s DQPSK signals using
DA ML PE and MLSD algorithms . . . . . . . . . . . . . . . . 52
xvii
LIST OF FIGURES
3.3 QPSK BER performance of MLSD with different sequence length
with LPN, LLW=1MHz, 50Gb/s . . . . . . . . . . . . . . . . . 55
3.4 16QAM BER performance of MLSD with different sequence
length with LPN, LLW=500kHz, 56Gb/s . . . . . . . . . . . . . 56
3.5 BER performance comparison 40-Gbit/s D8PSK signals using
DA ML PE and MLSD algorithms . . . . . . . . . . . . . . . . 57
3.6 OSNR penalty comparison of DA ML PE and MLSD with L=3
and L=5 for 50Gbit/s DQPSK signals . . . . . . . . . . . . . . 58
3.7 OSNR penalty comparison of DA ML PE and MLSD with L=3
and L=5 for 25-Gbaud/s D8PSK signals . . . . . . . . . . . . . 60
3.8 Bit-symbol mapping of 16QAM constellation . . . . . . . . . . 61
3.9 BER performance comparison of MLSD and DA ML PE for
56-Gbit/s 16QAM signals with LLW=500kHz . . . . . . . . . . 62
3.10 BER performance comparison of MLSD and DA ML PE for

56-Gbit/s 16QAM signals with LLW=1MHz . . . . . . . . . . . 63
3.11 Block diagram of a long-haul optical transmission system . . . . 65
3.12 BER performance comparison between MLSD and DAML for
50-Gbit/s DQPSK with transmission of 25×100km transmission 66
3.13 BER performance of MLSD and DAML for 50-Gbit/s DQPSK
with transmission of 25×100km transmission with different win-
dow length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.14 BER performance of MLSD and DAML for 50-Gbit/s DQPSK
with transmission of 25×100km transmission with different win-
dow length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.15 BER performance of MLSD and DAML for 50-Gbit/s DQPSK
with transmission of 25×100km transmission with different win-
dow length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.16 Phase estimation error variance vs. memory length L . . . . . . 73
xviii
LIST OF FIGURES
4.1 Filter gain of the adaptive MLSD phase reference with different
LLW . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
4.2 Filter gain of the adaptive MLSD phase reference with different
E
b
/N
0
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
4.3 BER performance comparison for 50-Gbit/s DQPSK signals with
LLW=200kHz and 5MHz . . . . . . . . . . . . . . . . . . . . . 84
4.4 BER performance comparison for 56Gb/s 16QAM signals with
LLW=500kHz . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.5 BER performance comparison for 56Gb/s 16QAM signals with
LLW=1MHz . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.6 SNR penalty for 16QAM signals using DA ML PE, MLSD and
adaptive MLSD algorithms . . . . . . . . . . . . . . . . . . . . 87
4.7 Q-factor of 50-Gbit/s DQPSK signals with transmission of 25×100km 88
4.8 Q-factor of 50-Gbit/s DQPSK signal with different number of
fiber spans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
4.9 BER performance comparison of 50-Gbit/s DQPSK signal with
transmission of 23×100km and LLW=10MHz . . . . . . . . . . 91
4.10 Experimental setup of the 100-Gbit/s PDM DQPSK system. I:
in-phase data ; Q: quadrature data; PBS: polarization bean split-
ter; PBC: polarization beam combiner; VOA: variable optical
attenuator; Ix: in-phase X-polarization; Qx: quadrature X-polarization;
Iy: in-phase Y-polarization; Qy: quadrature Y-polarization . . . 92
4.11 Detail transmitter structure of the 100G experiment. PPG: pulse
pattern generator; Mux: multiplexer; Clk: clock . . . . . . . . . 94
4.12 Received signal constellation. left: X-pol; right: Y-pol . . . . . 95
4.13 Signal constellation after CMA filters. left: X-pol; right: Y-pol . 96
4.14 Recovered signal constellation after carrier phase estimation.
left: X-pol; right: Y-pol . . . . . . . . . . . . . . . . . . . . . . 96
xix
LIST OF FIGURES
4.15 BER performance 100-Gb/s PDM DQPSK system with ECL
laser for X-Polarization . . . . . . . . . . . . . . . . . . . . . . 97
4.16 BER performance 100-Gb/s PDM DQPSK system with ECL
laser for Y-Polarization . . . . . . . . . . . . . . . . . . . . . . 98
4.17 BER performance 100-Gb/s PDM DQPSK system with DFB
laser for X-Polarization . . . . . . . . . . . . . . . . . . . . . . 99
4.18 BER performance 100-Gb/s PDM DQPSK system with DFB
laser for Y-Polarization . . . . . . . . . . . . . . . . . . . . . . 99
5.1 Effect of the phase reference error on symbol decision for QPSK
signal. DB: decision boundary . . . . . . . . . . . . . . . . . . 104

5.2 Structure of the transmitted data for (a) MLSD; (b) PA DA ML;
(c) PA MLSD.T: termination symbol. . . . . . . . . . . . . . . 105
5.3 BER vs. SNR performance comparison of 40-Gb/s 16QAM sig-
nals for PA DAML and PA MLSD . . . . . . . . . . . . . . . . 108
5.4 BER performance comparison of 40-Gb/s 16QAM signals for
PA DAML and PA MLSD with 6×100km transmission . . . . . 109
5.5 Experiment setup of a 10-Gbaud/s B2B 8QAM system . . . . . 110
5.6 Received S-8QAM constellation after re-sampling . . . . . . . . 111
5.7 Signal constellation after CMA filter . . . . . . . . . . . . . . . 111
5.8 Recovered signal constellation of S-8QAM with Left: ECL laser;
Right: DFB laser . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.9 Recovered signal constellation of R-8QAM with Left: ECL laser;
Right: DFB laser . . . . . . . . . . . . . . . . . . . . . . . . . 113
5.10 BER performance comparison between PA MLSD and Mth-
power with DE for S-8QAM signals with LLW=5MHz . . . . . 114
xx
LIST OF FIGURES
5.11 BER performance comparison between PA MLSD and Mth-
power with DE for R-8QAM signals with LLW=5MHz . . . . . 115
xxi
LIST OF FIGURES
xxii
List of Abbreviations
ADC Analog-to-Digital Converter
ASE Amplified Spontaneous Emission
ASIC Application-Specific Integrated Circuit
ASK Amplitude-Shift Keying
AWG Arbitrary Waveform Generator
AWGN AdditiveWhite Gaussian Noise
B2B Back-to-Back

BER Bit-Error Rate
BPS Blind Phase Search
BPSK Binary Phase Shift-Keying
CD Chromatic Dispersion
CDC Chromatic Dispersion Compensation
CMA Constant Modulus Algorithm
CMOS Complenentary Metal-Oxide-Semiconductor
CPE Carrier Phase Estimation
DA Decision-Aided
DAC Digital-to-Analog Converters
DCF Dispersion-Compensating Fiber
xxiii
LIST OF ABBREVIATIONS
DE Differential Encoding
DFB distributed feedback
DOF Degrees of Freedom
DPSK Differential Phase-Shift-Keying
DQPSK Differential Quadrature Phase-Shift-Keying
DSP Digital Signal Processing
ECL External Cavity Lasers
EDFA Erbium-Doped Fiber Amplifier
FEC Forward Error Control
FFT Fast-Fourier Transform
FIR Finite Impulse Response
GVD Group Velocity Dispersion
ICI Inter-Carrier Interference
IF Intermediate Frequency
IIR Infinite Impulse Response
IM/DD Intensity Modulation with Direct Detection
ISI Intersymbol Interference

LDPC Low-Density Parity Check
LLW laser linewidth
LMS Least-Mean Square
LO Local Oscillator
LPN Linear Phase Noise
ML Maximum Likelihood
MLSD Maximum Likelihood Sequence Detection
MPSK M-ary phase-shift-keying
xxiv
LIST OF ABBREVIATIONS
MZM Mach-Zehnder Modulator
NLMS Normalized Least-Mean Square
NLPN Nonlinear Phase Noise
NRZ Non-Return-to-Zero
OFDM Orthogonal Frequency-Division Multiplexing
OOK On-Off Keying
OSNR Optical Signal-to-Noise Ratio
PAPR Peak-to-Average Power Ratio
PBS polarization beam splitter
PDM Polarization Division Multiplexing
PE Phase Estimation
PLL Phase Lock Loop
PMD Polarization-Mode Dispersion
PPG Pulse Pattern Generators
QAM Quadrature-Amplitude-Modulation
QPSK Quadrature Phase-Shift-Keying
RZ Return-to-Zero
SBS Symbol-By-Symbol
SDM Space-Divisoin Multiplexing
SE Spectral Efficiency

SNR Signal-to-Noise Ratio
WDM Wavelength-Division-Multiplexing
XPM Cross-Phase Modulation
xxv