New strategies for generating panoramic images for imperfect image series 1

NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES
FOR IMPERFECT IMAGE SERIES
GAO JUNHONG
NATIONAL UNIVERSITY OF SINGAPORE
2013
NEW STRATEGIES FOR GENERATING PANORAMIC IMAGES
FOR IMPERFECT IMAGE SERIES
GAO JUNHONG
(B.Sc., Sichuan University of China, 2008)
A THESIS SUBMITTED FOR THE DEGREE OF
Doctor of Philosophy
in
SCHOOL OF COMPUTING
NATIONAL UNIVERSITY OF SINGAPORE
SINGAPORE, 2013
c
 2013, GAO Junhong
Declaration
I hereby declare that this thesis is my original work and it has
been written by me in its entirety. I have duly acknowledged all
the sources of information which have been used in the thesis.
This thesis has also not been submitted for any degree in
any university previously.
Signature: Date:

To my parents, my wife and my coming son.

Acknowledgements
First of all, I own my deepest gratitude to my advisor Dr. Michael
S. Brown for his patient guidance during my Ph.D. candidature. I
am, and will always be, thankful for all of his encouraging advice,

shared experiences and technical/financial support. All the things that
he taught me I’ll treasure for the rest of my career.
I am thankful to my co-authors: Seon Joo Kim, Tat-Jun Chin and Li Yu
for their great contribution to my research work, for all their valuable
suggestions and auxiliary works that contributed to this thesis. It is
always my pleasure to collaborate with these brilliant people and to
work on exiting research topics together.
I am grateful to my colleagues in the Computer Vision Lab: Lin Haiting,
Deng Fanbo, Cheng Dongliang, Cheng Yuan etc. , for every colorful day
I spent at the National University of Singapore (NUS). I really enjoyed
the life in Singapore with these warmhearted and reliable friends.
At last, I would like to thank my parents for their unconditional support
and encouragement. A special thank to my father, Gao Rongting, who
has contributed at least two sets of photo data to this thesis. I owe my
deep appreciation to my wife, Zhang Lingyan, for her sincere love and
all the wonderful things she has done for me. May this thesis be my
best present for her and our coming baby.

Abstract
The success of commercial image stitching tools often leads to the im-
pression that image stitching is a ”solved problem”. The reality, how-
ever, is that many tools give unconvincing results when the input data
violates fairly rigid imaging assumptions; the main two assumptions
being that the photos correspond to views that differ purely by rotation
or that the imaged scene is planar. In fact, many existing image stitching
tools depend critically onpost-processing routines to hide misalignment
artifacts. As a result, the defacto image-stitching pipeline involves two
distinct steps: 1) a geometric alignment step; and 2) a post-processing
step that computes optimal seam-cuts between overlapping images to
hide misalignment artifacts.

In this thesis, we re-examine this established panoramic image con-
struction pipeline and introduce three strategies that are useful when
the input images are less than ideal. First, we introduce a method to
compute a more flexible geometric alignment when the imaged scene
contains two dominant planes. We show that this method, termed dual-
homography alignment, is able to stitch photographs that current state-
of-the-art methods cannot. Second, we show that instead of selecting an
alignment based on the geometric fit of matched points between images,
we can use the seam-cut step to evaluate several possible geometric
transformations and select the alignment that gives the most “percep-
tually optimal” seam. We demonstrate several cases where this seam-
driven image stitching is able to produce better results than existing
methods. Lastly, for examples when the automatic image stitching pro-
cedure fails, we introduce an interactive method that provides the user
tools to manually edit the seam-cuts and/or geometric alignment in a
local manner. These three works collectively provide solutions for im-
age stitching using imperfect input images while targeting on different
parts of the image stitching pipeline.
Contents
List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Current Image Mosaicing Process . . . . . . . . . . . . . . . . . . . . . 4
1.3 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.4 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Road Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2 Background and Preliminaries 11
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Feature Registering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.2.1 SIFT Feature Matching . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 RANSAC Process . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Transformation Estimation . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Homography Model Estimation . . . . . . . . . . . . . . . . . 17
2.3.2 Rotation Model Estimation . . . . . . . . . . . . . . . . . . . . 18
2.3.3 Cylinder Mapping . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4 Post-Processing Techniques . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.1 Blending . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.2 Seam-cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Challenges for Imperfect Image Stitching and Its Related Works . . . 27
2.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
i
CONTENTS
3 Dual-Homography Image Stitching 33
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 System Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3 Dual-Homography Alignment . . . . . . . . . . . . . . . . . . . . . . 36
3.3.1 Dual-Homography Estimation . . . . . . . . . . . . . . . . . . 36
3.3.2 Extending to Multiple Images . . . . . . . . . . . . . . . . . . . 43
3.4 Post-Processing the Mosaic . . . . . . . . . . . . . . . . . . . . . . . . 45
3.4.1 Global Straightening . . . . . . . . . . . . . . . . . . . . . . . . 45
3.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4 Seam-Driven Stitching 57
4.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . . . . 57
4.2 Seam-Driven Image Stitching . . . . . . . . . . . . . . . . . . . . . . . 58
4.2.1 Generating Homography Candidates . . . . . . . . . . . . . . 60
4.2.2 Computing the Seam-Cut . . . . . . . . . . . . . . . . . . . . . 61
4.2.3 Evaluating the Cut . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.3 Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

5 Interactive Post-Processing Tools for Correction 67
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.2 System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2.1 Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
5.2.2 Local seam-editing . . . . . . . . . . . . . . . . . . . . . . . . . 70
5.2.3 Content-aware snapping . . . . . . . . . . . . . . . . . . . . . . 72
5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6 Conclusion 81
6.1 Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.2 Discussion and Limitations . . . . . . . . . . . . . . . . . . . . . . . . 84
6.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
A Intermediate Result of Seam Driven Stitching Approach 87
ii
CONTENTS
B Interface of the Interactive Panorama Correction Tool 91
C Input Images for All Examples used in This Thesis 95
Bibliography 98
iii
CONTENTS
iv
List of Figures
1.1 An example of panorama construction. . . . . . . . . . . . . . . . . . 2
1.2 An illustration of traditional photo taking assumption for construct-
ing panorama . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 An illustration of traditional image mosaicing pipeline. . . . . . . . . 5
1.4 An example result of image stitching result using traditional image
mosaicing pipeline. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 An illustration of our works with corresponding targeting part of
the traditional image mosaicing pipeline. . . . . . . . . . . . . . . . . 8

2.1 A comparison of the performance of different feature descriptors. . . 13
2.2 An example of SIFT feature detection and RANSAC filtering . . . . . 15
2.3 An example of homography warping and cylinder warping . . . . . 20
2.4 An example of post-processing techniques of image mosaicing. Multi-
band blending and seam-cut results are compared. . . . . . . . . . . . 23
2.5 An illustration of seam-cut theory. . . . . . . . . . . . . . . . . . . . . 24
2.6 Seam cut example with blending . . . . . . . . . . . . . . . . . . . . . 25
2.7 An illustration of graph cut labeling . . . . . . . . . . . . . . . . . . . 27
3.1 A scene containing two dominant planes targeted by our mosaicing
approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 Work flow of our dual-homography computation . . . . . . . . . . . 37
3.3 A synthesized example of an ideal scene of two plains and its cor-
respondent results of two homographies with explicit segmentation
method and dual-homography method. . . . . . . . . . . . . . . . . . 40
v
LIST OF FIGURES
3.4 A synthesized example of a more realistic scene of two plains with
stair structures in the middle and its correspondent results of t-
wo homographies with explicit segmentation method and dual-
homography method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.5 Illustration of concatenate multiple images . . . . . . . . . . . . . . . 43
3.6 An example of content-aware straightening . . . . . . . . . . . . . . . 47
3.7 An example where the user can specify regions that should remain
intact in the straightening process. . . . . . . . . . . . . . . . . . . . . 50
3.8 A failure case of dual-homography . . . . . . . . . . . . . . . . . . . . 51
3.9 A comparison of our dual-homography stitching approach with
commercial softwares. 1 . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.10 A comparison of our dual-homography stitching approach with
commercial softwares. 2 . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.11 A comparison of our dual-homography stitching approach with

commercial softwares. 3 . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.1 An illustration of our approach and traditional image stitching ap-
proach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.2 A comparison between the traditional image stitching process and
our seam-driven process. . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.3 An illustration of randomly generated homographies with respec-
tive seam cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
4.4 An illustration of seam-cut evaluating. . . . . . . . . . . . . . . . . . . 62
4.5 A comparison of energy for different seam cut. . . . . . . . . . . . . . 63
4.6 Comparison of panoramas constructed based on a traditional image
stitching pipeline (Adobe Photoshop CS6) and our method. . . . . . 65
5.1 An example of the usage of our interactive tool. . . . . . . . . . . . . 68
5.2 Overview of our approach. . . . . . . . . . . . . . . . . . . . . . . . . 69
5.3 Illustration of seam adjustment. . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Illustration of content-aware warping. . . . . . . . . . . . . . . . . . . 72
5.5 A comparison example with other softwares . . . . . . . . . . . . . . 74
5.6 Example results generated by our interactive approach(1) . . . . . . 75
5.7 Example results generated by our interactive approach(2) . . . . . . 76
vi
LIST OF FIGURES
5.8 Example results generated by our interactive approach(3) . . . . . . 77
5.9 User-study result of comparing our panorama correction tool and
Photoshop CS5. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
5.10 An example where Photoshop (and AutoStitch and ICE) fails to
generate a panoramic image. . . . . . . . . . . . . . . . . . . . . . . . 79
A.1 Input images used in Chapter 4 . . . . . . . . . . . . . . . . . . . . . . 87
A.2 Intermediate results for image set 1. . . . . . . . . . . . . . . . . . . 88
A.3 Intermediate results for image set 2. . . . . . . . . . . . . . . . . . . . 89
B.1 The control dialog for the interactive panorama correction tools. . . . 91
B.2 An example of the display window in seam-editing mode. . . . . . . 92

B.3 An example of the display window in warp mode. . . . . . . . . . . . 93
C.1 Input image series for Figure 3.9 . . . . . . . . . . . . . . . . . . . . . 95
C.2 Input image series for Figure 3.10 . . . . . . . . . . . . . . . . . . . . . 95
C.3 Input image series for Figure 3.11 . . . . . . . . . . . . . . . . . . . . . 96
C.4 Input image series for Figure 5.1 . . . . . . . . . . . . . . . . . . . . . 96
C.5 Input image series for Figure 5.6 . . . . . . . . . . . . . . . . . . . . . 96
C.6 Input image series for Figure 5.7 . . . . . . . . . . . . . . . . . . . . . 96
C.7 Input image series for Figure 5.8 . . . . . . . . . . . . . . . . . . . . . 97
vii
LIST OF FIGURES
viii
List of Tables
2.1 An illustration of different types of 2D motions . . . . . . . . . . . . . 18
3.1 Pros and cons of the compared two homographies with segmenta-
tion method and dual-homography method. . . . . . . . . . . . . . . 42
5.1 Processing time of the examples in Figure 5.6, 5.7, 5.8 . . . . . . . . . 75
ix
LIST OF TABLES
x