New approaches to automated annotation of pathology level findings in medical images

New approaches to automated annotation of
pathology-level findings in brain images
DINH THIEN ANH
Bachelor of Computing
National University of Singapore
A THESIS SUBMITTED
FOR THE DEGREE OF DOCTOR OF PHILOSOPHY
SCHOOL OF COMPUTING
NATIONAL UNIVERSITY OF SINGAPORE
2013
Acknowledgements
First and foremost, I would like to express my deepest gratitude to my thesis advisor,
Dr. Tze-Yun Leong, for her incisive guidance, encouragement, patience and immense
support through out my Ph.D career. And I have learned a lot from her. She has also
provided me with an excellent research environment that is full of freedom. Without
her help and belief, I would not have finished my dissertation.
I am very grateful to have Dr. Choie Cheio Tchoyoson Lim from National Neu-
roscience Institute as my medical advisor. Despite his extremely busy schedule, he
is always available to share with me his valuable medical knowledge and insightful
feedbacks for my research. In addition, thanks to his generous reference, I have the
honor to receive the Singapore Millennium Scholarship for my graduate study.
I am also much indebted to Dr. Tomi Silander for being such an excellent mentor
and for his inputs in my research. He has helped me to overcome so many obstacles
in my research. Together with Dr. Tze Yun Leong, he has reviewed my thesis and
provided many thoughtful suggestions, which help me to improve this thesis tremen-
dously. I cannot thank him enough for his devotion. And I have also benefited so
much from his wide knowledge and constructive advices.
I am very fortunate to have several other mentors and collaborators along the way.
I am thankful to Dr. Chew Lim Tan for his financial support which funded me as
Research Assistant through the last year of my study. I would like to thank Dr. Boon

Chuan Pang and Dr. Cheng Kiang Lee from National Neuroscience Institute for pro-
viding me the traumatic brain injury dataset. My sincere thank goes to Dr. Tianxia
i
Gong for providing me the labelled training dataset and her valuable experience in the
medical imaging field.
I wish to extend my thanks to Dr. Dinh Truong Huy Nguyen, Dr. Duc Hiep
Chu, Thang Truong Duc, Dr. Bolan Su, Quang Loc Le, Thuy Ngoc Le, Thanh Trung
Nguyen, Zhuoru Li, and many more great friends and colleagues through out the years
for their friendship, ideas, encouragement and support. Without their accompanies, I
would not have had that much fun in my life.
My heartfelt gratitude goes to my fiance Ngoc Yen for her unconditional love,
encouragement, patience, loyalty and for standing by me in both good and bad times.
She has been virtually working as hard as me on this thesis. I am completely amazed
at her willingness to proof read my writing countlessly. She is truly a gift that I am
so blessed to have. Thank you dear from the bottom of my heart and I am looking
forward to starting a family with you.
Last but not least, I am extremely grateful to my parents for their unbounded love
and sacrifice; and my elder brother and sister-in-law for their encouragement and un-
derstanding. My parents have been giving me many wonderful opportunities in life.
I am forever thankful to have such an amazing family, and there is no word that can
describe how much I love them. The past six years have been a bumpy ride for me.
And their love, care, sacrifice, support and encouragement have made it become much
easier. Thus, I owe this to my family.
ii
Table of Contents
1 Introduction 1
1.1 Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Approaches and contributions . . . . . . . . . . . . . . . . . . . . . 5
1.3 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.4 Road map of this thesis . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 The medical domains 11
2.1 Ischemic stroke . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2 Traumatic brain injury . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3 An overview of the pathology-level medical image annotation system 19
3.1 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
3.2 Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.3 Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.4 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4 Related work 27
4.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
iii
4.1.1 Generative models vs. Discriminative models . . . . . . . . . 27
4.1.2 Ensemble learning . . . . . . . . . . . . . . . . . . . . . . . 29
4.2 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2.1 Global features . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.2.2 Local features . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3 Annotating natural images . . . . . . . . . . . . . . . . . . . . . . . 33
4.3.1 Translation paradigm . . . . . . . . . . . . . . . . . . . . . . 34
4.3.2 Relevance Models . . . . . . . . . . . . . . . . . . . . . . . 35
4.3.3 Other approaches . . . . . . . . . . . . . . . . . . . . . . . . 35
4.3.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.4 Annotating medical images . . . . . . . . . . . . . . . . . . . . . . . 38
4.4.1 Organ-level annotation . . . . . . . . . . . . . . . . . . . . . 39
4.4.2 Pathology-level annotation . . . . . . . . . . . . . . . . . . . 40

4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5 A generative model based approach 43
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3 Image Processing Component . . . . . . . . . . . . . . . . . . . . . 46
5.3.1 Automated lesion segmentation . . . . . . . . . . . . . . . . 47
5.3.2 Feature Extraction . . . . . . . . . . . . . . . . . . . . . . . 48
5.4 Generative model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.5 Content-based retrieval . . . . . . . . . . . . . . . . . . . . . . . . . 56
5.6 Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
6 A discriminative-model based approach 61
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
6.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.3 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.3.1 Feature extraction component . . . . . . . . . . . . . . . . . 65
iv
6.3.2 Classification system . . . . . . . . . . . . . . . . . . . . . . 66
6.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.4.1 Without global features . . . . . . . . . . . . . . . . . . . . . 74
6.4.2 With global features . . . . . . . . . . . . . . . . . . . . . . 76
6.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
7 Unsupervised classification by combining case-based classifiers 81
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
7.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
7.2.1 System architecture . . . . . . . . . . . . . . . . . . . . . . . 84
7.2.2 Gabor feature extraction . . . . . . . . . . . . . . . . . . . . 84
7.2.3 Sparse representation-based classifier . . . . . . . . . . . . . 87

7.2.4 Ensemble of weak classifiers . . . . . . . . . . . . . . . . . . 89
7.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.3.1 Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
7.3.2 Experimental setup . . . . . . . . . . . . . . . . . . . . . . . 92
7.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
7.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
8 Automatic Traumatic Brain Injury prognosis 99
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
8.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
8.3.1 Preprocessing and feature extraction . . . . . . . . . . . . . . 103
8.3.2 Classification of CT image slices . . . . . . . . . . . . . . . . 104
8.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
8.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
9 Prototype implementation and informal evaluation 113
9.1 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
v
9.1.1 GUI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
9.1.2 Annotator . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
9.2 Informal evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
9.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
10 Conclusion 121
10.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
10.2 Proposed approaches and contributions . . . . . . . . . . . . . . . . . 122
10.2.1 The generative model based approach . . . . . . . . . . . . . 122
10.2.2 The discriminative model based approach . . . . . . . . . . . 123
10.2.3 The unsupervised classification by combining case-based clas-
sifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
10.3 Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124

vi
Summary
Medical image annotation aims to improve the e↵ectiveness and efficiency of keyword-
based image retrieval. In this work, we focus on automated pathology annotation that
tries to identify potential pathologies, abnormalities and diseases from brain images.
This is a challenging task because pathology annotation demands a deep understand-
ing of the structural and functional changes induced by diseases. Existing works in
pathological annotation often require large and fully annotated training data, reliable
segmentation, and domain knowledge for hand-crafted feature extraction and selec-
tion. Since these prerequisites are not always feasible, they reduce the level of au-
tomation, desirability, and practicality of the annotation systems.
To mitigate the requirements of annotated training data and reliable segmentation,
we propose to use probabilistic generative models, since they support the integration of
expert knowledge and e↵ectively handle the uncertainties inherent in the images and
segmentation. However, when a priori knowledge is not available, these generative
models are not able to achieve their best performance. In this case, we suggest us-
ing a discriminative model which incorporates an automated feature selection method
to tackle the problem. Specifically, sparse group lasso provides a flexible selection
mechanism that helps to handle annotation problems without relying on the domain
knowledge.
The performance of existing annotation methods heavily depends on the quality
of hand-crafted features extracted from an automatic image segmentation. To achieve
good performance, constructing the system requires a considerable amount of man-
ual work. We propose to combine an unsupervised feature extraction technique with
a case-based classification in an ensemble learning framework to improve the adapt-
ability and automation of the annotation systems. The unsupervised nature of this
non-parametric technique can significantly reduce the time and e↵ort for system cali-
bration.
To evaluate these approaches, we select two important neurological disorders - is-
chemic stroke and traumatic brain injury, as illustrative domains because imaging find-

ings of these diseases play significant roles in their diagnosis. Despite the additional
challenges due to the relaxation of the common prerequisites in existing systems, our
vii
proposed frameworks still show reasonable performance. An informal evaluation with
expert users has also demonstrated the practical promise of the proposed system.
viii
Publications from the dissertation
research work
1. Automated predication of Glasgow Outcome Scale for Traumatic Brain Injury,
Bolan Su, Thien Anh Dinh, Abhinit Kumar Ambastha, Tomi Silander, Shijian
Lu, Boon Chuan Pang, C. C. Tchoyoson Lim, Cheng Kiang Lee, Chew Lim Tan,
Tze-Yun Leong,
Proceedings of the 22nd International Conference on Pattern Recognition (ICPR
2014),
Stockholm, Sweden. August 2014. (To appear)
2. Unsupervised medical image classification by combining case-based classifiers,
Thien Anh Dinh, Tomi Silander, Bolan Su, Tianxia Gong, Boon Chuan Pang, C
C Tchoyoson Lim, Chiang Kiang Lee, Chew Lim Tan, and Tze Yun Leong,
Proceedings of the 14th World Congress on Health and Medical Informatics
(MEDINFO 2013),
Copenhagen, Denmark. August 2013.
3. An automated pathological class level annotation system for volumetric brain
images,
Thien Anh Dinh, Tomi Silander, C. C. Tchoyoson Lim, and Tze Yun Leong,
Proceedings of the American Medical Informatics Association Annual Sympo-
sium (AMIA 2012),
Chicago, USA. November 2012.
4. A generative model based approach to retrieving ischemic stroke images,
Thien Anh Dinh, Tomi Silander, C. C. Tchoyoson Lim, and Tze Yun Leong,
Proceedings of the American Medical Informatics Association Annual Sympo-

sium (AMIA 2011),
Washington D.C., USA. October 2011.
ix
x
List of Tables
1.1 Summary of the proposed approaches. The rationale of developing
di↵erent methods is to address di↵erent perspectives of the annotation
problem. The generative model proposed in Chapter 5 will be the most
powerful method if expert knowledge and prior knowledge are avail-
able for the system. This framework is capable of using weakly an-
notated training data, handling variable size input data, and outputting
many labels. However, prior knowledge might not always be available
for the inherently perceptual tasks of medical image pattern identifica-
tion. Therefore, generative model approach is not always an optimal
choice. The unsupervised method proposed in Chapter 7 aims to im-
prove the automation and reduce the amount of required manual work
for setting up the system. Although it doesnt focus on performing
better than the discriminative method proposed in Chapter 6, its ob-
jective is to achieve a reasonable result as compared to less automated
methods. Hence, we employ a generic and straightforward feature
extraction method (Gabor filter) and sparse representation-based clas-
sifier. While the proposed discriminative method in Chapter 6 yields
better performance, it involves more preprocessing and calibration ef-
forts (e.g., region based features, time-consuming feature selection
process). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3.1 Summary of the proposed approaches for automatic pathological an-
notation in brain images . . . . . . . . . . . . . . . . . . . . . . . . . 24
xi
5.1 An example of a feature vector for a single scan. Notice the missing
information for true lesions. For the training data the TOAST class of

the stroke is also known . . . . . . . . . . . . . . . . . . . . . . . . . 50
5.2 Precision and recall of the classifier for each subtype. . . . . . . . . . 58
5.3 Detail breakdown on the performance of classifiers . . . . . . . . . . 58
6.1 A volumetric CT brain scan with 19 slices . . . . . . . . . . . . . . . 64
6.2 Preprocessing and segmentation process. (a) original image, (b) im-
age after skull removal (c) image after normalization process, (d) seg-
mented region . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
6.3 Extracted features for each potential region . . . . . . . . . . . . . . 68
6.4 Description of the global features . . . . . . . . . . . . . . . . . . . . 68
6.5 Precision and recall of classifiers using di↵erent feature selection tech-
niques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
6.6 Precision and recall of classifier for di↵erent  and 
G
75
6.7 Confusion matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.8 Average precision and recall of classifiers and their standard devia-
tions after integrating global features . . . . . . . . . . . . . . . . . . 76
7.1 Images (a) and (c) are examples of extradural hematoma (EDH), while
image (b) features subdural hematoma (SDH) . . . . . . . . . . . . . 91
7.2 Average precision and recall for di↵erent methods. The standard de-
viations over several foldings listed in parenthesis. . . . . . . . . . . . 94
7.3 Average precision and recall of classifiers when varying the ensemble
size and fixing the number of features at 1000. . . . . . . . . . . . . . 95
7.4 Average precisions and recalls of classifiers when varying number of
features and fixing the number of classifiers at 50. . . . . . . . . . . . 95
7.5 Comparison of the proposed framework and Gong et al. [38] . . . . . 96
8.1 Glasgow outcome scale (GOS) . . . . . . . . . . . . . . . . . . . . . 101
8.2 GOS prediction accuracy of di↵erent methods . . . . . . . . . . . . . 109
xii
List of Figures

1-1 An example of an ischemic stroke dataset . . . . . . . . . . . . . . . 3
2-1 A CT image with ICH. . . . . . . . . . . . . . . . . . . . . . . . . . 14
2-2 A CT image with EDH. . . . . . . . . . . . . . . . . . . . . . . . . . 15
2-3 A CT image with SDH. . . . . . . . . . . . . . . . . . . . . . . . . . 15
3-1 An overview of the pathology-class image annotation system . . . . . 20
5-1 The population of ischemic stroke dataset. . . . . . . . . . . . . . . . 45
5-2 System overview - These three components correspond to the feature
extraction component, modelling component and annotation compo-
nent in the general image annotation framework described in Chapter
3 46
5-3 Segmentations of ischemic lesion. . . . . . . . . . . . . . . . . . . . 49
5-4 A single lesion tracked through multiple slices of a DWI scan. . . . . 49
5-5 The generative model describes the joint probabilities of scans S (see
Table 5.1 for its structure) and tags (T
1
, T
2
, ,T
k
). The random vari-
able O denotes the TOAST subclass of the scan and it may take one
of the values from the set {LAA, SVO, CE, non-stroke}. C is a binary
vector of size N denoting which lesions are real. . . . . . . . . . . . . 51
5-6 KDE for P(size
i
| c
i
, O) for di↵erent subtypes LAA, SVO, CE. . . . . 54
5-7 KDE for P(z
i

| c
i
, O) for di↵erent subtypes LAA, SVO, CE. . . . . . . 54
5-8 KDE of P(x
i
, y
i
| O, c
i
, z
i
) with z
i
= {12, 6, 20} after EM for di↵erent
subtypes LAA, SVO, CE respectively . . . . . . . . . . . . . . . . . 54
5-9 Retrieval component . . . . . . . . . . . . . . . . . . . . . . . . . . 56
xiii
6-1 The population of each TBI subtype in the experiment dataset. . . . . 63
6-2 Overview of the classification system (or the modelling and annotation
components in the general framework described in Chapter 3) . . . . 69
6-3 Mapping from a variable set of features to a uniform-length vector . . 70
7-1 System architecture: The Gabor features of preprocessed images are
randomly sampled to form input spaces for weak SRC classifiers W
1
, ,W
w
,
which are then combined to get the final class label. In this architec-
ture, the preprocessing and Gabor feature extraction correspond to the
feature extraction component in the general framework. Meanwhile,

the weak classifiers and the ensemble system are the modeling and
annotation components correspondingly. . . . . . . . . . . . . . . . . 85
7-2 Values of the sparse coefficients x recovered from Algorithm 3 of a
single test image on our training data. The number of nonzero coeffi-
cients is only 5% of the total. . . . . . . . . . . . . . . . . . . . . . . 88
7-3 Histogram of the ratios of nonzero coefficients over all coefficients of
all testing images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
7-4 The population of each TBI subtype in the experiment dataset. . . . . 91
8-1 Architecture of our proposed auto-prognosis system . . . . . . . . . . 103
8-2 The workflow of construction training data for logistic regression model107
9-1 An overview of the pathology annotation prototype system. The dashed
line indicates the preprocessing phase of the system. The connected
line indicates the annotation phase of the system. . . . . . . . . . . . 114
9-2 Five main components of the GUI: 1) Visualization of an upload im-
age, 2) Classification result, 3) Decision stratification, 4) Control panel
(Upload image, Predict, Save image and Next image) and 5) Visual-
ization of related images. . . . . . . . . . . . . . . . . . . . . . . . . 115
9-3 Examples of the system when predictions are made for uploaded images118
xiv
Chapter 1
Introduction
The importance of medical imaging in patient care has grown significantly in recent
years, especially in the diagnosis and treatment processes. Advances in digital imag-
ing technologies have led to a huge number of digital images being generated every
day. There is an increasing need for efficiently accessing and retrieving relevant im-
ages for teaching, research and diagnosis [111, 91, 56, 36]. In a modern radiology de-
partment, retrieval systems have become even more vital as evidence-based medicine
is widely adopted [88].
By indexing the correspondence of keywords and images, traditional text-based
image retrieval (TBIR) systems can be used to solve the image search problem. TBIR

systems, however, require laborious manual annotation of the images with the corre-
sponding keywords. Content-based retrieval (CBIR) systems, on the other hand, use
a query image to search for images with similar low-level visual features. However,
due to their reliance on the low-level features, CBIR systems are not suitable for an-
swering abstract or high-level queries. Automatic annotation technology combines
the advantages of both TBIR and CBIR systems by first annotating the images with
their semantic content, and then allowing the users to perform text-based search on
the image databases. Semantic labels are automatically associated to the images by
the annotation system. Generally, this association can also be considered as a classifi-
1
cation problem.
The are two major challenges in image annotation [79]. The first challenge is the
semantic gap problem, which refers to the difficulty of deriving semantic informa-
tion from low-level image features, e.g., texture and color. The second challenge is
the lack of correspondence between the image regions and annotation keywords in
the training data. Although many automatic image annotation techniques have been
proposed, most of the e↵orts are targeted at annotating natural images. Since medical
images and natural images (e.g., a Computed Tomography (CT) scan of a human brain
and a natural scene image) are fundamentally di↵erent in terms of their appearance,
dimensionality (e.g., 3-D data vs. 2-D data), structure and content, existing techniques
for annotating natural images are not always suitable for annotating medical images.
As a result, automated annotation techniques for medical images need to be designed
di↵erently.
There are two types of medical image annotation: organ-level annotation and
pathology-level annotation. Organ level annotation is the process of annotating gen-
eral aspects of the images such as their modality (e.g., CT) or the anatomical structures
in an image (e.g., organ identification). Alternatively, pathology-level annotation fo-
cuses on the information about the pathologies, abnormalities and diseases shown in
the image. For example, in the domain of ischemic stroke, each Magnetic Resonance
Imaging (MRI) brain scan often needs to be categorized according to its subtype.

Many past e↵orts mainly focused on organ-level annotation problems [19, 104, 24].
In recent years, pathology annotation has gained more attention [3] because it not
only enables more semantic-based queries but also empowers clinical decision sup-
port tools. Nevertheless, pathology-level annotation is a usually a more challenging
task due to the uncertainty inherent in the abnormalities, the complexity of the patho-
logical patterns exhibited in the images, and the unavailability of the training data.
2
1.1 Problem
Before we formally define the pathology-level image annotation problems and their
challenges, we will briefly describe the medical domain of ischemic stroke that serves
as a motivating illustration for tackling this class of problems.
Figure 1-1: An example of an ischemic stroke dataset
Stroke is a major cause of death and permanent disability in the world. An early
classification of the stroke subtype can improve both short-term and long-term treat-
ment and management of patients. Clinical imaging features, which are usually ac-
quired by MRI, are important factors in the classification process. Based on these
visual findings, a patient can be classified into di↵erent ischemic stroke subtypes [64].
Specifically, each MRI brain scan represents a 3-D scan of a patient’s head (Figure 1-
1). In stroke imaging, clinical imaging features are the abnormalities which often are
the hyper intensive regions in the MRI images. The objective of pathology annota-
tion in ischemic stroke is to automatically annotate a 3-D MRI brain scan with its
corresponding pathology class. For instance, when small scatter lesions are observed
in one vascular territory of a patient’s brain, the scan images can be annotated as the
3
large-artery atherosclerosis ischemic stroke subtype [57].
The following are the main challenges in pathology annotation in this domain:
1. Semantic gap: The mapping between low-level image features and high-level
image semantics is challenging in pathology annotation due to the complex-
ity of the pathology. A subtle change in the image could indicate a di↵erent
pathological class. Uncertainty and noise in the medical image feature extrac-

tion techniques would also adversely a↵ect the performance of the annotation
system.
For example, in ischemic stroke, patterns of lesions determine a pathology class
of an image. Unsupervised or supervised feature extraction methods are often
used to recognize abnormal lesions. Although unsupervised methods are more
autonomous, they are not always able to di↵erentiate the lesions from noises.
Alternatively, supervised methods (e.g., image segmentation), which are accu-
rate for extracting abnormalities, are not always reliable due to the complexity
of the images [88]. Manual calibrations are often required when new data ar-
rives.
2. Spatial relations of abnormal regions: Abnormal regions in medical images are
often found to be structured. Relationships among the findings are critical for
understanding the images. It is common that the annotation of a certain disease
or disorder does not always depend on a single finding but rather depend on a set
of findings. For example, to annotate the pathological class for ischemic stroke
images, the lesions need to be considered in groups. They can be scattered in
di↵erent slices of a brain scan.
3. Domain knowledge: Expert knowledge plays an important role in the pathology
annotation system. It can make up for the lack of training data and guide the sys-
tem to obtain better results. A natural question that arises concerns integrating
such knowledge into the system. At the same time, since it is not always easy
4
to extract and quantify domain knowledge, the requirement of prior knowledge
could become an obstacle.
4. Annotated training data: Although medical imaging data is abundant, anno-
tated data is very rare. It is labour-intensive and time-consuming to (manually)
annotate medical images. Lack of annotated data poses challenges in apply-
ing standard machine learning techniques to the annotation task. For example,
in our ischemic stroke dataset, the training data is only labelled with ischemic
stroke subtype while the exact locations of lesions are still unknown. Classifica-

tion techniques requiring a large training dataset are not practical since labelled
training data are limited as compared to other domains.
The above challenges are relevant not only to ischemic stroke but also to other
brain diseases or disorders (such as Traumatic brain injury) and in di↵erent image
modalities (such as CT, MRI or X-ray).
1.2 Approaches and contributions
The aim of this thesis is to propose solutions to the challenges of annotating medical
images, especially of the brain, at the pathology level. The annotation is guided by
abnormalities found in the image. An abnormality can be a tumor, a malignancy or a
lesion.
Many methods have been proposed for pathology annotation of brain images.
However, large sample size, fully annotated training data, reliable segmentation re-
sults, and availability of domain knowledge for hand-crafted feature extraction and
selection are often the prerequisites for these techniques. Since these demands are ex-
pensive and difficult to be fulfilled, they reduce the automation, robustness and practi-
cality of the annotation systems. In addition, most existing techniques can only handle
2-D brain images.
5
We propose three novel techniques (Table 1.1) based on the generative model, the
discriminative model and ensemble learning to address these problems. Although they
are all solving the annotation problem, each of them addresses di↵erent requirements.
No single method is optimal for all the requirements. They are summarized as follows:
• We present a generative model based framework to address the requirements
of large and annotated training data. Since it supports integration of expert’s
knowledge, the generative model is e↵ective in handling the uncertainty inher-
ent in the medical images and image segmentation. Empirical studies demon-
strate that the proposed system can achieve good performance even when using
limited training datasets.
• When prior knowledge is inadequate, generative models cannot achieve their
best performance. In order to mitigate the dependency on domain knowledge,

we introduce a discriminative model based framework. This combines the sparse
group lasso feature selection technique, which allows simultaneous group selec-
tion and feature selection, and the kernel-based discriminative method. Without
relying on prior knowledge, the proposed framework can handle a large number
of candidate features and yield promising results in the annotation task.
• The performance of existing annotation methods is significantly influenced by
the quality of hand-crafted features extracted from automatic image segmenta-
tion. Hence, system calibration is often required. Consequently, the robustness
and automation of the annotation systems are compromised. We propose to
combine an unsupervised feature extraction technique with case-based classi-
fication in an ensemble learning framework to circumvent these manual pro-
cesses. Despite additional challenges in further automating some of the system
tasks, the proposed framework still performs reasonably as compared to existing
methods.
We also extend the case-based classication framework with logistic regression
6
Method Requirements Main objective Strengths
Generative based method 1. Expert knowledge
2. Hand-crafted features
1. Small dataset
2. Complicated visual patterns
1. Weakly annotated training
data
2. The most limited training
dataset
3. Spatial relations of abnor-
mal regions
4. Integrates domain knowl-
edge
5. Multi-class labeling

Discriminative based method 1. Hand-crafted features
2. Feature selection process
1. Less domain knowledge
2. Small dataset
1. Weakly annotated training
data
2. Small training dataset
3. Provides insight on struc-
tures of data
Unsupervised method 1. Automation
2. Reduces manual work
1. Automation
2. Minimum manual calibra-
tion
Table 1.1: Summary of the proposed approaches. The rationale of developing di↵erent
methods is to address di↵erent perspectives of the annotation problem. The generative
model proposed in Chapter 5 will be the most powerful method if expert knowledge
and prior knowledge are available for the system. This framework is capable of us-
ing weakly annotated training data, handling variable size input data, and outputting
many labels. However, prior knowledge might not always be available for the inher-
ently perceptual tasks of medical image pattern identification. Therefore, generative
model approach is not always an optimal choice. The unsupervised method proposed
in Chapter 7 aims to improve the automation and reduce the amount of required man-
ual work for setting up the system. Although it doesnt focus on performing better
than the discriminative method proposed in Chapter 6, its objective is to achieve a rea-
sonable result as compared to less automated methods. Hence, we employ a generic
and straightforward feature extraction method (Gabor filter) and sparse representation-
based classifier. While the proposed discriminative method in Chapter 6 yields better
performance, it involves more preprocessing and calibration e↵orts (e.g., region based
features, time-consuming feature selection process).

to support risk stratification of based on medical images. Unlike existing ap-
proaches, which only work on 2-D images, this method can be extended to
directly handle 3-D brain images without requiring annotated training data.
In summary, our contributions are as follows
• We have carefully examined the challenges in pathology-level annotation.
• We have proposed three novel approaches to existing unaddressed challenges of
pathology-level annotation problem.
7
• Our work provides a solid step toward improving the automation and practicality
of existing pathology-level annotation systems.
• The proposed methods are evaluated into two important neurological domains,
ischemic stroke and traumatic brain injury.
1.3 Problem formulation
We will now present a more systematic formulation of the problem(s) addressed. Dis-
cussions on the proposed solutions will be based on this formulation. Suppose that we
are given a brain image I which is known to exhibit a certain disease or disorder D.
We know that disease D can have M di↵erent types. The problem is bounded to brain
images which are acquired in 2-D or 3-D format (such as MRI, X-ray or CT brain
image).
Assume that in the disease or disorder D, a di↵erent subtype t can be categorized
by di↵erent visual patterns of abnormal findings in an image. For example, an abnor-
mal finding can be a malignant tumor or a lesion. This is the core assumption which
enables our proposed system to perform properly. For simplicity, we also assume that
each image exhibits only one subtype t of abnormality. This assumption, while ap-
plicable in many domains and tasks including those that we have examined, can be
relaxed in future.
Each image I have N abnormal findings. Each image has a di↵erent number of
N(I) abnormal findings. N(I) is influenced by a subtype t. Spatial relationships among
these abnormalities can be important factors in recognizing the disease or disorder. For
example, in ischemic stroke, depending on the locations of multiple small lesions, a

patient can be categorized to di↵erent subtype of ischemic stroke.
A feature extraction technique is applied to each image to identify K potential
findings. We use the word potential to indicate that an identified finding might not be
a “true” abnormal finding. Rather a finding can be an artifact or noise resulting from
8
image acquisition process or a false signal picked up by the feature extraction from the
image. K can be smaller or bigger than the original number N( I). For each finding,
we compute a set of real-value features representing the visual properties such as its
position, size, texture, intensity and shape.
Due to the expensive and time-consuming process of acquiring annotated training
data which indicates exactly true findings from a set of potential findings, such anno-
tated data is often not available. As a result, N(I) is unknown too. Each image I and
its corresponding disease subtype t are represented as a pair (I, t).
In summary, brain image annotation can be formulated as a classification prob-
lem with ambiguous features. Ambiguous features are mainly due to a noisy set of
abnormal findings and partially annotated training data. For example, in the task of
annotating ischemic stroke images with their ischemic stroke subtypes, each scan is
represented by a variable set of abnormal findings. Abnormal findings, which are
extracted by automated segmentation techniques, consist of both true lesions and arti-
facts. The training data only consists of subtype t for each scan I.
In the scope of this thesis, we aim to address the ambiguities arising from feature
extraction techniques and the lack of annotated training data. We also address the
general lack of domain knowledge and limited training data size.
1.4 Road map of this thesis
The following is a road map of the remaining chapters of this thesis.
In Chapter 2, we discuss the domain knowledge and related work in ischemic
stroke imaging and traumatic brain injury imaging.
In Chapter 3, we present the general framework of a pathology-level image anno-
tation system.
In Chapter 4, we survey existing techniques in annotating natural and medical

images. Before going into the details of these techniques, we will briefly present the
9