data fusion process refinement in intrusion detection alert correlation systems

DATA FUSION PROCESS REFINEMENT IN INTRUSION DETECTION ALERT
CORRELATION SYSTEMS


A Thesis
Presented to
The Graduate Faculty of The University of Akron



In Partial Fulfillment
of the Requirements for the Degree
Master of Science





David Sheets
May, 2009
ii



DATA FUSION PROCESS REFINEMENT IN INTRUSION DETECTION ALERT
CORRELATION SYSTEMS

David Sheets


Thesis

Approved: Accepted:

_______________________________ _______________________________
Advisor Dean of the College
Dr. Xuan-Hien Thi Dang Dr. Ronald F. Levant

_______________________________ _______________________________
Faculty Reader Dean of the Graduate School
Dr. Yingcai Xiao Dr. George R. Newkome


_______________________________ _______________________________
Faculty Reader Date
Dr. Zhong-Hui Duan

_______________________________
Department Chair
Dr. Wolfgang Pelz
iii



ABSTRACT

Computer systems are getting larger in size, contain a greater variety and volume of data,

and communicate personal and confidential information, making security critical as well as
making them appealing targets for malicious activities. The need to keep these systems secure
has been approached from several different aspects, one of which is the employment of intrusion
detection systems. An evolution of the intrusion detection system occurs in alert correlation
systems, which take raw alerts from numerous sensors within a network and generate broader
situational awareness by combining the individual findings of each sensor into a bigger picture
state of the system. This study looks at improving the ability of an existing alert correlation
system to pull all the relevant pieces of an intrusion into that picture in order to further reduce the
output, enabling quicker analysis by a system administrator. Through experimentation and
analysis, the benefits of utilizing the look-ahead system have demonstrated an ability to decrease
the total number of alerts in the system, thereby reducing the work-load of system administrators
by increasing the ability of the system to reduce the overall number of alerts the administrator
must analyze.
iv



ACKNOWLEDGEMENTS

It is with deep gratitude that I thank the people who have helped make accomplishing this
thesis possible.
I would like to thank my wife Kristen, who for so long put up with me paying more
attention to the thesis and academic works than to her. Her patience and contributions were
instrumental to reaching completion.
Dr. Xuan-Hien Thi Dang, my thesis advisor, who worked with me through the arduous
task of completing a master’s degree while working full-time (and often greater than full-time).
Her advice provided the key elements that allowed the research and work combine smoothly into
this thesis.
Dr. Yingcai Xiao and Dr. Zhong-Hui Duan, who took the time out of their busy
schedules near the end of the semester to provide invaluable feedback. This allowed the paper to

take shape and to become a complete work. Their efforts and dedication have provided the
critical finishing touches.
I would also like to extend my appreciation to BGI-LLC and my professional colleagues
who allowed me the time and resources to complete this endeavor. If not for their commitment, I
would have never been able to dedicate myself to this achievement.
v



TABLE OF CONTENTS
Page
LIST OF TABLES viii
LIST OF FIGURES x
CHAPTER
I. INTRODUCTION 1
1.1 Statement of the Problem 1
1.2 Importance of the Study 2
1.3 Objectives 3
1.4 Contributions 5
1.5 Organization of the Thesis 6
II. BACKGROUND OF THE STUDY 7
2.1 Intrusion Detection 7
2.2 Data Fusion 8
2.3 Alert Correlation 10
2.4 Data Sets 11
vi

2.5 Alert Message Standardization 11
2.6 Correlation Engine Framework (STAT) 12
2.6.1 Data Fusion and Alert Correlation Concepts 12

2.6.2 Correlation Framework 14
2.6.3 AlertSTAT Framework 17
2.6.4 Limitations 20
III. ALERT CORRELATION WITH LOOK-AHEAD 25
3.1 Correlation Process 25
3.2 Dynamic Time Window 26
3.3 Process refinement through a look-ahead 27
3.3.1 Algorithm 28
3.3.2 Applying the Dynamic Time Window 30
3.3.3 Providing Early Notification 31
3.3.4 Look-Ahead System 31
3.3.5 Benefits for process refinement 32
3.4 Examples 33
3.5 Example Output 38
3.6 Metrics 41
vii

IV. RESULTS AND ANALYSIS 45
4.1 Experiments 45
4.1.1 Data Sets 45
4.1.2 Execution 47
4.2 Results 47
4.3 Analysis 60
V. SUMMARY 63
5.1 Conclusion 63
5.2 Future Work 63
5.2.1 Improvements to the look-ahead 63
5.2.2 Further process refinement 64
5.2.3 Visualization 64
5.2.4 STAT Compiler Updates 65

BIBLIOGRAPHY 66

viii


LIST OF TABLES
Table Page
2.6-1 Output from static window example in Figure 2.6-4 23
3.4-1 Output from dynamic window example in Figure 3.4-1 34
3.4-2 Time window sizes in Figure 3.4-1 35
3.4-3 Output from extended static window example in Figure 3.4-2 38
4.1-1 Test platform 45
4.1-2 Statistics of available data sets 46
4.2-1 Overall results for full data set with default 120 second time windows 48
4.2-2 Overall results for full data set with session time window of 1,200 seconds 48
4.2-3 Session results for full data set with 120 second time window 49
4.2-4 Session results for full data set with 1,200 second time window 49
4.2-5 One2Many results for full data set with 120 second session time window 50
4.2-6 One2Many results for full data set with 1,200 second session time window 50
4.2-7 Many2One results for full data set with 120 second session time window 50
4.2-8 Many2One results for full data set with 1,200 second session time window 51
4.2-9 False positive predictions on 120 second session time window 53
ix

4.2-10 False positive predicitions on 1,200 second session time window 53
4.2-11 Results for subset 1 54
4.2-12 Results for subset 2 55
4.2-13 Results for subset 3 56
4.2-14 Results for subset 4 57
4.2-15 Results for subset 5 58

4.2-16 Results for subset 6 59
4.2-17 Results for subset 7 60
x


LIST OF FIGURES
Figure Page
2.2-1 Data fusion example using alerts 9
2.6-1 Alert correlation concept associated with data fusion 13
2.6-2 The AlertSTAT system implementation 19
2.6-3 Basis for correlation examples 22
2.6-4 Correlation example with static time window 23
3.3-1 Pseudo-code for calculating the dynamic window extension 29
3.3-2 Pseudo code for the window timer calculation 30
3.3-3 Look-ahead depth and technique used 32
3.4-1 Correlation example dynamic window with look-ahead depth of 2 34
3.4-2 Correlation example with extended static windows 37
3.5-1 Example output 40
3.5-2 Example IDMEF correlation alert 41
3.6-1 Example affect of a look-ahead false positive 43
4.2-1 Stacked plot of reduction with 120 second session time window on full data 51
4.2-2 Stacked plot of reduction with 1,200 second session time window on full data 52
xi

4.2-3 Stacked plot of reduction in subset 1 54
4.2-4 Stacked plot of reduction in subset 2 55
4.2-5 Stacked plot of reduction in subset 3 56
4.2-6 Stacked plot of reduction in subset 4 57
4.2-7 Stacked plot of reduction in subset 5 58
4.2-8 Stacked plot of reduction in subset 6 59

4.2-9 Stacked plot of reduction in subset 7 60

1



CHAPTER I
INTRODUCTION

1.1 Statement of the Problem
Computer systems and networks have become a voluminous and complex web of data
upon which our daily activities have come to rely. Devices such as laptops and cell phones have
found their way onto networks through advances in wireless technology, further expanding the
range and depth of these networks. High-speed networks provide immensely greater bandwidth,
allowing large amounts of data communication across these systems. The availability of the
Internet has become ingrained as a critical component of the infrastructure in nearly all facets of
society, from work to leisure.
These reasons, as realized in the responsibilities of system administrators, have a
compounding and complicating effect on the necessity to monitor and protect computer systems
and networks. Intrusion Detection Systems (IDSs) have come to the aid of administrators in this
regard, allowing the automation of network and host analysis. IDSs monitor for events by
placing a sensor on a data stream and generate alerts when suspicious activity is detected. The
situational awareness (SA) of a sensor, i.e. the context by which it can generate alerts, is limited
to the information it can gather from the data stream being processed. This limits the big picture
aspect of the alerts that are generated, leaving the problem of correlating alerts from various
sensors in the system to the administrators.
2

A novel concept toward increasing the situational awareness of the system is by gathering
and correlating alerts from numerous sensors at a central location. This effort is complicated by

the lack of an accepted standard format in the intrusion detection community for generating
alerts. An alert correlation component can preprocess alerts from a sensor and translate the
proprietary formats into a common form consumable by the rest of the correlation framework,
reducing the impact of this issue. The complexity of correlating alerts derives from the dynamic
nature of the system, where steps in an attack can be performed in various order. Steps may also
occur from different sources or across several days. This nature forces correlation components to
keep the context of alerts for some period of time in order to recognize the completion of an
attack when it occurs. This imposes severe limitations when the complexity of alert correlation is
further reduced in a strictly sequential processing order.
1.2 Importance of the Study
This research focuses on further correlating the output in an effort to reduce the output.
This in turn provides convenience to a system administrator, primarily through the employment
of intrusion detection and data fusion centered on the need for enhancing computer security. The
amount of effort put into these areas individually is extensive and the combination of these fields
is beginning to take the spotlight as we try to glean information from the variety of data available.
Intrusion detection systems have become a valuable tool in determining when an attack
occurs, tracking forensics, and even responding to an attack in order to prevent further
compromise. The value of these systems has resulted in sensors being written for each part
within a system, such as hosts, networks, and critical applications. In today's large systems, this
results in dispersed sensors generating numerous independent alerts that represent individual
pieces of a larger picture.
3

Data fusion deals with combining data collected from a variety of sensors into a single
common picture that is easier to analyze than the disparate detections of each individual sensor.
Through the combination of data, it is possible to further assess the situation and to derive
additional information.
Data fusion is being explored by numerous entities around the world. In terms of
computer security, it is being applied to intrusion detection in the form of alert correlation. These
correlation systems have proven themselves as major advancements with great potential to

provide crucial capabilities to computer security [2]. However, the data fusion picture is
incomplete and the alert correlation and data fusion communities sometimes diverge in
methodology.
By studying the evolved data fusion models and applying them to the correlation of alerts
in an intrusion detection system, we are able to produce an outlook of the system that can be
refined and lends itself toward providing visualization of that state and improving the situational
awareness of a system administrator.
1.3 Objectives
The objective of this research is to propose and implement improvements to intrusion
detection alert correlation. Alert correlation is the process of taking alerts from numerous
intrusion detection sensors that monitor networks, hosts, and applications, and combine those
alerts into a single picture with more context than the sum of the pieces. By combining alerts
from several different sensors it is possible to detect multi-step attacks from initial login through
the host compromise, reduce false positives, and provide a concise and complete picture of the
state of the system. This concept has been explored in several existing studies.
4

The concept of alert correlation closely parallels that of data fusion, which is traditionally
applied in a tactical sense to combine data from various sensors such as radar, sonar, satellite
imagery and many other sources. In both cases, the systems have sensors with different strengths
and weaknesses that generate data regarding a specific part of a greater picture. This data is
generally useful on its own but limited in context since a single sensor can only monitor a specific
area. Through data fusion, it is possible to take the data generated by these sensors and combine
the individual context into a bigger picture. This process has the benefit that it is often possible to
determine additional information from these individual pieces by accentuating the strengths and
minimizing the weaknesses of each sensor. For this reason, the data fusion concepts should be
the basis of alert correlation research and techniques.
In an extensive research effort, researchers at the University of California, Santa Barbara
(UCSB) released an alert correlation concept that seeks to define the process. In order to prove
the concept, The Computer Security Group (formerly the Reliable Software Group) at UCSB has

developed an alert correlation system called AlertSTAT that applies the research to correlate
alerts. AlertSTAT is built upon an existing intrusion detection framework called STAT, also
available from the Computer Security Group. The STAT framework is an extensible system
capable of dynamically loading components that define its behavior and capabilities. AlertSTAT
is implemented in an essentially sequential processing order where several components process a
small window of all the alerts in the application. This means that of all the alerts in the system,
each component can only utilize a small part of that data because each component must operate in
turn.
The alert correlation concept presented is a well thought out and complete model with
extensive correlation capabilities in terms of pulling alerts together into a correlated picture.
However, studying this alert correlation concept in light of existing and mature data fusion
5

models shows that the process refinement level provided in data fusion is lacking in the alert
correlation concept. The process refinement level in data fusion provides the critical function of
utilizing information in the system to refine the picture generated by the other levels. That is, the
alert correlation concept provides accurate results but does not consider all the available
information to refine those results.
The objective of this study is to introduce the concept of process refinement into the
existing alert correlation concept in order to provide a more complete picture of the system.
1.4 Contributions
The research contributes an initial look at accounting for process refinement into the alert
correlation concept. This is done through the introduction of a look-ahead that is capable of
analyzing alerts in other parts of the system, providing a greater range of context than is currently
available. The look-ahead system utilizes this additional information to dynamically adjust its
processing windows to account for additional correlations that analyzes alerts within the system.
This capability leads to improved reduction of the overall input stream and minimizes the lag
time for an event to transition from one correlation component to the next.
The look-ahead provides advantages over the original static window by dynamically
adjusting the time windows that the correlation components use. This dynamic adjustment allows

the system to expand the extents of the time window allowing the capture of additional
correlations when it’s needed and shrink the window to increase throughput when it’s not needed.
This combination provides the benefits of reducing the data set through additional correlations
while maintaining early warning and notification of the correlations within the data stream.
Furthermore, the dynamic time window calculations save time required to manually adjust static
windows in order to reap the benefits of additional correlation since different static windows must
6

be set to achieve the optimal correlation. Both the ability to achieve further reduction through
additional correlation and generating messages regarding the possible correlations in the data are
important factors for a complete and timely view of the system state.
Ultimately these contributions are applicable to the greater picture of introducing events
that have already passed from the system by accessing a database of these alerts and supplying
them to the look-ahead algorithm. In this case, the application becomes somewhat easier because
alerts outside of the sequential processing structure can be immediately incorporated as a
correlated alert. This application would provide further reduction of the overall dataset with
benefits increasing over time. The additional system to manage previous alerts in a database and
reapply them was not implemented but the algorithms presented are directly applicable to that
functionality.
1.5 Organization of the Thesis
The remainder of the thesis is organized into four additional chapters. Chapter Two
discusses the background and information relevant to understanding the work performed in this
study. Chapter Three introduces the work performed and the contributions of that work. Chapter
Four presents the experiments, the results of those experiments and the analysis of the results.
Chapter Five provides a summary of the conclusion and follows up with potential future ideas for
advancing this area.
7

CHAPTER II
BACKGROUND OF THE STUDY

This study involves the area of computer security with a focus on intrusion detection
systems. This is further scoped to the specific area of alert correlation via Data Fusion. This
section discusses the background related to this study.
2.1 Intrusion Detection
Intrusion Detection Systems (IDS) perform a variety of roles, using a variety of methods,
but all essentially monitor system resources, via a data stream, for malicious or abnormal activity.
When this kind of activity is detected, the IDS issues an alert indicating the nature of the detected
event and information surrounding it. The component that analyzes the data stream is generally
referred to as a sensor or analyzer.
An IDS may employ either anomaly detection or misuse detection in order to determine
what constitutes an event. Misuse detection makes use of attack signatures and compares that
against the data stream it is monitoring. As such, misuse systems generally have a low rate of
false positives but require diligent signature definitions and constant updates to keep the rate of
false negatives low. On the other hand, anomaly detection utilizes historical data about the
system in order to define what constitutes normal use and then seeks deviations from this
definition. This system is considerably more dynamic, allowing it to detect previously unknown
8

attacks and thereby reducing the rate of false negatives. However, this system is susceptible to
higher false positives and requires a period of time to learn the normal behavior it should expect.
In addition to the method of intrusion detection employed, sensors are also categorized by
the domain in which they operate, the environment, and level of abstraction. Some examples of
intrusion detection domains include host, network, and application domains. Host-based sensors
run on the monitored host and analyze host data streams such as the system log. Network-based
sensors are concerned with network traffic. Application-based sensors monitor application
specific details such as an application log, communication between an application and a database
or one of several other examples.

2.2 Data Fusion
Data fusion is commonly utilized in a tactical sense to combine the data from numerous
sensors to form a complete picture. The most widely used method is the JDL, which has been
studied and revised by several other parties [11,12]. This model discusses several levels of data
fusion including Sub-Object Assessment, Object Assessment, Situation Assessment, Impact
Assessment, and Process Refinement. Figure 2.2-1 demonstrates the effect of each level on a set
of alerts. In this diagram, the individual alerts within the system are represented by the smallest
circles. As illustrated by the varying shades in (a), Raw Alerts come into the system from
different sensors in different formats. Sub-Object Assessment (b) includes pre-processing the
raw alerts to put them into a common format with a uniform identity. Object Assessment (c)
associates the individual alerts into the attacks that they represent, such as pulling all the alerts
related to a port scan together into one object. Situation Assessment (d) involves the
determination of the interaction and relations between the entities. An example of this would be a
multi-step attack, which requires multiple individual attacks to succeed. Impact Assessment (e)
involves the determination or prediction of the impact the attack has, or would have, on the
9

system including the criticality of the assets involved in the attack. Process refinement (f)
includes monitoring and refining the current assessment with additional, whether it be current or
previously determined fusion information or from supplemental databases.
b) Sub-Object Assessment
c) Object Assessment
d) Situation Assessment
f) Process Refinement
a) Raw Alerts
13
10
8
7
9

14
12
16
11
15
6
18
19
17
13
10
8
7
9
14
12
16 11
15
6
18
1917
13
10
8
7
9
14
12
16
11

15
6
18
19
17
5
3
13
10
8
7
9
14
12
16
11
15
6
18
19
17
e) Impact Assessment
10
7
9
12
11
6
17
13

14
8
16
15
18
19

Figure 2.2-1 Data fusion example using alerts

10

2.3 Alert Correlation
The concept of alert correlation is currently being shaped through the efforts of various
individuals and groups. While there is no general consensus on how alert correlation should be
performed, there are some common concepts that have come to the fore-front. For example, the
distributed nature of individual sensors requires normalization of alerts to a common format and
attack identification [1.10].
There are also several techniques utilized to perform correlation, the primary ones being
filtering, fusion, and multi-step correlation. In filtering, alerts are filtered from the system using
knowledge of the network and the attacks that are detected. One example is filtering attacks on
an Apache web-server that attempt to exploit IIS vulnerabilities. Another example is an attempt
to exploit a service that is not running on a particular host. Filtering requires that the intrusion
detection know something about the system it is monitoring as well as something about the alerts
that are detected. Fusion correlates alerts by determining the degree of similarity of the alerts to
some determine criteria. For example, two alerts detected by different sensors that are only a few
milliseconds apart, contain the same source and target information, and the same attack type
might be correlated into a single alert. The fusion component requires that certain critical
information is available and consistent. The multi-step correlation operates through state changes
that move as the pre-conditions of each state are met. An example of correlating alerts using this
method is the recon-break-in-escalate model, where an attacker tests the system for

vulnerabilities, uses a discovered vulnerability to break-in to the system, and then finds a way to
obtain access. These actions are often abbreviated as r2l (remote to local), and u2r (user to root)
representing the attackers objective in the action. In this situation, the detection of recon would
transition the component to check for break-in attempts. A subsequent break-in would transition
the component to check for escalation of privilege.
11

2.4 Data Sets
Generating quality data sets has always been a problem for intrusion detection
verification. This largely weighs on the need for the data set to not only include attacks, but for
the analyst to understand the attacks and the desired detection of these attacks. When this
information is known, it becomes easy to optimize the detection routines for the attacks in the
data set, although this may not always lead to the most accurate detection as attacks vary just as
the background traffic varies.
Another issue with collecting data sets is that most significant networks are proprietary
and the organizations that own them are not willing to release this information to the public.
2.5 Alert Message Standardization
As mentioned above, the distributed nature of individual sensors and the lack of common
structure beg for a standardized approach. The Internet Engineering Task Force (IETF) created
the Intrusion Detection Working Group (IDWG) in order to define a common standard for
exchanging intrusion detection messages. This effort builds upon preliminary work as well as the
Common Intrusion Detection Format (CIDF). This working group developed the Intrusion
Detection Message Exchange Format (IDMEF) in order to address the need for a standard
message format. However, this working group adjourned without solidifying the IDMEF into an
official RFC. It is currently available in an experimental form but has been adopted by a few
intrusion detection systems, including the STAT Framework.
In addition to IDMEF, the IDWG also proposed the Intrusion Detection Exchange
protocol that utilizes Tunnels and BEEP channels [9] to exchange messages in a secure manner.
12

2.6 Correlation Engine Framework (STAT)
There are several existing alert correlation systems [4,6,10] at various stages of
development and availability. The STAT framework was chosen for this effort due to its
availability and configurability. This correlation engine was developed by The Computer
Security Group at the University of California, Santa Barbara.
The STAT framework is a highly extensible framework that allows abstract
implementations to be loaded and executed by a core component. This requires the construction
of Providers, Extensions, Scenarios, and Responses. Providers are modules that provide raw
alerts to the STAT framework. Extensions provide the extended events utilized by the specific
scenario prototypes within the system. Scenarios define prototypes, which are abstracted rules
that model the state transitions of a successful attack in consuming, non-consuming, and
unwinding transitions. Finally, responses can be provided that allow the system to execute some
action based on a state transition. This response can be a simple notification to the user to
something more complex such as gathering forensics to support the detected alerts.
2.6.1 Data Fusion and Alert Correlation Concepts
Conceptually, the data fusion concept described above must be applied to the concept of
alert correlation, taking into account the specifics of computer security and network configuration
to obtain a higher fidelity description. In a series of papers and a book [1,2,3], UCSB has
proposed an alert correlation concept outside the specific reference of data fusion research.
However, given the degree and depth that research has gone in the field of data fusion, it is
imperative that this connection be made. In fact, the alert correlation concept proposed by UCSB
can be entirely categorized into the revised JDL data fusion model [11]. One level that was left
out of this concept is process refinement, which includes the continual refinement of the big
13

picture generated by the other levels of the data fusion. This correlation concept has been
mapped to the revised JDL data fusion levels where applicable in Figure 2.6-1.
ID Sensor
Network
Object Assessment

Impact Assessment
Situation Assessment
Sub-Object Assessment
Network
Asset
Database
Active
Probing
System Administrator
Normalization
Convert Event to
common format
Sensor Events
Sensor
Ontology
Pre-Processing
Fill in missing
information
Alert Fusion
Combine independent
detection of same
atack
Alert Verification
Determine the validity
of attack
Thread
Reconstruction
Combine alerts of
single attack on a
single host.

Attack Reconstruction
Factors in network-
based alerts into the
thread
Focus Recognition
Identify hosts that are
the target/source of
the attacks
Multi-Step Correlation
Identify attack
patterns
Impact Analysis
Determine the impact
on the system
Prioritization
Prioritizes the attacks
based on impact
Reduced Data Set

Figure 2.6-1 Alert correlation concept associated with data fusion

In the sub-object assessment level, the UCSB alert correlation concept applies the
normalization and pre-processing stages, which are concerned with ensuring a complete
definition of the input alerts provided by individual sensors available within the system. Object
assessment encompasses alert fusion, alert verification, thread reconstruction, and attack
recognition as these stages of the UCSB correlation system deal with combining the alerts that
14

pertain, for the most part, to the construct of a single stage attack, or object. Note that alert
verification might utilize information gathered from the system through actively probing the

network as well as consulting a database of the assets and sensor network present in the
monitored system. Situation assessment categorizes the focus recognition and the multi-step
correlation since these stages in the alert correlation concept deal primarily with putting together
complex attacks and determining the focus of the attack. Impact assessment handles the impact
and prioritization of the impacts within the system. This stage might consult knowledge of the
dependencies and criticality of each part of the system. The process refinement level might be
loosely qualified by the application of sensor ontology and asset databases but it is largely
missing in the UCSB correlation concept.
2.6.2 Correlation Framework
The actual STAT system contains several modules that work together to form the
correlation system. It is also of importance to mention that the STAT framework was not built
solely for alert correlation. In fact, its roots are in the general sense of intrusion detection where
it would monitor the system log or network traffic. It is for this reason that the following
descriptions discuss the general function of each component. The specific use of the components
by the AlertSTAT application is explained in section 2.6.3 below.
Each of the components of a STAT application contributes a specific role that is briefly
introduced here and described in detail in the subsequent sections. The Core provides the
coordination between each of these modules. The Extension modules define the event types that
are utilized by the other components. Providers are the components that are responsible for
bringing messages into the system by packaging relevant data into the appropriate extension types
and returning them when polled by the core. Scenarios are responsible for modeling attacks and
detecting anomalies in the event stream. A specific instance of a scenario, which monitors the