A framework for formalization and characterization of simulation performance 5

145



Chapter 5


Conclusions



5.1 Summary

This thesis has proposed a framework for the formalization and characterization of
simulation performance. The proposed framework provides a basis for understanding
and analyzing simulation performance from the simulation problem to the simulation
implementation.

Figure 5.1 summarizes the proposed framework. We formalize simulation event
orderings that provide the theoretical foundation for simulation performance analysis
based on the partially ordered set (poset). We characterize simulation performance in
three layers: physical system, simulation model and simulator. Our analysis focuses on
time (event parallelism) and space (memory requirement) performance. Because of the
different definitions of time used at different layers, a normalization process is required
to allow time performance comparison across layers. Event parallelism depends on
event dependency and time. We have proposed a time independent measure called
strictness for quantifying the degree of event dependency of a simulation event ordering,
Chapter 5. Conclusions 146

and a relation called stricter to compare the degree of event dependency of simulation
event orderings.

Figure 5.1: Framework for Formalization and
Characterization of Simulation Performance

In summary, this thesis offers three main contributions: formalization of event orderings,
time and space performance characterization, and strictness measurement.

5.1.1 Formalization of Simulation Event Orderings

We have proposed the formalization of simulation event orderings based on poset. Poset
is a branch of discrete mathematics that studies the ordering of elements in a set. In
event ordering, we study the ordering of events in a given set of events. At the physical
system, events are ordered based on their physical time of occurrence. A number of event
orderings can be used at the simulation model layer. A simulator maintains a specific
event ordering at runtime. We have extracted and formalized the event orderings
Physical
System
Simulation
Model
Simulator
Event
Ordering
Perf
Perf
Perf
Π
prob

M
prob

Π
or
d

M
or
d

Π
sync

M
sync
, M
to
t
, M
sh
r
, M
ds
t

Normalization
Perf
Strictness (
ς
)
Stricter (<

ς
)
Comparison
Poset
Perf. = performance measure
Chapter 5. Conclusions 147

maintained by a number of simulators (sequential and parallel). This formalization
provides a theoretical foundation for carrying out performance analysis of simulation.
The existing theories in poset such as Dilworth’s chain covering theorem can be applied
to explain the performance of different simulation event orderings. The proposed
formalization can also be extended to other computer systems that involve ordering in
their operations such as memory operation orderings in the memory consistency model
[CULL99], instruction orderings in processor design [HENN03], and job orderings in
grid computing [THAI03].

5.1.2 Time and Space Performance Characterization

We have proposed characterizing simulation performance along the three natural
boundaries in simulation, i.e., physical system, simulation model and simulator
(implementation). This thesis focuses on the simulation of physical systems that can be
modeled as queuing networks. Therefore, at the physical system layer, we view a
physical system as a queuing network. At the simulation model layer, based on the
virtual time paradigm [JEFF85], we view a simulation model as a set of logical processes
(LPs). Every LP models a physical process (service center) in the physical system. The
interaction among physical processes in the physical system is modeled by exchanging
events among LPs in the simulation model. At the simulator layer, a simulator that
implements a simulation model is executed on a platform consisting of one or more
physical processors (PPs).

We have formalized event parallelism and memory requirement at each of the layers.
Event parallelism is defined as the number of events executed per unit of time. At the
Chapter 5. Conclusions 148

physical system layer, the physical time unit is used. At the simulation model layer, the
time unit is timestep. At the simulator layer, the wall-clock time unit is used. Analysis at
each layer can be done independently. To compare event parallelism at different layers,
normalization is necessary because the time units used at the three layers differ.

The memory requirement at the physical system layer is derived from the maximum
queue size of each service center in the system. At the simulation model layer, the
memory requirement is derived from the maximum event list size of each LP. At the
simulator layer, the memory requirement is derived from the maximum size of the data
structure that is used for synchronization purposes. The memory requirement at each
layer can be analyzed independently.

The total memory requirement is measured at the simulator layer. At runtime, in addition
to the data structure that is used for synchronization purposes, other data structures (such
as a linked list) are also used to implement queues and event lists. These data structures
constitute the total memory requirement of simulation.

The proposed three-layer performance characterization can be extended to other areas. In
shared-memory computer architecture, for example, a program that is going to be
executed constitutes the physical system layer, a memory consistency model that
specifies the ordering of memory operations independent of implementation constitutes
the model layer, and an implementation of the memory consistency model constitutes the
implementation layer. With this characterization, we can analyze the performance from
the physical system layer to the implementation layer. Similarly, in processor design (to
exploit instruction level parallelism), a program that is going to be executed constitutes
Chapter 5. Conclusions 149

the physical system layer, the different instruction orderings analyzed using an idealized
processor model constitutes the model layer, and an implementation of the instruction
orderings on a real processor constitutes the implementation layer.

5.1.3 Strictness of Event Orderings

We have proposed a relation called stricter and a measure called strictness to compare
and quantify the degree of event dependency of event orderings respectively. An event
order R
1
is stricter than R
2
if event x is ordered before event y in R
2
then x is also ordered
before y in R
1
but not vice versa. For any two events, they are either comparable or non-
comparable (concurrent). We have shown that if event order R
1
is stricter than R
2
, then
R
1
will never execute more events than R
2
for the same measurement duration.

The strictness of an event ordering is measured from the number of comparable event
pairs which is divided by the number of comparable event pairs in the total event
ordering. A strictness value of x means that the probability of two events being
comparable is x. If all events are comparable (non comparable) in an event ordering, then
the strictness of the event ordering is one (zero).

The relation stricter has been used in the memory consistency model [GHAR95]. It can
be extended to compare the degree of dependency of different orderings in areas such as
instruction orderings in the processor design [HENN03]. The measure strictness can
also be extended to measure the degree of dependency in an ordering such as memory
operation orderings in the memory consistency model [CULL99] and instruction
orderings in processor design [HENN03].
Chapter 5. Conclusions 150


5.2 Future Works

We highlight three possible research areas that can be further explored:
a. Hardware Implementation.
The formalization of event orderings is motivated by research in the memory
consistency model. A number of hardware implementations have been built based on
the existing memory consistency models [CULL99, GHAR95]. Learning from
research in the memory consistency model, it may be possible to explore the
hardware implementation of a simulation event ordering. This may improve
simulation performance.
b. Empirical Study on Other Protocols.
This thesis has used only a CMB protocol with demand driven optimization in the
experiment. We have formalized a number of event orders and each event order can
be implemented in many ways, resulting in different simulation protocols. Therefore,
empirical study of the various protocols will be useful in supporting the proposed

framework. Further, the proposed framework also provides the foundation for
studying future event orderings and new protocol implementations.
c. Extreme Value Analysis.
We have formalized the memory requirement based on the total of a maximum value,
such as maximum queue size at the physical system layer, maximum event list size at
the simulation model layer, and maximum buffer size at the simulator layer. The
stochastic behavior of a maximum value is the focus of research in the extreme value
theory [COLE01]. Therefore, the extreme value theory may provide some analytic
model that can explain the stochastic behavior of the memory requirement.