Kiểm thử tự động cho các ứng dụng đa phương thức tương tác (Luận án tiến sĩ)

MINISTRY OF EDUCATION & TRAINING
THE UNIVERSITY OF DANANG

----------

LE THANH LONG

AUTOMATIC TESTING OF
INTERACTIVE MULTIMODAL
APPLICATIONS

ENGINEERING DOCTORAL THESIS

Da Nang, 11/2017


MINISTRY OF EDUCATION & TRAINING
THE UNIVERSITY OF DANANG

----------

LE THANH LONG

AUTOMATIC TESTING OF
INTERACTIVE MULTIMODAL
APPLICATIONS
Major:

Computer science

Code of Major:

62 48 01 01

ENGINEERING DOCTORAL THESIS
Supervisors:
1. Prof. Dr. Ioannis Parissis
2. Assoc. Prof. Dr. Nguyễn Thanh Bình

Da Nang, 11/2017


REASSURANCES
I hereby certify this thesis done by my work, under the guidance of
Prof. Dr. Ioannis Parissis and Assoc. Prof. Dr. Nguyễn Thanh Bình.
I certify that the research results presented in the thesis are true and are
not copied from any other documents. All quotations origins are recorded
clearly and completely.
PhD. Student

Lê Thanh Long

1


TABLE OF CONTENTS
TABLE OF CONTENTS .............................................................................................................. 1
INTRODUCTION .......................................................................................................................... 9
Chapter 1. INTERACTIVE MULTIMODAL APPLICATIONS ........................................ 12
1.1. Multimodality .................................................................................................................. 12
1.2. Features of Multimodal Interaction ................................................................................ 13

1.3. Multimodal Fusion .......................................................................................................... 14
1.4. Design Spaces and Theoretical Frameworks ................................................................. 14
1.4.1. The TYCOON Theoretical Framework .................................................................... 14
1.4.2. CASE Design Space ................................................................................................... 15
1.4.3. The CARE Properties ................................................................................................. 16
1.5. Intoduction to Software Testing ..................................................................................... 18
1.5.1. Model-Based Testing ................................................................................................. 18
1.5.2. Operational Profile-Based Testing ............................................................................ 19
1.5.3 Requirement-Based Testing ........................................................................................ 20
1.6. Testing Interactive Multimodal Applications ................................................................ 21
1.6.1. ICO Method ................................................................................................................ 21
1.6.2. Event B Method .......................................................................................................... 26
1.6.3. Synchronous Approach .............................................................................................. 30
1.7. Conclusion ....................................................................................................................... 34
Chapter 2. BACKGROUND OF A NEW TEST MODELING LANGUAGE ................... 35
2.1. Task Trees ........................................................................................................................ 35
2.2. The Interactive Multimodal Application Memo ............................................................ 38
2.3. Operational Profiles ......................................................................................................... 42
2.4. Probabilistic Finite State Machines ................................................................................ 44
2.5. Setting a Level of Abstraction for Testing ..................................................................... 45
2.6. Generation of Tests at Dialog Controller Level ............................................................ 46
2.7. Taking into Account Conditional Probabilities ............................................................. 49
2.8. Evaluation of the Results of Extending the CTT with Conditional Probabilities ....... 55
2.9. Conclusion ....................................................................................................................... 57

2


Chapter 3. TTT: A NEW TEST MODELING LANGUAGE FOR TESTING
INTERACTIVE MULTIMODAL APPLICATIONS ............................................................ 59

3.1. Introduction ...................................................................................................................... 59
3.2. The User Actions Traces ................................................................................................. 61
3.3. Definition of the TTT Language .................................................................................... 62
3.4. Basic Structure of a TTT Model ..................................................................................... 62
3.5. Supporting Conditional Probability Specifications for all the CTT Operators ........... 65
3.6. Storing the Traces of the User Actions .......................................................................... 73
3.7. Transformation Rules from CTT to Test Model by Using the TTT Language ........... 76
3.8. Taking into Account Multimodality ............................................................................... 77
3.8.1. Generating Tests for Multimodal Events .................................................................. 77
3.8.2. Checking the Validity of CARE Properties .............................................................. 80
3.8.2.1. Equivalence Property .............................................................................................. 81
3.8.2.2. Redundancy-Equivalence Property ........................................................................ 82
3.8.2.3. Complementarity Property...................................................................................... 84
3.9. Modeling the Interactive Multimodal Application Memo by the TTT Language ...... 86
3.10. Advantages and disadvantages of the TTT Language ............................................... 91
3.11. Conclusion ..................................................................................................................... 91
Chapter 4. TTTEST: THE SUPPORT TOOL FOR TESTING INTERACTIVE
MULTIMODAL APPLICATIONS .......................................................................................... 93
4.1. Introduction ...................................................................................................................... 93
4.2. Test Execution Environment .......................................................................................... 93
4.3. The TTTEST Tool ........................................................................................................... 94
4.4 .Translating TESTCTT Model into C Program .............................................................. 95
4.4.1. Translation Problems.................................................................................................. 95
4.4.2. Automatic Translation Solution................................................................................. 98
4.5. Experimentation ............................................................................................................. 100
4.5.1. Modeling the NotePad Application by the TTT Language ................................... 100
4.5.2. Testing the Memo Application ................................................................................ 103
4.5.3. Testing the Map Navigator Application.................................................................. 105
4.6. Evaluation of the Resulted Test Cases ......................................................................... 112
CONCLUSIONS AND FUTURE WORKS ........................................................................... 116

3


PUBLICATIONS ....................................................................................................................... 118
REFERENCES ........................................................................................................................... 119

4


ACRONYMS
No.

Acronyms

Meaning

1

API

Application Programming Interface

2

AUT

Application Under Test

3

CARE

Complementarity, Assignment, Redundancy and Equivalence

4

CASE

Concurrent, Alternate, Synergistic and Exclusive

5

CTT

ConcurTaskTrees

6

DFA

Deterministic Finite State Automaton

7

FSMs

Finite State Machines

8

HMD

Head Mounted Display

9

IMA

Interactive Multimodal Application

10

ICO

Interactive Cooperative Objects

11

MBT

Model-Based Testing

12

NFA

Nondeterministic Finite State Automaton

13

ObCS

Object Control Structure

14

OPBT

Operational Profile-Based Testing

15

PFSM

Probabilities Finite State Machine

16

RBT

Requirement-Based Testing

17

SQL

Structured Query Language

18

TW

Temporal Window

19

UML

Unified Modeling Language

20

TTT

Task Tree – based Test

21

TTTEST

Testing IMA by means of the TTT language

5


LIST OF FIGURES
Figure 1.1. The TYCOON Theoretical Framework for studying multimodality. .... 15
Figure 1.2. The CASE design space.......................................................................... 16
Figure 1.3. A window of Tuple editor. ..................................................................... 21
Figure 1.4. The class Editor. ..................................................................................... 23

Figure 1.5. The ObCS of the class Editor. ................................................................ 24
Figure 2.1. The interactive multimodal application "Memo". .................................. 38
Figure 2.2. The Memo application struture [4] ......................................................... 40
Figure 2.3. The CTT for the Memo application. ....................................................... 42
Figure 2.4. The CTT with unconditional probabilities for the Memo application. .. 43
Figure 2.5. A multimodal application organized along the PAC-Amodeus model . 45
Figure 2.6. FSM Example for the Memo application [20]........................................ 48
Figure 2.7. The behavior of the choice operator. ...................................................... 51
Figure 2.8. The behavior of the concurrency operator. ............................................ 51
Figure 2.9. The behavior of the deactivation operator. ............................................. 52
Figure 2.10. The behavior of the option operator. .................................................... 52
Figure 2.11. The behavior of the suspend-resume operator. ..................................... 53
Figure 2.12. The extended CTT with conditional probabilities for the Memo
application. ................................................................................................................ 55
Figure 4.1. The TTTEST Testing Environment. ....................................................... 93
Figure 4.2. The TTTEST tool interface. ................................................................... 94
Figure 4.3. Transformation diagrams from TESTCTT model into C program. ....... 98
Figure 4.4. Translating TESTCTT to C program with Lex/Yacc. ............................ 99
Figure 4.5. The extended CTT for application NotePad. ........................................101
Figure 4.6. Multimodal interaction with a map. .....................................................105
Figure 4.7. The extended CTT for the Map navigator application. ........................ 106

6


LIST OF TABLES
Table 2.1. The CTT Operators [27] .......................................................................... 36
Table 2.2. Test data generated with unconditional probabilities and conditional
probabilities ............................................................................................................... 57
Table 2.3. Extensions of CTT operators with conditional probabilities ................... 57

Table 3.1. The TTT Syntax ....................................................................................... 62
Table 3.2. A basic structure of a TESTCTT ............................................................. 63
Table 3.3. A structure of a function .......................................................................... 65
Table 3.4. The CTT Syntax ....................................................................................... 65
Table 3.5. The behavior of choice operator .............................................................. 67
Table 3.6. The behavior of Concurrency operator .................................................... 68
Table 3.7. The behavior of Deactivation operator .................................................... 70
Table 3.8. The behavior of Suspend-resume operator .............................................. 71
Table 3.9. The behavior of Option operator.............................................................. 72
Table 3.10. The SQL-like syntax .............................................................................. 74
Table 3.11. The conditional constructs syntax .......................................................... 74
Table 3.12. Transformation rules from augmented CTT to Test Model .................. 76
Table 3.13. The semantics of modalities operator .................................................... 77
Table 3.14. Events are generated by Modalities operator ........................................ 80
Table 3.15. The behavior of TestEquivalence operator ........................................... 81
Table 3.16. The result of TestEquivalence operator ................................................. 82
Table 3.17. The behavior of TestRedundant_EquivalenceEarly operator ................ 83
Table 3.18. The result of TestRedundant_EquivalenceEarly operator ..................... 84
Table 3.19. The behavior of TestcomplementaryEarly operator .............................. 85
Table 3.20. The result of TestComplementaryEarly operator ................................... 86
Table 3.21. TESTCTT model for the Memo application .......................................... 87
Table 4.1. Lexical substitutions ................................................................................ 95
Table 4.2. Syntactic transformations ......................................................................... 96
Table 4.3. Transformation of choice operator........................................................... 97
7


Table 4.4. Transformations from create table statement in the TTT language into the
C language ................................................................................................................. 97
Table 4.5. High-level NotePad model .....................................................................102

Table 4.6. The result of TestEquivalence operator .................................................103
Table 4.7. The result of TestRedundantEquivalenceEarly operator ....................... 103
Table 4.8. The result of TestComplementaryEarly operator ...................................104
Table 4.9. The TESTCTT model for the Map Navigator application ....................107
Table 4.10. Multimodal Events are generated for the Map Navigator ...................112
Table 4.11. Results of the Experiment 1 .................................................................113
Table 4.12. Results of the Experiment 2 .................................................................113
Table 4.13. Results of the Experiment 3 .................................................................113

8


INTRODUCTION
1. Context and Motivation
Interactive Multimodal Applications (IMAs) support communication with
the user through different modalities such as voice and gesture. They can greatly
improve human-computer interaction, because they can be more intuitive, natural,
efficient, and robust. Multimodality brings an intuitive, natural affinity between the
machine and the user, such as in virtual reality mobile application. Efficiency is
obtained when the user can use equivalent modalities for the same tasks while
robustness can result from the integration of redundant or complementary
inputs [35].
The CARE properties (Complementarity, Assignment, Redundancy and
Equivalence) can be used as a measure to assess the usability of the multimodal
interaction. Equivalence and assignment represent the availability and, respectively,
the absence of choice between multiple modalities for performing a task while
complementarity and redundancy express relationships between modalities. The
flexibility and robustness of interactive multimodal applications result in an
increasing complexity of the design, development and testing. Therefore, ensuring
their correctness requires thorough validation [34].

Approaches based on formal specifications automating the development and
the validation activities have been proposed to deal with this complexity. In [21],
Laya Madani et al. present a technique of test case generation for testing CARE
properties by means of a synchronous approach. According to the proposed
approach, CARE properties are translated into an enhanced version of the Lustre
synchronous language. An improved method presented in [22] uses task trees and a
fusion model to perform test data generation for IMAs.
The above presented approach uses several notations, inspired fom existing
modeling languages, to build test models : a model of the application behavior, a
model of the interactive tasks, operational profiles (annotations on CTT) and

9


modality specifications. The variety of notations makes the modeling process hard.
So, as an additional improvement to this previous research work, in this thesis
“Automatic Testing of Interactive Multimodal Applications”, our objective is to
define a single specification and modeling language, called TTT (Task Tree based
Test) making possible to express, in a single and consistent syntax:
 Scenarios and conditional operational profiles for IMAs,
 Test oracles,
 Expected properties of the IMAs.
We built an automatic test generation approach based on this test modeling
language. The approach allows specifying multimodal events of interactive
multimodal applications and CARE properties as well as checking the validity of
CARE properties. We also develop a tool that automates the test of such interactive
multimodal applications.
2. Main Contributions of the Thesis
The thesis has the following main contributions:
(1) On the basis of analyzing the characteristics of ConcurTaskTrees (CTT), a

well-know notation for specifying interactive application, extensions of CTT
operators with conditional probabilities are proposed in order to take into
account IMA. More precisely, the user behavior on IMA is often influenced
by conditions on the application and its environment.
(2) A new test modeling language is defined for interactive multimodal
applications based on task trees. This language defines all the CTT operators,
supports state definition, multimodality and conditional probabilities for
IMA.
(3) The transformation rules from CTT into a test model in the TTT language are
formally developed.
(4) A solution to generate test data for interactive applications is proposed.

10


(5) A specification of multimodal interactions and CARE properties is integrated
into the TTT language. The specification consists of two different issues
when testing IMA: generating tests for multimodal events and checking the
validity of the CARE properties.
(6) The TTTEST tool is developed for automating the test of such interactive
multimodal applications.

3. Structure of the Thesis
Chapter 1 presents the background of interactive multimodal applications
which includes the definition of multimodality and key features of multimodal
interaction. Testing methods for interactive multimodal applications are also
summarized in this chapter.
In chapter 2, we present, with more details, the testing approach that we
focus on and the related background consisting of task trees, finite state machines,
multimodal interaction, CARE properties, operational profiles, conditional

probabilities.
In chapter 3, we propose a test modeling language for testing interactive
multimodal applications, called TTT, which makes it possible to express scenarios
and conditional operational profiles. The transformation rules from CTT
specifications into TTT test models are also developed.
Chapter 4 presents the TTTEST tool for testing interactive multimodal
applications which includes the test execution environment, and the underlying
implementation of this tool. We also introduce two interactive multimodal
applications and describe how to test them.

11


Chapter 1. INTERACTIVE MULTIMODAL APPLICATIONS
Interactive Multimodal Applications (IMA) [35] ensure the access to various
commercial services and are increasingly involved in many critical domains such as
flight or industrial process control. Designing IMA is a complex and error-prone
activity, because of the importance of the human-computer interaction aspect. This
is especially true when these interactions use multiple modalities (voice, gesture…).
They have the potential to greatly improve human-computer interaction,
because they can be more intuitive, natural, efficient, and robust. Flexibility is
obtained when the user can use equivalent modalities for the same tasks while
robustness can result from the integration of redundant or complementary inputs. As
a result, thoroughly testing such applications is particularly important and requires
more effort than for traditional interactive applications.
In this chapter, we present the background of interactive multimodal
application which includes the definition of multimodality and key features of
multimodal interaction. Testing methods for interactive multimodal applications are
also summarized in this chapter.

1.1. Multimodality
A modality is a channel or path of communication between the human and
the computer. It is one of different senses through which the human can perceive the
output of the computer (audition, vision, touch, smell, and taste); modalities will
also cover the input devices and sensors allowing the computer to receive
information from the human such as speech, pen, touch, manual gestures, gaze and
head and body movements [7]. So, a modality is defined as the representational
application, and the physical I/O device used to convey expressions of the
representational application [34].
Multimodal interaction processes two or more combined user input modes in
a coordinated manner with multimedia application output. Multimodal interaction
aims at recognizing naturally occurring forms of human language and
12


behavior [35]. Multimodal interaction can come either from the user action or from
the environment. Multimodal data can be analyzed concurrently, as well offline or
online, in order to get richer and more robust information over context and
participants.
The advantage of multimodal interaction is increased usability. The
weaknesses of one modality are offset by the strengths of another. On a mobile
device with a small visual interface and keypad, a word may be quite difficult to
type but very easy to say. It can process two or more combined user input modes
and consider these same input modes in a non-combined, exclusive or equivalent
manner.

1.2. Features of Multimodal Interaction
Multimodal interaction can be thought as a sub branch of human-computer
interaction using a set of modalities to achieve communication between users and
machines. Features of multimodal interaction include the following [1]:

 Permitting the flexible use of input modes, including alternation and
integrated use.
 Supporting improved efficiency, especially when manipulating graphical
information.
 Supporting shorter and simpler speech utterances than a speech-only
interface.
 Giving users alternatives in their interaction techniques.
Evaluations [2] have shown that users made 36% fewer errors while using a
multimodal interface in place of a unimodal interface. Furthermore, between 95%
and 100% of all users of these evaluations confirmed their preference for the
multimodal interface over other types of human-machine interfaces. Moreover,
multimodal interfaces, and in particular pen/speech interfaces, have shown greater
expressive power and greater potential precision in visual-spatial tasks.

13


Luận án đủ ở file: Luận án full