Master Thesis Training Effectiveness Evaluation


MBA Program
24th Intake (2015 - 2017)

PROFESSIONAL DISSERTATION

EVALUATE EFFECTIVENESS OF TEAM
LEADERS TRAINING PROGRAM
AT VNPT EPAY., JSC
Author
Tutor

: Pham Van Binh
: Ass. Pr. Dr. Vu Hoang Ngan

Hanoi, July 2017


ABSTRACT
VNPT Electronic Payment Joint-stock company (VNPT EPAY) is one of three
biggest companies in the financial technology sector in Vietnam. In order to make its
goal in 2025 happen, training for staffs especially team leaders to develop high-quality
human resources become a key strategic task. With a large approved budget for the
training program for team leaders, the company needs to know how their team
leaders’ development through training program is. In the other word, the company
needs to know the effectiveness of training program for team leaders. However, the
current method of evaluation of training programs for team leaders is carried out at the
reaction level of Kirkpatrick's model of evaluation (Kirkpatrick, 1967), giving a little
indication of the effectiveness of the training programs.
The main purpose of this study thus evaluate the effectiveness of the programs at all
level which is the reaction, learning, change in behavior, and result. From the result of

the evaluation, we propose to suggest and recommendation to continuously improve
quality of the program in the coming years. Data from 42 team leaders and surveys,
interviews are used.
The result from research shows that VNPT EPAY has developed good training
content that met stated objective; actively at creating environment and opportunities
for learners apply their new skill/ knowledge. Otherwise, many aspect need to be
improved, such as trainers do not prepare good enough for the training program, the
traditional training method is popularly used, information is out of date, training
program use only internal trainer, homework and exercises are unrelated to learners’
job.
From the result of the study, we propose solutions to improve the effectiveness of
training program for team leaders at VNPT EPAY in following years.


ACKNOWLEDGMENT
I would like to express my gratitude to Professor Vu Hoang Ngan for meaningful
instructions, comments, and encouragement throughout study process of this thesis.
My special gratitude to my family for unconditional support, both financial and
emotional during my degree. To my partner, Ms. Ban Thi Huyen, who always beside
me at all time.
I would like to thanks the participants (team leaders at VNPT EPAY) in my survey,
who have willingly shared their precious time during the process of answering the
questionnaire and participate in the interview.
Thank you my friends, who have given me the energy to overcome all obstacle during
my degree and thesis study process.
Thank you!


LIST OF TABLES
Table 2.1.Summarize the Kirkpatrick model ............................................................ 09

Table 3.1.Scale of trainee’s satisfaction .................................................................... 18
...................................................................................................................................
Table 3.2.Scale of knowledge acquisition ................................................................. 20
Table 3.3.Scale of changing behavior ....................................................................... 21
Table 3.5.Scale of working performance ................................................................. 22
Table 3.6.Research characteristic .............................................................................. 25
Table 3.7.Analysis flame ........................................................................................... 27
Table 4.1.Name and time frame of training program for team leader at VNPT EPAY
................................................................................................................................... 32
Table 4.2.Number of trainee attend training program for team leader every year ... 33
Table 4.3.Satisfaction with program objective/content ............................................ 36
Table 4.4.Satisfaction with Instructor/ facilitator ...................................................... 38
Table 4.5.Satisfaction with instructional activities .................................................. 40
Table 4.6.Satisfaction with program time/length ...................................................... 42
Table 4.7.Satisfaction with logistic/administration .................................................. 43
Table 4.8.Learners’ learning ..................................................................................... 45
Table 4.9.Learners’ behavior .................................................................................... 47
Table 4.10.Result ...................................................................................................... 49


LIST OF FIGURES
Figure 2.1.The procedures for general evaluation for training results ..................... 05
Figure 3.1.Research process ...................................................................................... 16
Figure 3.2.Group course based on different evaluation level .................................... 17
Figure 4.3.Satisfaction with program objective/content ............................................ 37
Figure 4.4.Satisfaction with Instructor/ facilitator .................................................... 39
Figure 4.5.Satisfaction with instructional activities ................................................... 41
Figure 4.6.Satisfaction with program time/length ...................................................... 43
Figure 4.7.Satisfaction with logistic/administration ................................................... 44
Figure 4.8.Learners’ learning ..................................................................................... 46

Figure 4.9.Learners’ behavior .................................................................................... 48
Figure 4.10.Result ...................................................................................................... 51


LIST OF APPENDIXES
APPENDIX 1.FOCUS GROUP WORK OUTLINE ............................................... 65
APPENDIX 2.STUDENT LIST AND FEEDBACK ............................................... 69
APPENDIX 3.QUESTIONNAIRE .......................................................................... 69


TABLE OF CONTENT
Chapter I: Introduction ............................................................................................ 1
1.1 Introduction ................................................................................................ 1
1.2 Problem statements ..................................................................................... 2
1.3 Research goal.............................................................................................. 2
1.4 Scope of the Research ................................................................................ 2
1.5 The meaning of the Research .................................................................... 3
1.6 Research structure ...................................................................................... 3
Chapter II: Literature review .................................................................................. 4
2.1 Research definition ..................................................................................... 4
2.1.1 Training in Human Resources ............................................................. 4
2.1.2 Evaluation of training programs.......................................................... 4
2.1.3 Evaluation procedure .......................................................................... 5
2.1.4 Evaluation of skill ............................................................................... 5
2.2 Evaluation Training Effectiveness Model .................................................. 7
2.2.1 Purpose of Training Assessment ........................................................ 7
2.2.2 Common Evaluation Technique ......................................................... 8
2.2.3 Kirkpartric’s model to Evaluate Training effectiveness ..................... 8
Chapter III: Research Methodology ........................................................................ 16
3.1 Research process ........................................................................................ 16

3.2 Research Method ........................................................................................ 17
3.3 Collecting information and analyzing method ........................................... 23
3.4 Research Model description ...................................................................... 25
3.5 Analysis flame ........................................................................................... 27
Chapter IV: Research Results .................................................................................. 28
4.1 Recent Training Evaluation in VNPT EPAY ............................................. 28
4.1.1General Introduction about VNPT EPAY............................................ 28
4.1.2 Training programs in VNPT EPAY .................................................... 31
4.1.3 Objective and methods of training programs ...................................... 33
4.1.4 The implementation of Training programs ......................................... 34
4.1.5 Recent Training effectiveness evaluation in VNPT EPAY ................ 35
4.2 Analysis the VNPT EPAY trainees’ satisfaction ...................................... 35
4.3 Analysis the VNPT EPAY learners’ learning ............................................ 44
4.4 Analysis the VNPT EPAY learners’ behavior ........................................... 46
4.5 Result analysis ........................................................................................... 49
4.6 Overall evaluation of the training effectiveness of training program
for team leader at VNPT EPAY ....................................................................... 51
Chapter V: Solution and recommendation ............................................................. 55
5.1 Solution....................................................................................................... 55
5.2 Recommendation ....................................................................................... 61
Conclusion .................................................................................................................. 62


CHAPTER I
INTRODUCTION
1.1 .Introduction
The economic and political reform launched in 1986 have positively effected in Vietnam
economy and transformed Vietnam from one of the world’s poorest to a lower middle –
income country (Vietnam economy report, World Bank). This change has made Vietnam
become one of the most dynamic emerging markets in South Asia region. Indeed,

according to report of General Statistic Office of Vietnam, Vietnam GDP growth average
6.5 percent every year in over past 5 years. As well as this, Vietnam GDP per capital
growth average 6.7% per year and reached 1684.87 USD in 2015.0 (Vietnam economy
report 2016).
Those factors have led Electronic Payment become an extremely attractive market in
Vietnam. The main reason is that increasing incomes will positive impact on rising of
buying force of population. Fortunately, almost consumer in Vietnam still using cash
payment, but they are changing to use electronic payment method.
In this growing market, VNPT EPAY is one of 3 biggest players and have been
experiencing business rapid growth period. Annual revenue increase by 50% resulted
from best quality services, enthusiastic employees. In addition, from 2016 UTC
investment Co., Ltd – a big Korean investment company bought 65% share to become
biggest shareholder of VNPT EPAY. That trading brought to VNPT EPAY more strength
and advantages to gain market share in not only domestic but also international market.
In this situation, the roles of Team leaders in VNPT EPAY become extremely important.
It could be because when the company growths, the number of employees in departments
increase, managers cannot manage department effectively without support of team
leaders.
Thus, VNPT EPAY business strategy has emphasized the important of team leaders
training program and its roles in the development of VNPT EPAY. And so the
Management Board has embarked a large budget for such training. With the approved
budget, Human Resources department has implemented annual training program for team
leaders with a hope that those will actively contribute for department and company’s
business effectiveness.

1


1.2 .Problem statements
With the large budget, programs are well prepared but after training course for team

leaders, there has not been a formal evaluation about the effectiveness of training
programs for team leaders, departments and company. The evaluation is quite simple,
only evaluates the satisfaction of trainee base on survey and test gained knowledge and
skills base on an short after training test. Because the training programs are considered as
the key in training activities and it cost a large amount of money, so that evaluate its
impact on trainee, department and company level is the urgent requirement from new
investor UTC. That is the reason why this research concentrate on evaluate efficiency of
VNPT EPAY training program of team leaders, point out its strengths and weaknesses of
training program with will be the basis to improving the effectiveness of the training
programs.
1.3 .Research goal
The research has two main goals. First, evaluate the effectiveness of training program for
team leaders of VNPT EPAY. Second, suggest solutions to improve effectiveness of
coming and future Training program for team leaders in VNPT EPAY. Besides, the
results of the research are also the basis for science and practice to help VNPT EPAY to
have model to evaluate the same training programs.
1.4 .Scope of the Research
The research is conducted on different phrases of training program for team leader within
VNPT EPAY, from preparation of the training until trainees apply the knowledge and
skills in working environment. Each stage corresponds to a level of evaluation of the
training results, so each level of Evaluation does not have the same scope of study:
Level 1: Evaluation of the middle and after training course. The purpose of the first level
is evaluation of team leaders on the quality of the program. The survey included 42 team
leaders of the courses in 2016 and in 2017.
Level 2: Evaluation after finishing training courses. The goal of the second level is
evaluate knowledge and skills that team leaders gained from the courses.
Level 3: Evaluation of the period after finishing training courses 3 month. The evaluation
is concentrate on measure the changing in behavior of team leaders after participate on
training courses.

2


Level 4: Evaluation of the period after training courses finish 6 months. The research’s
content of this level is evaluation the effectiveness of training course on team leaders
performance, teams and department performance.
1.5 .The meaning of the Research
The research helped Training team and HR department report the effectiveness of
training program for team leaders with in VNPT EPAY. Especially, the research
provided for VNPT EPAY an overview of training program for team leader, so that
VNPT EPAY have chance to review and apply solutions to improve next training
programs.
In addition, this research can be used as reference material for researchers and training
people as well as training units in this field.
1.6 .Research structure
The thesis consists of 5 chapters, including:
Chapter 1. Research introduction: This chapter covered the context and criticality of
the topic; research objectives, objects, scope of research, practical significance of the
topic.
Chapter 2. Literature review: This chapter presented theoretical foundations, models
for evaluating effectiveness of training in the world. An overview of VNPT EPAY and
team leader’s training program also be presented in this chapter.
Chapter 3. Research methodology: This chapter presented the topic analysis as
well as research method.
Chapter 4. Research result: Analyze the generalized results of each level of
evaluation.
Chapter 5. Discussion and conclusion
This is the last chapter; this chapter proposes a number of measures to improve
the training effectiveness of the training program for team leaders of VNPT EPAY.

3


CHAPTER II
LITERATURE REVIEW
2.1 Research definition
2.1.1 Training definition
According the Education Dictionary (2001), training is a systematic process of
transferring experiences, knowledge, and skills. Setting and preparing the mind for
learners to enter working life independently, build and protect the country. Today,
training is gradually advanced to bilateral cooperation with learners to help them
actively and intellectually occupy their knowledge, equip them with professional
knowledge.
Generally, training refers to the teaching of practical skills, occupations or
knowledge related to a particular field so that people learn and master knowledge,
skills and occupation in a system to prepare him or her to adapt to life and
ability to take over a certain job. There are many types of training: basic training
and intensive training, professional training and vocational training, re-training, distance
training, self-training... Briefly, training focuses on “learn how”.
2.1.2 Evaluation of training programs
2.1.2.1. Evaluation
Worthen and Sanders (1987) assumed that evaluation is the process of determining
the quality, effectiveness and value of a program, product, project or training
program, including the methods of investigation, interview, judgment, and proceeding
step by step:
First, identify the criteria for quality Evaluation and decide whether the standards
are relative or absolute;
Second, collect related data and information;
Third, apply the selected criteria to determine quality

4


2.1.2.2. Training program
Eddie Davies (2007), a multi-purpose training program, an organization that
implements the training program for a variety of reasons, one of which may be:
- Change awareness, increase knowledge and skills;
- Provide human resources with new skills to cope with the challenges of new
technology;
- Provide employees with new knowledge and skills to make job changes;
- Add the missing skills of each employee to complete the job beyond expectations.
2.1.3. Evaluation procedure
The general procedure for evaluation of training program including 3 steps.

Step 1

Step 2

Step 3

• Define evaluation criteria

• Data collection and related information

• Evaluate selected criteria

Figure 2.1. The procedures for general evaluation for training results
Source: Huỳnh Lợi, 2009
2.1.4. Evaluation of skills

5


According to Kavita Gupta (2007), there are some methods to evaluate training
effectiveness, including:
Structured interview:
Quantitative research methods are often conducted in survey studies to ensure that
each candidate is asked the same topic in the correct order and can be fully
aggregated.
The answer can be compared carefully among the subgroups or among different surveys.
Define evaluation criteria
Data collection and related information
Evaluate selected criteria
The interviewers read exactly the questions orderly in survey. The answer is mainly for
closed-ended questions, although sometimes open-ended questions also appear in
structured interviews.
Semi-structured interview (later interview):
Unlike structured interviews, semi-structured interviews have more questions or
more general topics. Relevant topics are identified, and the intimate relationship
between the themes and issues becomes the basis for questions that the
respondents do not need to prepare in advance, which allows both interviewers and
candidates to have the flexibility to learn more about the details or discussed issue.
The questioners should prepare questions in the interview, from that the interview
can develop like a dialogue.
Observation:
Observe the working environment and activities of employees (office skills,
communication tools, information technology systems, communication methods).
Survey by questions:
Questionnaire is a survey that consists of a series of questions and other notes to
gather information from respondents. People often design questions to analyze the

statistics from the answers.

6


Gathered group discussion
The qualitative research method aims to gather extensive information about the
ideas and perceptions of a group and at the same time focus more on the
common questions and answers.
Organize in small groups (usually six to twelve) under the guidance of a facilitator, in
which members can discuss freely and continuously on a given topic.
Hold conference:
An educational seminar or series of meetings usually focuses on interacting and
exchanging information among a small group of participants, thereby enhancing
common skills or knowledge in a number of ways.
Discuss the need for employee training needs through interviews and interview
results.
2.2 Evaluation Training Effectiveness Model
2.2.1 Purpose of Training Evaluation
Evaluating a training program can indicate whether the training is meeting its intention
and goals. Analysis may include a review of the program’s objectives, methods,
feedback, and results in terms of its overall effectiveness. Goldstein (2002) states training
evaluation as a “systematic collection of descriptive and judgmental information
necessary to make effective training decisions related to the selection, adoption, value
and modification of various instructional activities” (p. 138). Training is futile if it does
not make a difference that provides practical relevance and value to an organization
(Holt, Boehm-Davis, & Beaubien, 2001). The impact of training may be seen on one or
more distinctive levels including the individual, the team, and the organization. As
mentioned earlier, purpose of training program for team leaders is to provide new skills
and knowledge which lead to change in team leaders, department and organization level.

So that, evaluate the effective of training will identify aspect that VNPT EPAY can
change or improve to increase the effectiveness of this training.

7


2.2.2 Common Evaluation Technique
There are several techniques used to evaluate the effectiveness of a training system or
program. According to Bennett (1982) and Bennett & Langford (1979), six key
criteria were established to define training effectiveness. This receives supports from
over 23 researchers and writers in the field of human resource development and
organizational
behavior. These criteria includes the followings:(i)General
effectiveness (e.g. goal settings, planning, etc) (ii) decision making, (iii) delegation (iv)
communication, (v) Job knowledge, and (vi) relationship. Although these criteria may
not altogether fit into every organizations concept of HRD effectiveness, the basic
point is that training and HRD can be classified into two main categories, skill
enhancement and behavioral changes. Skill improvement may be achieved through
training program such as seminars and workshops on specific and identified “skill gap”.
On the other hand, behavioral change may show in form of attitude and mode of
communication to supervisors, peers and subordinates. Magerison (1982) and Gough
(2006) further pointed out that that HRD and training must aligned with strategic
direction of the organization and invariably it must be focus on organizational goal
and culture. This is necessary to meet the ever changing operating environment such as
trends in international and local markets, changing attitudes of workers, productivity
requirement. Moreover, for HRD and training to succeed, it must receive the support of
the top management. Kirkpatrick’s model - Evaluation of Training Phillips (1997)
defined training as a systematic process of examining the worth, value, or meaning
of an activity or a process. Since a particular method of evaluation can be applied in all
cases there is the need to develop several method of measurement. While there

are several model and format developed for measuring HRD and training effectiveness,
the most accepted model is that developed by Kirkpatrick. He proposed four areas
that required measurement, when analyzing the effectiveness of training programthat is emotional reaction, achievement of objectives, behavioral changes and
organizational impact.
2.2.3 Kirkpatrick’s model to Evaluate Training effectiveness
2.2.3.1. Overview about Kirkpatrick model
Donald Kirkpatrick is an great professor in Wisconsin university, he is also work
as former president of American Society of Training and Development (ASTD). In
1959, he introduced the evaluation mode of training four levels in American Society of

8


Training and Development magazine. This model was updated twice in 1975 and
1994 when he launched his famous 'training program evaluation'. Summarize four
levels of Kirkpatrick model in table 2.1

Table 2.1.Summarize the Kirkpatrick model
Level

Focus on measuring

The necessary questions

1- Response

The students' awareness

What do students think about training?

2- Study

Achieved knowledge and Is there an increase in knowledge, skill
skills
after training?

3- Behavior

Real
observation
workplace

4- Result

Affect the organization

in Are new knowledge and skills used in
working?
What results are created in organization?
Source: Eddie Davies, 2007

The scale to measure the evaluation is:
1- Really good;
2- Quite good;
3- Unacceptable;
4- Not meet the requirement;
5- Very bad.
2.2.3.2 Reactions
Kirkpatrick’s first level of training evaluation, “reactions,” is concerned with simply the
satisfaction of the participants with the training. It is a measure of whether the reactions

9


of the trainees were positive or negative. In essence, it is a way to determine customer
satisfaction and often referred to as a “happy sheet.”
Kirkpatrick (1996) contents that assessing how people feel about the training is important
for two primary reasons. First, management decisions regarding training are often based
on feedback provided from trainees (Kirkpatrick, 1996). If there exists a general feeling
that the training seems useless or unpleasant, it may be subject to termination. Managers
and supervisors deeply desire for their employees to enjoy the training in their
organization. Besides, training is not cheap. A great deal of resources, manpower and
money, are directed towards successful training programs. Organizations genuinely yearn
for these investments to be worthwhile so as to avoid wasting time and assets on training,
potentially leading to pessimistic attitudes.
Secondly, it is important to understand trainees’ reactions simply because reactions may
influence learning. Research has been unsuccessful in directly linking satisfaction with
the effectiveness of a training program (Mathieu, Tannenbaum, & Salas, 1992; Alliger &
Janak, 1989). However, few would argue that it makes sense to conduct a training
program with unfavorable reactions. Positive responses, unlike negative, are rarely
harmful to the learning process. In fact, many would argue that positive reactions may
enhance an individual’s motivation to learn (Salas, Burke et al., 2001). An individual
who enjoys a training course can become readily enthusiastic and receptive to learning.
On the other hand, if a person feels the training is of no value, it can reduce the likelihood
learning took place (Kirkpatrick, 1996). It is important to realize negative reactions are
not always adverse. Honest reactions can inform decision makers for potential areas of
improvement in a course or prospective changes to instruction methods.
Assessing the reactions of participants is common practice in many training
organizations. Unfortunately, analyzing reactions is often misconstrued as a method to
evaluate a program in its entirety. Just because a training program receives enthusiastic

reviews does not correlate to the value added or make the training necessarily effective
(Kirkpatrick, 1996). Likewise, learning and change can occur even if the participant does
not particularly enjoy the instruction. Organizations commonly fail to conduct necessary
higher-level evaluations (Lovell, 2007). Reaction assessment, although important and
constructive, is only the first step in the evaluation process, and further efforts are needed
to assure learning and outcomes are going to result from the training.
Evaluating reaction, often called a level 1 evaluation, typically consists of a reaction
sheet. The purpose is to measure how the attendees feel about the program.
Unfortunately, some people put little faith in this evaluation level and negatively refer to

10


it as smile sheets. In fact, some recent literature suggests that these smile sheets do more
harm than good. Instead of perpetuating that notion, let’s review the definition of
reaction, which measures customer satisfaction. Customer satisfaction is essential. If
customers report bad things on a reaction sheet, they likely will say bad things when they
return to their job. The result: Higher management may judge the training based on what
they hear. Therefore, reactions are important to evaluate, not only to benefit the
instructor, but to know that customers have gone back to their jobs with a positive
attitude toward the program and the trainers who presented it. Additionally, if you don’t
measure customer reaction and satisfaction, you are subtly telling them that their
feedback isn’t wanted or needed. The answer is simple: use reaction sheets for all
programs.
Guidelines for evaluating reaction include
1. Determine what aspects of the program you want to evaluate.
2. Design a form that will quantify reactions.
3. Encourage written comments.
4. Get 100 percent immediate response for an instructor-led program or as close
to 100 percent you can get if evaluating an e-learning program.

5. Develop acceptable standards against which to measure actual ratings.
6. Make trainer decisions based on actual ratings against the standard.
2.2.3.2 Level 2 Evaluation: Learning
The level 2 in evaluation system which is related to students' study results. The
learning outcomes are based on the amount of knowledge, skills and attitudes that
the trainee learns from the course. The Evaluation of level 2 is intended to
determine the extent to which the trainee can improve, enhance and expand their
knowledge and skills after taking the course. This level can be taken throughout
the course and use a variety of Evaluation
methods including questionnaire
surveys, surveys, observations, paper theoretical tests, practice tests, group test,
self-Evaluation .
Normally, the level of change in knowledge, skills and attitudes of trainees
through training is determined by comparing and contrasting the results of the pre-

11


entry exam and the exam after the course ends. The differences between two
above exams will show clearly what students can learn from the course. Based on the
students' score, the enterprise can determine quite precisely the impact of training on
students. However, the evaluation of training results has to base on set goal of
training program. Therefore, the test and exam has to stick to these goals.
The following instruction can be used for evaluating the levels:
- Use a group control (if possible);
- Evaluate the knowledge, skills, attitude before and after the training course;
- Use the written test to evaluate the skills and attitude;
- Use the performance tests to Evaluation of the skills;
- Achieve 100% response from students.
2.2.2.3 Level 3 Evaluation: Behavior

What happens when trainees leave the classroom and return to their jobs? How much
transfer of knowledge, skills, and attitudes occurs? That is what level 3 attempts to
evaluate: behavior. In other words, what change in job behavior occurred because people
attended a training program?
This level is more complicated than the first two levels. First, trainees cannot change their
behavior until they have an opportunity to do so. For example, if you decide to use some
of the principles and techniques described here, you must wait until you have a training
program to evaluate. Likewise, if the training program is designed to teach a person how
to conduct an effective performance appraisal interview, the trainee cannot apply the
learning until an interview is held.
Second, it is impossible to predict when a change in behavior will occur. Even if a trainee
has an opportunity to apply the learning, he or she may not do so immediately. In fact,
behavior changes may occur at any time after the first opportunity, or they may never
occur.
Third, the trainee may apply the learning to the job and come to one of the following
conclusions:
■The participant liked what happened and will use the new behavior.

12


■The participant didn’t like what happened and will continue the old behavior.
■The participant liked what happened, but the boss or time restraints prevent him
or her from changing.
We all hope that the rewards for changing behavior will cause the trainee to come to the
first of these conclusions. It is important, therefore, to provide help, encouragement, and
rewards when the trainee returns to the job from the training class. One type of reward is
intrinsic. This term refers to the inward feelings of satisfaction, pride, achievement, and
happiness that can occur when the new behavior is used. Extrinsic rewards also are
important. These rewards come from the outside. They include praise, increased freedom

and empowerment, merit pay increases, and other forms of recognition that come as the
result of the change in behavior.
Guidelines for evaluating behavior include
1. Measure on a before-and-after basis if possible.
2. If not possible, measure on an after basis and ask the participant to compare it with his
or her before behavior.
3. Allow time for new behaviors to be used. This may vary from one day to three to six
months.
4. Get 100 percent response or a sampling. The sampling depends on the resources
available.
5. Repeat at appropriate times. For example, if the first measurement was done after three
months, the second might be done after six months. During that period, some new
behaviors might have occurred or some behaviors that were tried were no longer used and
the participant went back to the old ways.
6. Use a control group if practical.
2.2.2.4 Level 4 Evaluation: Results
Finally we get to what it’s all about: results. It is important to go through all three
evaluation levels before trying to evaluate results—and ROI in particular. Results are the
last thing to evaluate, but the first thing to consider when planning a training program.
When planning a program, the questions to ask, in order, are

13


■What results do we hope to achieve through a training program?
■What behaviors are needed to accomplish those results?
■What knowledge, skills, and attitudes are necessary to initiate these behaviors?
■How can I present a training program that is practical, interactive, and enjoyable?
Some guidelines for evaluating results include
1. Measure on a before-and-after basis.

2. Allow time for possible results to take place.
3. Repeat at appropriate times.
4. Use a control group if practical.
5. Consider costs versus benefits.
6. Look for “evidence” because absolute “proof” is not possible.
Evaluating results is usually easier than evaluating changes in behavior because specific
data are often readily available. For example, when measuring the reduction in turnover,
figures are easy to get. Likewise, increase in sales, increase or retention of customers,
change in profits, amount of scrap, and time to complete a job are usually available.
Soft skills such as leadership, communication, managing change, and decision making
are more difficult to measure and cannot be measured in dollars or in ROI. Some specific
measures are available that can relate to these programs if the organization has an annual
attitude survey, but converting the comparisons to money is not viable unless they can be
converted to such measurable items as turnover.
Note that a “chain of evidence” can be created by evaluating all four of the levels in
sequence. Each bit of evidence can help build the overall case that the program was
effective. So, when evaluating, do all four levels and don’t yield to the temptation of
jumping from learning to results or ROI.
CHAPTER II SUMMARY
What began as a dissertation topic in the 1950s has become an integral part of training
programs around the world. Four simple and practical words have formed the basis for

14


this phenomenon: reaction, learning, behavior, and results. These four levels can be used
not only to evaluate the effectiveness of a learning program but can also be useful in
getting the desired changes to take place. Although learning is the desired goal of any
training program, the most critical aspect is that the learning creates the desired change in
behavior that must take place for positive results to be achieved.

15


CHAPTER III
RESEARCH METHODOLOGY
3.1 Research process
RESEARCH GOAL

Theological Research
Basic definitions
Training theology
Evaluation procedure
Common evaluation
model
Kirkpatrick Model

Data collection

Primary Data
Survey all managers in the course:
- Evaluate Reaction
- Evaluate skills and knowledge
acquisition
- Evaluate personal applicant
level
- Evaluate influence in team,
department and company

Secondary Data

An overview about
VNPT EPAY Jsc
- Recent training program
for Team leaders:
+ Training goals
+ Procedure, training
form, organized courses;
+ Evaluating training
effectiveness
-

Research Result

Evaluate effectiveness of training program

Evaluation criteria
Factor group 1

Stage 1: Reaction

Factor group 1

Stage 2: Skills and Knowledge acquisition

Factor group 1

Stage 3: Personal application

Factor group 1

Stage 4: Influence of training program

Result of training evaluation

Recommend solution to
enhance training program
for Team leaders at VNPT
EPAY Jsc

16

YES

NO
Positive factors
Good evaluation

Figure 3.1.Research process

Maintaining the recent
training program


3.2. Research method
3.2.1. Preliminary research
Use qualitative research methods. Group discussion techniques with internal
instructors and trainee administrators to explore, adjust, and supplement the
observational variables that measure the research concepts. The results of the group
discussion were that the observational variables of the four Kirkpatrick training
evaluation levels were adjusted in accordance with the specific characteristics of

the Training program for Team leaders at VNPT EPAY.
3.2.2. Official research
Use descriptive statistics after direct interview with pre-designed questionnaires.
A survey of 42 trainees who are Team leaders participated in the training of Team
leaders.
Group course based on different evaluation level as in Figure 3.2.
Level

Time

Program

Level 1

From 5.2017 – 6.2017

Training course 1 – 3

Level 2

From 6.2017 – 7.2017

Training course 1 – 3

Level 3

From 5. 2017 – 8.2017

Training course 1 – 3

Level 4

From 5.2017 – 8.2017

Training course 1 – 3

Design observational variables and survey question table
Design questionnaire evaluating the effectiveness of the training program for
team leaders at VNPT EPAY. The questionnaire focused on 4 key components
including: (1) satisfaction after training; (2) learning; (3) Application; (4) Work results.

17


The questionnaire used a 5-point Liker scale to Evaluation of the respondent's
level of agreement with: 1 - Strongly Disagree, 2 - Disagree, 3 - Unconscious, 4
- Agree and 5 -Very Cop Mind
The dimensions of trainees' satisfaction after the course
Lee, S.H. and Ming, J.A., 1999 have introduced comprehensive dimensions to evaluate
reaction of a training program including:
 Program objective(s)/ content
 Program materials
 Delivery method/technologies
 Instructor/facilitator; instructional activities
 Program time/length
 Training environment
 Planed action/transfer expectation
 Logistic/ administration
 Overall evaluation
 Recommendation for program improvement.

In this research, we refer dimensions above because two reasons: First, dimensions and
its questions are appropriate to fully and detail evaluate a training program. Second, the
dimensions are suitable with the purpose of evaluating the training program for team
leaders in VNPT EPAY. However, with the scope of the research, we use only following
dimensions. As well as this, we adjusted some questions and removed some unnecessary
questions. The dimension used in this research:
 Program objective(s)/ content
 Instructor/facilitator; instructional activities
 Program time/length
 Logistic/ administration
Thus, used satisfaction questionnaire is designed as following:
Table 3.1: Scale of trainee’s satisfaction
Sign

Observed Variables

A1

The program content met the stated objective

18