Practicing Organization Development (A guide for Consultants) - Part 37 docx

EVALUATION 331
Figure 12.4. Organization Development Evaluation Model
10. Administer
Longitudinal Evalutation
1. Do Assessment
7. Analyze Results
8. Present Results
9. Make Revisions
2. Determine OD Needs
3a. Determine Level
of Evaluation
3b. Design Intervention
4b. Develop Evaluation
Instrument
4a. Develop Intervention
5b. Administer Formative
Evaluation
5a. Implement
Intervention
6. Administer
Summative Evaluation
18_962384 ch12.qxd 2/3/05 12:18 AM Page 331
IMPORTANCE OF EVALUATION
If you don’t know where you are going, any path will get you there. And if you
don’t know where you are, you’ll never know where you are going. The process
of evaluating is truly about determining whether the path you are on has been
the most efficiently executed and effective path for the system you are in. In
turn, this will influence how you move forward into the future. The funda-
mental premise is self-awareness. A system of any size—from one person to a
whole organization—will be most amenable to change when it has all relevant
data. The more that the people in a client system are able to understand them-

selves, the greater the likelihood that they will take ownership in determining
how to best move forward.
Within this context, the key change agents are more able to influence the sys-
tem with data that demonstrate success and failure. Action learning focuses on
important outcomes—What was planned? What happened? What was learned?
What will be done next time? One critical step to creating learning is addressed
in the question: What happened? Success and failure are concepts critical to cre-
ating a learning organization, and this is where the consultant as intervener
comes into the picture. By providing the system with avenues for having a bet-
ter understanding of itself through rigorous evaluation, the system will begin to
change its thinking paradigm. When the system begins to change its thinking
paradigm, the learning has begun. There is no other more powerful way to cre-
ate change than through learning rooted in the shifting of how we think. This
learning is what will lead to new and sustainable organizational habits that
result in long-term health and prosperity.
WHERE EVALUATION FITS IN THE
INTERACTION WITH CLIENTS
As indicated earlier, it is important to establish the evaluation processes during
the pre-launch phase. In this way, what is important for the client, the consul-
tant, the participants, and the process can be established before the process is
begun. With a plan in place, continuous improvement in the OD process, based
on identified evaluative feedback, can occur.
Evaluation can occur at three points in the OD process: during the process
(formative), at the conclusion of the process (summative), or at some time after
the process has ended (longitudinal). The most important of these times is
during the OD process—formative evaluation. To discover that something could
have been done differently or that the process did not create the intended
332 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION
18_962384 ch12.qxd 2/3/05 12:18 AM Page 332
outcomes after the process is over does not allow for adjustments or modifica-

tions along the way.
WHO SHOULD DO THE EVALUATION?
This decision, too, should occur during the pre-launch phase. There are three
common answers to the question of who should perform the evaluation: the
client organization, the consultant, or a third party. Many factors will influence
where the responsibility for evaluation will rest. Given the preference for for-
mative evaluation expressed earlier, the responsibility for evaluation will fall
equally with the consultant and the client. Regular, ongoing evaluation will pro-
vide feedback continuously, if so designed, and will allow the consultant and
the client to partner to maximize the potential in the OD process. For summa-
tive and longitudinal evaluation, there are pros and cons for each potential eval-
uator. These are summarized in Exhibit 12.1.
KEY MEASURES IN OD
When taking a systems approach to measuring in OD, we advocate the inclu-
sion and integration of six key dimensions: performance, morale, emotion, crit-
ical customer service behaviors, organization climate, and personal characteristics
(Cady & Lewis, 2002). Each component falls into one of two categories of mea-
surement, soft or hard. Soft measures are perceptual in nature. That is, the scor-
ing of soft measures is based on an individual’s judgment or opinion. For
example, an employee is asked to respond in a survey to the statement, “I am
satisfied with my job.” The employee chooses a number on a seven-point Likert
scale ranging from strongly disagree with the statement (equal to 1) to strongly
agree with the statement (equal to 7) (Rea & Parker, 1997). Hard measures are
observable and can be objectively counted (that is, not based on opinion), often
based on behavior or units of some kind. For example, the number of units sold
or made is a hard measure.
Taking a systems approach to evaluation is important in encouraging mem-
bers of an organization to understand how their actions and behaviors (soft
measures) influence important bottom-line outcomes (hard measures). A sys-
tem model can be used to provide meaningful ongoing feedback at all levels, to

diagnose departmental or organization-wide performance issues, and to iden-
tify appropriate interventions that will positively impact performance. We will
now review the six key measurement dimensions mentioned earlier that should
be considered when taking a systems approach to evaluation.
EVALUATION 333
18_962384 ch12.qxd 2/3/05 12:18 AM Page 333
334 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION
Exhibit 12.1. Advantages and Disadvantages to Possible Evaluators
Advantages Disadvantages
Client Organization
• Not an additional cost to the organization
• Will be present to the organization long
after the specific OD process is over
• Knows the culture of the organization
better than an outside party can
• Knows what is important to the
organization and, thus, what needs to
be evaluated
• Because of responsibility for participating
in planning the OD process, can build
evaluation into pre-launch phase
• May be too closely associated with the
organization to see what is necessary in
the evaluation
• May fear that the outcome of the
evaluation will cause them to lose
credibility if they were responsible for
the contract
• May not have time to do evaluation
effectively because of other projects

• May not have the necessary competencies
to carry out the evaluation
Consultant
• Knows the OD process and its intended
outcomes as well
• Should have the necessary competencies
to carry out the evaluation
• Because of the responsibility for partici-
pating in planning the OD process, can
build evaluation into the pre-launch phase
• Gives the consultant the opportunity to
modify processes used or provide
adequate compensation for mistakes
made
• Adds to the project‘s costs
• Unless a long-term involvement is
envisioned, the consultant may not be
around to complete the evaluation
• May not be viewed as objective as there
is an incentive to make the evaluation look
good (for affirmation of client‘s choice) or
for additional needs to surface (for
additional work)
• May create a sense of client‘s dependence
on the consultant
Third Party
• Viewed as more objective because not
involved in any other way in the OD
process
• Usually more expert because of the ability

to specialize in evaluation
• Has nothing to lose or gain in the outcome
of the evaluation
• Clearly, an additional cost
• A third party may not want to take on such
a limited component of a project
• Is not integral to the project, so may not
know either the process or the outcomes
to be assessed
• Is probably not in a position to influence
improvements in the process or in im-
proving the ODs consultants processes
• Guarantees that the evaluation will not be
formative
Measurement Dimension 1—Profitability and the Bottom Line
When discussing the bottom line, many people think of profitability first. In this
chapter, profitability is part of a broader definition of performance that is based
on a balanced approach to assessing the bottom line. There are three perfor-
mance measures, the first of which is profitability. Profit is one measure of per-
formance that has dominated throughout the history of economic ventures, and
it remains a central focus today. However, there is growing recognition that
18_962384 ch12.qxd 2/3/05 12:18 AM Page 334
profit is not the only indicator of the real bottom line (for example, Kaplan &
Norton, 1996). For non-profit organizations, consider cash flow as a measure
that replaces profitability for a for-profit organization.
There are other measures, along with profit, that provide a more balanced
approach to assessing the bottom line. That is why the word performance is
used in place of bottom line. In addition to profit, there are internal and exter-
nal performance measures, such as customer retention, organizational learning,
new products developed, and many more.

Measurement Dimension 2—Morale
Measures of morale are soft by nature. That is, asking the employees or cus-
tomers their opinions regarding a variety of issues assesses each measure. Mea-
sures of morale address attitudes and intentions from both the employee and
customer perspective. An attitude is a bi-polar affect regarding some target (for
example, I like it or I don’t like it). Attitudes focus on such things as job satis-
faction for employees and product/service satisfaction for customers. Intentions
are cognitive choices made with regard to pending action or lack of action (for
example, I will do this or I will do that or I will do nothing). Intentions focus
on such things as intention-to-leave a job or intention-to-return to buy a product/
service again. As mentioned earlier, these measures tend to focus on a degree
of agreement or disagreement on a Likert scale.
Measurement Dimension 3—Emotion
Emotions are similar yet distinct from morale. Emotions encompass glad, sad,
mad, afraid, and ashamed. There are also the emotions of excited, peace, and
numb. These emotions tend to be related to discrete events and are reactionary
in nature. The reason that they are considered similar to morale is that they are
known to influence morale directly. For example, a failed performance by an
employee (for example, a lost sale) can lead to anger and guilt. These emotions
can subsequently lead to an exaggerated low level of job satisfaction and a
heightened level of intention-to-leave. Over time, these levels of satisfaction and
intention-to-leave can return to the original levels prior to the failed perfor-
mance. That is, emotions generally have a temporary influence on morale, not
a permanent one.
Measurement Dimension 4—
Critical Customer Service Behaviors
Customers have certain expectations when receiving products and services.
There are all sorts of support required and questions to be asked. In the process,
the customer can define a set of employees’ behaviors that add value to the
product or service. These behaviors are applicable across industries, yet must

EVALUATION 335
18_962384 ch12.qxd 2/3/05 12:18 AM Page 335
be customized to fit a particular industry (service or product). There are five
critical employee behaviors that can lead to higher levels of customer satisfac-
tion and retention: response time, personalized attention, professionalism,
expertise, and comforting.
Measurement Dimension 5—Organization Climate
There is a considerable amount of research and numbers of survey measures
that are directed toward a better understanding of those factors that directly
influence task performance. These task-dependent measures fall into three
variables: job climate factors, teaming behaviors, and leadership behaviors. Job
climate factors are directly related to the work being done and include envi-
ronmental support, personal capabilities, and reward systems. Teaming behav-
iors are the critical behaviors team members should exhibit that lead to a
high-performing team. Teaming behavior measures include the following: antic-
ipates and fulfills team needs; manages own personal contribution; displays a
positive attitude; and effectively communicates. Leadership behaviors are those
critical behaviors that leaders should exhibit to create a high-performing team.
Leadership behaviors include the following: supporting, informing, monitoring,
recognizing and rewarding, empowering, and clarifying.
Measurement Dimension 6—Personal Characteristics
There are two types of personal characteristics, each of which contains differ-
ent measures. First, there are biographical characteristics that are considered
hard measures and are usually found in most personnel files. Biographical char-
acteristics include such things as age, race, gender, tenure, and function. The
second type of personal characteristic is personality. Personality refers to more
stable behavioral traits that people exhibit. For example, there are a variety of
personality models, such as the Myers-Briggs Type Indicator (MBTI), the Big
Five (extraversion, emotional stability, agreeableness, conscientiousness, and
openness to experience), and other non-dimensional measures, such as locus-

of-control and pessimism. While these tend to be very popular in use, their
validity is suspect.
TYPES OF EVALUATION
There are many ways to perform an appropriate evaluation. In planning an OD
evaluation, it is important that an agreement be reached in the beginning about
what is important and how it will be determined, whether the outcome has
336 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION
18_962384 ch12.qxd 2/3/05 12:18 AM Page 336
been achieved or not. As described earlier in this chapter, it is important to
decide who will conduct the evaluation. Outcome measures must be identified.
A time frame—formative, summative, and longitudinal—must be determined.
And then the appropriate tool or approach must be chosen.
Formal/Informal
A formal approach to evaluation will be decided among the parties involved in
the OD process. Informally, however, all of the parties involved will be con-
stantly looking at ways to improve their roles in the OD process. Sometimes this
may be apparent, as when a consultant might simply ask a client contact how
things are going. Often, however, the informal evaluation process will not be
apparent. It may consist of continually observing how people are responding
and participating, checking on available statistics that might reflect on the
desired outcome measures, and even watching stock prices. The concern with
informal evaluation is that the results are not shared and are not systematic. As
a result, partnering between consultant and client loses synergy, and results that
surface are biased by happenstance observations rather than systematic ones.
It also makes great sense in summative and longitudinal evaluation to repeat
whatever was done in the assessment stage. Since that process was successful in
identifying the focus for the OD process undertaken, it makes sense to use the
same approach in evaluating the outcomes of that process.
Thus, the recommendation of the authors is that a shared, systematic per-
spective be selected from among the many formal modes of evaluation that are

available. There are many ways in which such approaches could be classified.
We have chosen to classify them as quantitative (also called hard) and qualita-
tive (also called soft). These will be detailed further in this section.
Quantitative
A preference still exists in business to have quantifiable data, and many of the
traditional approaches to evaluation in OD rely on quantifiable data. These data
come from a variety of sources—surveys (both simple and complex), measur-
able outcomes (such as discussed earlier in this chapter—both in terms of meet-
ing pre-specified goals and pre-process and post-process comparisons), and
benchmarking (especially with competitors) are the primary approaches used.
Surveys
These can be as simple as a basic satisfaction survey (often referred to as a
happy sheet) to a somewhat more complicated feedback form (for example, an
assessment of how meetings are going) to multi-rater feedback to a very com-
plex culture survey of the organization.
EVALUATION 337
18_962384 ch12.qxd 2/3/05 12:18 AM Page 337
The basic satisfaction survey is probably the easiest to construct and use and,
thus, is frequently used. Items to be measured might include participants’ satis-
faction with the OD process (activities, materials, consultant, facilities, and so on).
Sample items are presented in Exhibit 12.2.
A somewhat more complicated survey, although based on the same con-
cepts as the satisfaction survey, is one that focuses on a specific function
within the organization—for example, how well executive team meetings are
going, level of customer satisfaction, product innovation, and so on. See
Exhibit 12.3 for an excerpt from a survey frequently used by one of the
authors.
Multi-rater assessment is another survey form that is often used when the
OD focus is on an individual or a small group of individuals. This is often
referred to as 360-degree feedback. When completed in a survey form, it pro-

vides quantitative data. There are many issues with this type of approach, espe-
cially when used inappropriately to determine salary and promotion potential
(see McLean, 1997). A prime concern with the use of such an instrument is
that research shows that different groups of reviewers provide different feed-
back, naturally, as people are observed by different groups of people in differ-
ent contexts.
Finally, the most complex form of survey is the climate (or culture) survey.
This is usually a comprehensive review of the factors in the organization that
affect the climate (or culture). A controversial issue that frequently surfaces with
surveys is whether to use a customized survey (developed by the consultant or
a team in the client organization) or a commercial, standardized survey. See
Exhibit 12.4 for a comparison of the two approaches.
338 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION
Exhibit 12.2. Sample Items from an OD Process Satisfaction Survey
Evaluation of Team Building OD Process
In responding to the following questions, circle the number that corresponds with
the following scale:
5 = Very Satisfied; 4 = Satisfied; 3 = Neutral; 2 = Dissatisfied; 1 = Very Dissatisfied
1. How well the need for team building 54321
was originally identified
2. How appropriate the retreat 54321
facilities were
3. How effective the team-building 54321
activities at the retreat were
18_962384 ch12.qxd 2/3/05 12:18 AM Page 338
EVALUATION 339
Exhibit 12.3. Survey to Determine Satisfaction with a Team Meeting Process
Team Meeting Assessment
Circle the number to the right of each item that matches your assessment of how
well the team is currently functioning. Use the following scale:

5 = Strongly Agree; 4 = Agree; 3 = Neutral; 2 = Disagree; 1 = Strongly Disagree
Planning
1. The right people are on the team. 54321
2. An agenda is distributed before 54321
meetings.
3. Supporting papers are distributed 54321
before meetings.
4. The agenda items are timed. 54321
5. The agenda contains expected 54321
outcomes.
6. Visual aids are available, including flip 54321
charts, markers, tape, sticky notes, etc.
7. Room arrangement maximizes 54321
interaction.
8. Participants bring distributed 54321
materials to meetings.
9. Presenters are prepared when 54321
scheduled.
10. Agendas adequately reflect meetings. 54321
Doing
11. Meetings start on time. 54321
12. Everyone is present on time. 54321
13. Meetings focus on improvement 54321
rather than providing information.
14. Minutes are taken at all meetings. 54321
15. Telephone calls do not interrupt 54321
meetings.
16. Discussion is focused on outcomes; 54321
it does not ramble.
17. Each team member fully participates 54321

when competent to do so.
18. Team members do not dominate 54321
discussion.
19. Each agenda item ends with a 54321
clearly developed action step.
20. Blaming does not occur among 54321
team members.
Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc. Reproduced
by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
18_962384 ch12.qxd 2/3/05 12:18 AM Page 339
340 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION
Exhibit 12.3. Continued
Doing
21. Team members do not interrupt 54321
each other.
22. I feel free to express my opinions 54321
without fear of reprisals.
23. Decisions are made on a 54321
consensus basis.
24. All team members accept responsibility 54321
for the quality of team meetings,
providing feedback during the meeting
on ways to improve meetings.
25. Risk taking during meetings is 54321
encouraged.
26. Meetings end on time. 54321
27. Meetings end early if the agenda 54321
is completed.
28. The group leader uses good 54321
facilitation skills.

29. Conflict is encouraged. 54321
30. Conflict is handled in a healthy way, 54321
focusing on facts and “I” statements.
31. Team members listen to each other. 54321
32. Team members listen to new ideas 54321
without responding with “Yes, but.”
33. Team members build on one 54321
another’s new ideas.
34. The agenda consists of items at an 54321
appropriate level, delegating detail
to appropriate employees.
35. Team members trust each other. 54321
36. Team members respect each other. 54321
37. Agenda items for the next meeting 54321
are identified before the end of
each meeting.
38. Appropriate problem-solving tools 54321
are used during meetings.
39. Sufficient resources are provided 54321
for the team’s action steps to be
carried out.
40. The next meeting date is set 54321
prior to adjournment.
Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc. Reproduced
by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com
18_962384 ch12.qxd 2/3/05 12:18 AM Page 340