PowerPoint presentation

Building University-Industry Learning and Development
through Innovation and Technology
VNU-HCM Center for Educational Accrediation

Effective Assessment and Evaluation of Student
Outcome Attainment
May 16, 2022


Presenters

Thai Tran
Program Manager for Accreditation
and Quality Assurance
Arizona State University

Ha Mai
Accreditation and
Quality Assurance Specialist
Arizona State University


Dr. Scott Danielson
Associate Professor
Ira A. Fulton Schools of Engineering
Arizona State University

The
Great
Master

Workshop
Coordinator
Chau Kha
BUILD-IT Coordinator
Arizona State University

The most important
person


Workshop Goals
Participants will
▪ review new AUN-QA requirements related expected learning
outcomes (course and program level),
▪ review basics of “measuring” student attainment of program level
expected learning outcomes (AUN-QA/ABET),
▪ practice developing performance indicators for selected student
outcomes,
▪ link performance indicators (PIs) to course learning outcomes,

▪ explore a specific example of assessing a PI,
▪ explore a specific example of SO assessment plan, and
▪ explore the evaluation of the assessment data for improvement.


Morning Session – May 16, 2022

Agenda

8:35 - 9:15 am

Continuous Improvement System Requirements &
AUN-QA Assessment at Program Level Version 4
Requirement Review

9:15 - 9:30 am

Selection of Program Student Outcomes to be
Assessed

9:30 - 10:00 am

Performance Indicators

10:00 - 11:25 am

Performance Indicators and Course Learning
Outcomes

11:25 - 11:30 am

Application: Performance Indicator Development

Afternoon Session – May 16, 2022
1:30 - 2:00 pm

Review of PI Development

2:00 - 2:20 pm

Assessment Planning Worksheet

2:20 - 3:20 pm

Development of the Planning Worksheet

3:20 - 4:30 pm

Evaluation of Assessment Data

4:30 pm

Wrap-up & Thank you!


Morning Session

Continuous
Improvement System
Requirements &
Performance Indicator
Development
May 16, 2022


HM

• Expected learning outcome
assessment is becoming/has become

an international standard of quality.

Workshop
Premises

• In an era of accountability and
transparency, continuous quality
improvement is not going away.
• It is important for us to define
anticipated student learning before
someone else does it for us.
• Because we are going to be
mandated to provide the evidence, it is
critical for us to develop assessment
processes that are consistent with our
institutional values and honor faculty
priorities.


Key Principle

Keep the process simple
and manageable!


In a Continuous Quality Improvement (CQI)
process, there are certain requirements for a
robust CQI system that assures quality.

STUDENT OUTCOMES

must be defined in measurable terms (performance
indicators).

Processes

(e.g., curriculum, co-curriculum) need to be examined and
aligned to understand how they contribute to the desired
outcomes.

Data

should be efficiently and systematically collected to see if
the performance indicators that define the outcomes are
being met.


PLAN-DO-CHECK-ACT FOR CONTINUOUS QUALITY IMPROVEMENT
PDCA is an iterative four-step method used in business for continual
improvement of processes and products (also known as the Deming circle).

Plan
In planning phase, the current
process is considered and a
plan developed to improve it.
Thinking about process
“outputs” as a way to help
develop the improvement plan.

Act

If the check phase shows there
was improvement, the new
process becomes the baseline
or standard practice (new
standards are thus said to be
enACTed).

Do
In the do phase, the plan is
implemented. Make the
changes!

Iterative
PDCA

Check (or Study)
The data and results from the
“do phase” are evaluated. Data
are compared to desired
outcomes of the improvement
plan.


Continuous Quality Improvements (CQI) Process
CQI process includes a clear understanding of:
 Mission (your purpose)
 Constituents (your customers)
 Objectives (what one is trying to achieve)

 Outcomes (learning that takes place to meet objectives)

 Processes (internal practices to achieve the outcomes)
 Assessment (Facts/data collection)

 Evaluation (interpretation of facts/data)
 Action (feedback to support decision making, change, improvement)


Quality Improvement of Student Learning
The Constructive Alignment: How well are we doing?

Institutional support

Establish &
Review Program
Educational
Objectives

Define/Refine
Student
Outcomes

Facility

Faculty

Student advising

Curriculum

Design and

Conduct
Assessments

Evaluate
Assessment
Findings

Customers

What students learn

How students learn

Use Results for
Decision Making


What is
Learning
Outcome?

A learning outcome is a concise
description of what students will learn
(knowledge, skills and attitude/
behaviors) and how that learning will
be measured/assessed upon
completion of a learning experience or
sequence of learning experiences.
Clearly articulated learning outcomes
make designing curriculum,

measuring/assessing student learning
progress, and facilitating learning
activities easier and more effective.

Learning outcomes can also help
students guide their learning and
develop effective study strategies.
Adapted from Stanford University-USA


AUN-QA uses “expected learning
outcomes” (ELOs) as related to BOTH
courses and a program. ABET calls these
“student outcomes” (SOs).

Student
Outcomes

Programs usually specified their program
level SOs (e.g. as required by AUN-QA and
ABET).
We will use this definition: Program student
outcomes (SOs) describe what students
are expected to know and be able to do by
the time of graduation.
SOs are the broad knowledge, skills, and
attitudes/behaviors students acquire by
time of graduation.
Graduation

School
Student Outcomes

Work
Program Educational Objectives

What students are expected to know and be
able to do by the time of graduation

What graduates are expected to attain within
a few years after graduation


“Effective” SOs
What is an “effective” SO?

Effective

Consider the definition of
effective as an adjective (in
English language): “working well
and producing the desired
result/goal.”**

Measurable

Learning
Outcomes

Effective can also mean

something that is “efficient,”
which means “achieving maximum
productivity with minimum wasted
effort or expense.”**
Now apply these definitions to
SOs.
**From Oxford and Macmillan online dictionaries


Characteristics of “Effective” SOs
When considering program expected learning outcomes, consider the
following recommendations:
be specific and well-defined

be realistic (attainable by students at time of graduation)
use active verbs in future tense, e.g., “will demonstrate”

SOs
should

reflect program goals for graduates, not specific classes
should be supported/enabled by the program’s curriculum (the
courses student have to take)

not “bundle” different skills, e.g., “design structure and
communicate that design to the public”
focus on what student will do, not how they learn to do it

consist of between 5–10 statements
Adapted from Stanford University-USA

INSTITUTIONAL LEVEL
(V2.0)
Criterion 2.16 Student
Assessment
Effective

AUNQA
Measurable

Learning
Outcomes

ABET

PROGRAM LEVEL (V4.0)
Criterion 2.1 Expected
Learning Outcomes
Criterion 2.4: Student
Assessment
PROGRAM LEVEL
Criterion 3: Student
Outcomes
Criterion 4: Continuous
Improvement


There are a number of reasons for measuring and assessing student
outcomes at the program level

Curriculum
Evaluation

Student Success
To monitor student success across the
program, identify gaps, and suggest
initiatives to enhance the educational
experience for all students.

To confirm that the actual knowledge student
acquire by completing the requirements of
the program is consistent with the intended
goals of the curriculum.

Evaluating
Alumni Success
To ensure that graduates demonstrate
competencies such as teamwork and
communication skills that employers in
all fields consistently identify as
prerequisites for success in a rapidly
changing economic environment.

Measuring
Student
Outcomes

Measuring
Effectiveness
To gather and aggregate evidence

across the program – not just in
individual courses – to measure
effectiveness and guide efforts to
continuously improve the quality of the
program.

Program Review
& Accreditation

Accountability
To respond to the increasing pressure from the public
and our constituents to be accountable and to
demonstrate the value students receive from
participating in our programs and services.

To meet regional and international accreditation
requirements (AUN-QA/ABET).

Adapted from The University of North Carolina-USA


AUN-QA
Guidelines
(Institutional Level)

AUN-QA criteria for IQA (version 2)
requires measuring achievement of all
program expected student learning
outcomes.

Assessment in AUN-QA guideline has
typically meant “grading” or assigning
marks to the student work.

AUN-QA
Guidelines
(Program Level)

However, the Guide to AUN-QA
Assessment at Programme Level
Version 4.0, dated August 2020, has
broadened the meaning of
assessment.*
Also, these new guidelines seem to be
clearer about the differences between
program-level expected learning
outcomes and course learning
outcomes.*

Dr. Scott Danielson’s opinion


Links to AUN-QA Criteria
(using AUN-QA 4.0, 2020, Guidelines)
Criterion 1. Expected Learning Outcomes

“Assessment” is needed to show achievement!

Links to AUN-QA Criteria
(using AUN-QA 4.0, 2020, Guidelines)
Criterion 4. Student Assessment
Unfortunately, this criterion still mixes grading or giving marks with
“assessment” of program student outcomes*

Sub-criteria 4.4 seem focused on grading or giving marks to student.
Sub-criteria 4.5 appears to be focused on programme student outcomes.
Dr. Scott Danielson’s opinion


Links to AUN-QA Criteria
(using AUN-QA 4.0, 2020, Guidelines)
Criterion 8. Output and Outcomes

The word “achievement” as shown in AUN-QA sub-criteria 4.5 and 8.4 has
a very similar meaning as the word “attainment” used in the ABET criteria.
Sub-criteria 4.5 and 8.4 require methods to determine data showing
attainment/achievement of the program expected learning outcomes (ABET
terminology is student outcomes).


ABET