The effect of digitizing and gamifying quizzing in classrooms

TheEffectofDigitizingandGamifyingQuizzinginClassrooms
AlfIngeWang,MengZhu,andRuneSætre
NorwegianUniversityofScienceandTechnology,Trondheim,Norway




Abstract:Theuseofgame-basedlearningintheclassroomhasbecomemorecommoninrecentyears.Many
game-basedlearningtoolsandplatformsarebasedonaquizconceptwherethestudentscanscorepointsif
theycanchoosethecorrectansweramongmultipleanswers.Thearticledescribesanexperimentwherethe
game-basedstudentresponsesystemKahoot!wascomparedtoatraditionalnon-gamifiedstudentresponse
system, as well as the usage of paper forms for formative assessment. The goal of the experiment was to
investigate whether gamified formative assessments improve the students’ engagement, motivation,
enjoyment, concentration, and learning. In the experiment, the three different formative assessment
tools/methods were used to review and summarize the same topic in three parallel lectures in an IT
introductorycourse.Thefirstmethodwastohavethestudentscompleteapaperquiz,andthenreviewthe
results afterwards using hand raising. The second method was to use the non-gamified student response
systemClickerwherethestudentsgavetheirresponsetoaquizthroughpolling.Thethirdmethodwastouse
the game-based student response system Kahoot!. All three lectures were taught in the exact same way,
teachingthesamesyllabusandusingthesameteacher.Theonlydifferencewasthemethodusetosummarize
thelecture.Atotalof384studentsparticipatedintheexperiment,where127subjectsdidthepaperquiz,175
used the non-gamified student response system, and 82 students using the gamified approach. The gender
distribution was 48% female students and 52% male students. Pre- and a post-test were used to assess the
learningoutcomeofthelectures,andaquestionnairewasusedtogetdataonthestudents’engagementand
motivation. The results show significant improvement in motivation, engagement, enjoyment, and
concentrationforthegamifiedapproach,butwedidnotfindsignificantlearningimprovement.

Keywords: Game-based learning, Student response system, Student engagement, Evaluation, Formative
assessment,Kahoot!

1. Introduction

Prototypes of student response systems (SRSs) have been around since the sixties (Judson 2002), and these
systemsstartedtobeusedinbiologyandchemistryteachingintheearlyseventies(BesslerandNisbet1971,
Casanova1971).ThefirstgenerationofSRSswasbasedonspecialhardwarethatallowedthestudentstogive
their answers using clickers, key-pads, handsets or zappers (Caldwell 2007). A major disadvantage with this
firstgenerationofsystemswasthattheyrequiredinvestmentinhardwaredevicesandinfrastructureaswell
as administration and maintenance of the hardware and software. The Bring Your Own Device wave has
opened up for a new generation of SRSs, where students can use their own devices to respond. After the
introduction of smart phones and tablets, easy access to wireless network access and support for HTML5,
many new SRSs and similar tools have populated the market: for example Socrative (Coca and Slisko 2013),
Quizlet (Gruenstein, McGraw et al. 2009), Poll Everywhere (Sellar 2011), iClicker (Lucas 2009), and Learning
Catalytics(Schell,Lukoffetal.2013).TheuseofHTML5web-technologymakesitpossibletousethesesystems
without installing any applications, and opens up for a range of new ways of interacting in the classroom.
Kahoot! is a game-based SRS (GSRS) that was introduced to the public in the fall 2013. The main difference
between a GSRS and a SRS is that the game-based version focuses more on engaging and motivating the
students through attractive graphical user-interfaces and audio, as well by gamifying the whole student
responseexperience.Thegamificationisdonebytemporarilytransformingtheclassroomintoagameshowas
shown on TV, where the teacher plays the role of a game show host and the students are the competitors.
Well-designedvideogamesaresaidtobelearningmachines(Gee2003),andtheyhavethepotentialtogetthe
playerssomotivatedandengagedthattheyarenotawarethatlearningisactuallyhappening.InK-12,games
have been found to be beneficial for academic achievement, motivation and classroom dynamics (Rosas,
Nussbaum et al. 2003). Games have also been found to have a similar effect in higher education (Sharples
2000).Previousresearchindicatesthatgamescanbemadeanintegratedpartoftraditionalclassroomlectures
toimprovelearning,motivationandengagement(CarverJr,Howardetal.1999,Carnevale2005,Wang,Øfsdal
etal.2007,Wang,Øfsdaletal.2008,Wu,Wangetal.2011).

ThisarticlepresentsanexperimentwheretheGSRSKahoot!wascomparedtoaSRS,aswellastotheusageof
paper forms for formative assessment. Section 2 presents material and methods including related work, a
description of the formative assessment tools used in the experiment, the data sources used, the research
context and participants of the experiment, the experiment procedures, and the data analysis. Section 3
presentstheresultsfromtheexperiment.Section4concludesthearticle.

2. RelatedWork
TherehavebeenmanyexperimentsandstudiesconductedonSRSs,andaliteraturestudyfrom2007reports
that such systems have been found to have a positive effect on student exam-performance, and that they
createamorepositiveandactiveatmosphereinclassrooms(Caldwell2007).Morespecifically,studentsusing
SRSsweretwiceaslikelytoworkonaproblempresentedduringclass (Cutts,Kennedyetal.2004),student
attendanceroseto80-90%(BurnsteinandLederman2001),andabout88%ofthestudentseither“frequently”
or “always” enjoyed using the SRS in class (Caldwell 2007) Further, Caldwell’s survey summarized some
commonusesofclickerquestionsfoundintheliterature:toincreaseormanageinteraction,toassessstudent
preparationandensureaccountability,tofindoutmoreaboutstudents,forformativeassessment,forquizzes
ortests,todopracticeproblems,toguidethinkingrevieworteach,toconductexperiments,tomakelectures
fun, to differentiate instruction, and to prompt discussion. Another study summarizes similar findings of
benefitsofSRSsinthethreeareasClassroomenvironment,Learning,andAssessment(KayandLeSage2009).
SRSs were found to improve attendance, provide more focused students, provide anonymous student
participation, improved student engagement, increase learning performance, improved teaching, and
generally improve interaction between teacher and students. The benefits listed in these surveys are all
benefitswehaveexperiencedfromusinggame-basedSRSinclassroomsaswell.

There are several studies on the effects of educational games related to learning outcome and increased
motivation,andwewillpresentsomeofthesestudieshere.Onestudyexploredtheimpactofusingagame
named Supercharged! on pre-service teachers’ understanding of electromagnetic concepts compared to
studentswhoconductedamoretraditionalinquiryorientedinvestigationofthesameconcept(Andersonand

Barnett 2011). The effectiveness was investigated through an experiment that used both qualitative and
quantitative data that included pre- and post-scores, student notebooks, video recordings of laboratory
activities and observations. The results of this study showed that the group using the video games
outperformed the group that did not use the video game in terms of learning outcomes (statistically
significant). In another study, the video game relative asteroids was used to teach physics (Carr and
Bossomaier 2011). A pre- and post-test with 8 questions was used to measure the learning outcome. The
resultsshowedimprovementofthetestscoresfornewlearners.Thestudentsalsofoundthatlearningphysics
through a game was motivating and engaging, and it was effective at improving their comprehension of
physics. In a chemistry class, an experiment was conducted to compare students’ achievement and attitude
fromtraditionalvs.game-basedteachingmethods(Tüysüz2009).Thestatisticallysignificantconclusionofthe
studywasthatgame-basedlearningincreasedthestudents’achievementinchemistrycomparedtotraditional
learning methods. The study also showed that game-based learning increased the students’ interest in the
course: they enjoyed the course more, and were more focused and engaged on the subject being taught.
Similarresultsonimprovedlearningoutcomewerealsofoundforusingaweb-basedadventuregametoteach
neuroscience (Miller, Schweingruber et al. 2002), in an experiment comparing teaching computer memory
knowledge with a game vs. a non-game application (Papastergiou 2009), and for using a mobile game to
engagestudentsinarithmeticpractices(Liao,Chenetal.2011).Inthecomputermemorystudy,thestudents
thatusedagamefoundthislearningapproachsignificantlymoreappealingandeducationalfruitfulthanthe
students with the non-game application (Papastergiou 2009). The same students also found their learning
approach more engaging, more effective, more active and relaxed compared to the students that used the
non-gameapproach.Therearealsostudiesthatshowthatintroducinggamesintotheclassroomnotalways
producepositiveresultsandcanresultincomplainingstudentsandlackofmotivation(Squire2005).

Kahoot! represents a new generation of student-response systems that has a main focus on student
motivationandengagementthroughgamification.ThetoolisaresultoftheresearchprojectLectureQuizthat
started in 2006 (Wang, Øfsdal et al. 2007), where results from experimentation of early prototypes showed
positive results in terms of increased engagement, motivation and perceived learning (Wang, Øfsdal et al.
2008,Wu,Wangetal.2011).Educationalgamescomparedtomainstreamentertainmentgamesareknownto
suffer from running on very few platforms (usually Windows PCs), too simplistic, being single player and

offline,offeringlowproductionvalue,andaretypicallymoretargetedtowardsparents,teachersandformal
learning curriculum than being fun for the students (Kirriemuir and McFarlane 2004). This is especially true
wheneducationalgamestrytocopyexistinggameconceptsandaddsomelearningontopofit.Kahoot!was
notdesignedtocopyanyexistinggame,butrathertofindagameconceptthatcouldfitaclassroomsetting
andthatcouldbealignmentwithTomMalone’stheoryofintrinsicallymotivatinginstructions(Malone1980).
Malone’s theory lists three categories that make things fun to learn: Challenge (goals with uncertain
outcomes), Fantasy (captivate through intrinsic or extrinsic fantasy), and Curiosity (sensor curiosity through
graphics and audio, and cognitive curiosity). As the game should be used in the classroom, it was also
importanttoincorporateasocialgameplay.Theresultwastodevelopagameconceptwherethefantasyis
that the classroom temporarily is changed to a game show where the teacher is the game host and the
students are the competitors. The challenge is to answer questions and compete against other players, and
thecuriosityisprovidedthroughinspiringgraphicsandaudio,aswellassolvingacognitivepuzzle.Thelackof
variety in game play is compensated bythecompetitivenature of playing against a whole class of students.
Reports from happy teachers and students all over the world give an indication that the concept works as
intended.Learninggamesarecommonlyusedtoreviewfactsusingmultiple-choicequestionssimilartowhat
is done in Kahoot!. However, such games can also be used to teach skills, judgment, behaviors, theories,
reasoning,process,procedures,creativity,language,systems,observation,andcommunicationusingvarious
approaches(Prensky2005).

3. MaterialandMethod
Thissectionpresentsthethreeassessmenttoolsused,thedatasources,theresearchcontextandparticipants,
researchprocedures,andthemethodfordataanalysis.

3.1 ResearchQuestionsandResearchApproach
Theresearchgoaloftheexperimentpresentedinthisarticlewastoinvestigatehowdigitizingandgamifying
quizzing in the classroom affects the students’ motivation, enjoyment, engagement, concentration and
learning. Specifically, this experiment investigates the impact of choice between running a quiz using pen &

paper,usingastudentresponsesystem(SRS)andusingagame-basedSRSaffectsthestudents’perceptionof
the quiz. The research method used is based on the Goal, Question Metrics (GQM) approach (Basili 1992)
wherewefirstdefinearesearchgoal(conceptuallevel),thendefineasetofresearchquestions(operational
level),andfinallydescribeasetofmetricstoanswerthedefinedresearchquestions(quantitativelevel).

3.1.1 ResearchGoalandResearchQuestions
TheresearchgoalofthisstudywasdefinedasthefollowingusingtheGQLtemplate:

Thepurposeofthisstudywastoevaluatetheeffectofchoiceofquiztoolforreviewtaughtmaterialfromthe
pointofviewofastudentinthecontextofalecture.

Thefollowingresearchquestions(RQs)weredefinedbydecomposingtheresearchgoal:
• RQ1:Howdoesthechoiceofquiztoolaffectthestudents’motivation?
• RQ2:Howdoesthechoiceofquiztoolaffectthestudents’enjoyment?
• RQ3:Howdoesthechoiceofquiztoolaffectthestudents’engagement?
• RQ4:Howdoesthechoiceofquiztoolaffectthestudents’concentration?
• RQ5:Howdoesthechoiceofquiztoolaffectthestudents’learningoutcome?

3.2 ThreeMethodsforRunningQuizzesinaClassroom
Intheexperimentpresented,threedifferentapproacheswereusedtorunaquizasapartofalecture.The
quizwasusedtoreviewwhatbeingtaughtinthelectureandconsistedof12multiple-choicequestions.The
quiz methods used were paper forms, a simple student response system named Clicker, and a game-based
student response system named Kahoot!. Figure 1 shows pictures from the three lectures of students doing
thequiz(left:paperquiz,middle:Clicker,right:Kahoot!).Thethreequizmethodswillnowbedescribedmore
indetail.

Figure1Picturesfromlecturesusingthreedifferentquizmethods

3.2.1 PaperFormandHandRaising
Thepaperquizisananalogue,well-knownandprovenapproachforrunningquizzesinaclassroom.Beforea
lecture, the teacher prepares paper forms with multiple-choice questions where students can tick of one of
twotofouranswersusingapenorapencil.Duringthelecture,theteacherhandsoutthepaperforms,and
the students answers as well as they can. The normal procedure for such quizzes is that the teacher then
collectsthepaperforms,andthestudentsgetfeedbackinthefollowinglectureonhowtheyhaveperformed.
Inourexperiment,wechangedthereviewparttobemorecompatiblewiththedigitalcounter-parts.Afterthe
students had completed their forms, the teacher reviewed their answers by going through the questions
askingstudentstoraisetheirarmsforthealternativetheyhadanswered.Inthisway,theteachergottoknow
howthestudentshadansweredandgavehimthechancetogivefeedbacktotheclass.Thepaperformsused
consistedofaquizoftwelvequestions,allwithfouralternativeanswers.

3.2.2 TheClickerStudentResponseSystem
Clickerisasimplestudentresponsesystem(SRS)allowingstudentstogivetheirresponsestoquestionsbeing
askedusingaweb-browseronanydigitaldevice.Thequestionsandanswersaretypicallyshownusinganother
toollikeKeynote,PreziorPowerPoint,andClickerisusedtocollect“votes”fromthestudents.Figure2shows
anoverviewoftheClickerSRSandthestepsneededtogetresponsesfromstudents.First,theteacherneeds
to name a classroom which will be the ID used for students to connect (Figure 2a). Second, the teacher
chooseshowthestudentscanresponsetoaquestion(Figure2b).Third,theteacherhastoshowthequestions
andanswersonthescreenusingPowerPointorsimilartoolaswellashestartsthevotingprocess(Figure2c).
In this step, the teacher can monitor how many students that have given their answers. Forth, the teacher
stopsthevoting,andadistributionofhowthestudentshavevotedisshown(Figure2d).Thedistributionof
answersisalsoshownonthestudentclient.TheClickerSRSdoesnotdirectlygivetheteacherorstudentsany
feedback on correctness of answers. It is up to the teacher to comment on correctness of the students’
responsesbasedonthegivendistributionofvotes.

Figure2OverviewoftheClickerStudentResponseSystem



3.2.3 TheKahoot!Game-basedStudentResponseSystem
Kahoot! is a game-based student response system (GSRS) launched by the teacher in a web-browser on a
laptop connected to a large screen. Unlike the Clicker SRS, Kahoot! provides a tool for creating quizzes




includingaddingpicturesandYouTubevideostothequestions.Kahoot!alsomakesitpossibletopublishand
share your own quizzes, and edit quizzes made by others. Another difference is the way Kahoot! is played.
Studentswilllogintothesystemusingagamepin(anumber)andanickname.Thegoalforthestudentsisto
answerthecorrectanswerasfastaspossibletogetasmanypointsaspossible.Figure3showshowKahoot!is
played. A question is shown on the large screen along with four or less alternatives in different colors with
associated graphical symbols. The students give their answers by choosing the color and symbol she or he
believescorrespondstothecorrectanswer.



Figure3PlayingKahoot!

Betweeneveryquestion,adistributionofhowthestudentsansweredisshownbeforeascoreboardofthefive
bestplayers.Thestudentsgetindividualfeedbackontheirquestionsintermsofcorrectness,thenumberof
points,theranking,howfarthestudentisbehindthestudentrankedabove,andthecorrectanswerifwrong
answerisgiven.AttheendofaKahoot!session,thewinner’snicknameandpointswillbeshownonthelarge
screen.Duringthequiz,Kahoot!usesaplayfulgraphicaluserinterfaceaswellasmusicandsoundstogiveita

playful and competitive atmosphere similar to a game show on TV. The students are also asked to give
feedbackonthequiztheyhaveplayedthroughgivingscoresonwhetherthequizwasfun,educational,canbe
recommendedtoothers,andhowyougenerallyfeelaboutthequiz.Finally,Kahoot!providesthefunctionality
fortheteachertodownloadtheresultsfromthequizinanExcelspreadsheet.

3.3 DataSources
Theinstrumentsforcollectingdatainourexperimentincludedadomainknowledgetestandaquestionnaire
on students’ engagement and motivation. The domain knowledge test consisted of seven multiple-choice
questionsdevelopedbyadomainexpert,andthisquestionnairewasusedbothasapre-testandapost-testto
measuretheknowledgebeforeandafterthelecture.

Themotivationquestionnairewasdevelopedtomeasurethemotivationandtheengagementofthestudents.
Thequestionnairewasadaptedfromthecoursemotivationsurvey(CMS)(Kebritchi,Hirumietal.2010)toour
research context, and integrated with relevant questions in the Motivated Strategies for Learning
Questionnaire (MSQL) (Pintrich 1991) and (Lepper, Corpus et al. 2005). The questionnaire used a five-point
Likertscalefromstronglydisagreetostronglyagree.

3.4 ResearchContextandParticipants
The experiment was performed in the IT introductory course (TDT4105) at Norwegian University of Science
andTechnology(NTNU).Thereweretworeasonsforchoosingthisparticularcoursefordoingtheexperiment.
First, the IT introductory course is a large course with many students, meaning that it would be possible to
collectdatafrommanysubjects.Second,duetothesizeofthiscourse,thesamelecturehastobetaughtin
three parallels. This means that the same teacher will teach exactly the same lecture for three parallels of
students.TheITintroductorycourseisamandatorycourseforallfirstyearstudentsattheuniversity,giving




thatthegroupsofstudentsintheexperimentshouldbefairlyuniform.Theexperimentwasconductedover
threedaysattheendofSeptember2013,andthetopicofthelecturewasonbasiccomputerknowledge.384

students participated in the experiment where the distribution of the subjects that completed the
questionnaireswas127subjectsforpaperquiz(58%femalevs.42%male),175subjectsforclickerquiz(37%
malevs.63%male),and82subjectsforKahoot!(54%femalevs.46%male).

3.5 Procedures
ThelectureintheexperimentwasconductedaccordingtoFigure4.First,theteacherintroducedthelecture
by presenting the agenda and the current topic, before the students carried out a paper pre-test on the
lecture’stopic.Second,theteachertaughtthetopicbasiccomputingusingPower-pointslides.Third,aquizon
thetopicwascarriedoutinthreevariationsforthethreeparallels(Paper,Clicker,andKahoot!).Forth,atthe
endofthelecture,thestudentshadtofillinamotivationquestionnaireaswellasdoingthepaperpost-test
(sameasthepre-test).

Figure4ExperimentProcedures



3.6 DataAnalysis
Theanswersfromthepre-andpost-testswereevaluatedtoascorefrom0to7points,whereeachcorrect
answer contributed one point to the total score. The learning outcome was computed by comparing the
difference between the post- and the pre-test scores. The Mann-Whitney test was used to compare the
differenceinthelearningoutcomesfromthedifferentquizmethods(Paper,ClickerandKahoot!).TheMannWhitney test is a nonparametric test for the significance of the difference between the distributions of two
independentsamplesofdifferencesizes.

TheKruskal-Wallistestwasrunonthedatafromthemotivationquestionnairetoinvestigatethedifferences
between the responses from the three groups Paper, Clicker, and Kahoot!. The Kruskal-Wallis test is a
nonparametric test for the significance of the differences among the distributions of in our case three
independentsamplesthathaddifferencesizes.

4. Results
This section presents the results from the controlled experiment. In the analysis we looked at differences in

students’ motivation, enjoyment, engagement, and concentration in regards to the used quiz method. This
sectionalsoreportsondifferencesinthelearningoutcome.

4.1 RQ1:EffectonMotivation
Table1showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedto
motivation.Theresultsshowthatthereisastatisticallysignificantdifferenceinthestudents’motivationfor
doingthequiz(statement1).TherewasnotmuchdifferencebetweenthePaperandtheClickerquizwhere
halfofthestudentshadinternalmotivationfordoingthequiz.ForKahoot!almostfouroutoffivestudents
hadaninternalmotivation,andonly5%didthequizonlybecausetheteachertoldthemto.However,there
was no significant difference related to the motivation of the lecture as a whole (statement 2), nor the




motivationforlearningmoreaboutthetopic(statement3).Thereisatendencythatthestudentsweremore
motivatedforalectureusinggame-basedquizzes(statement2),butnotstatisticallysignificant.

Table1ResultsonMotivation
Statement
Group
Disagree Neutral Agree H
P
1. I did the quiz only because the Paper
50%
25%
25%
teachertoldmeto
Clicker 56%
29%
15%

14.2 0.0008
Kahoot! 77%
18%
5%
2.Idonotthinkthislecturewasworth Paper
56%
29%
15%
mytimeandeffort
Clicker 51%
32%
17%
2.79 0.2478
Kahoot! 63%
24%
12%
3. I enjoyed the quiz so much that I Paper
37%
43%
20%
wanttoknowmoreaboutthetopic
Clicker 34%
39%
27%
1.4
0.4966
Kahoot! 35%
49%
16%

4.2 RQ2:EffectonEnjoyment
Table2showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedto
enjoyment.Thetableshowsthatthereisastatisticallysignificantdifferenceregardingtheengagementofthe
quizitself(statement5)wherethegame-basedapproachwasclearlyperceivedasmoreengagingcomparedto
the paper quiz and the Clicker student response system. No statistically significant differences were found
related to completing the quiz (statements 4 and 6), although there is a tendency that the game-based
approach to a larger degree gave students a satisfaction on completion compared to the two other
approaches.

Table2ResultsonEnjoyment
Statement
Group
Disagree Neutral Agree H
P
4. To complete the quiz gave me Paper
35%
33%
32%
satisfaction
Clicker 28%
41%
31%
3.1
0.2122
Kahoot! 22%
38%
40%
5. The quiz was boring and not Paper
60%
34%

6%
engaging
Clicker 63%
26%
10%
9.59 0.0083
Kahoot! 84%
12%
4%
6. It gave me satisfaction to complete Paper
18%
41%
41%
thequizinasatisfactoryway
Clicker 16%
40%
44%
1.65 0.4382
Kahoot! 15%
34%
51%

4.3 RQ3:EffectonEngagement
Table3showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedto
engagement.Thetableshowsthattherewasastatisticallysignificantdifferenceinhowthestudentsperceived
the engagement of the quiz for both statements. For Kahoot! over half of the students expressed that the
most satisfactory in the lecture was to do well on the quiz. There was even a larger difference for the
statementonwhetherthequizcauseincreasepulsewhereabouthalfofthestudentsdoingthegame-based
quizclaimedtohaveincreasedpulse,comparedtoaround10%fortheothertwoapproaches.

Table3ResultsonEngagement
Statement
Group
Disagree Neutral Agree H
P
7.Todowellonthequizwasthemost Paper
38%
40%
22%
satisfactoryinthelecture
Clicker 35%
37%
27%
15.91 0.0004
Kahoot! 22%
26%
52%
8. I felt increased pulse when I Paper
65%
22%
13%
answeredquestionsinthequiz
Clicker 66%
26%
8%
31.78 <0.0001
Kahoot! 34%
20%
46%

4.4 RQ4:EffectonConcentration
Table4showsthedescriptivestatisticsandtheresultsfromtheKruskal-Wallistestforstatementsrelatedto
concentration. The table shows that the statistically significant difference for students doing a game-based
quiztoalargedegreewishedtodobetteronthequizthantheirfellowstudents(statement11).Almost70%
ofthestudentsdoingtheKahoot!quizhadahigherwishtodowellonthequiz,comparedtoaround40%for
the two other approaches. On the statement directly related to concentration on doing the quiz, there is a
tendencythatstudentsdoingtheClickerquizconcentratedmorethanstudentsdoingthepaperquiz,andthe
studentsdoingtheKahoot!quizconcentratedmorethanthestudentsdoingtheClickerquiz.Interestinglyon
thequestionregardingworkingindependently(statement10),thereisatendencythatthestudentsusingthe
studentresponsesystemstoalargerdegreewantedtoworkontheirowncomparedtothosedoingthepaper
quiz.

Table4ResultsonConcentration
Statement
Group
Disagree Neutral Agree H
P
9. I concentrated on the quiz to get Paper
8%
31%
61%
correctanswer
Clicker 7%
22%
71%
4.59 0.1008

Kahoot! 6%
16%
78%
10. I wanted to answer quiz without Paper
20%
20%
60%
helpfromothers
Clicker 10%
17%
73%
5.19 0.0746
Kahoot! 9%
20%
72%
11. I wished to do better on the quiz Paper
24%
39%
38%
thanmostotherstudentsintheclass
Clicker 16%
41%
43%
20.66 <0.0001
Kahoot! 4%
28%
68%

4.5 RQ5:LearningOutcome
WewereableonlytocomparethelearningoutcomefromthetwoparallellectureswherePaperandKahoot!

wereusedastheresultsfromthepost-testintheClickerquizlecturewereincompleteduetolackoftimeat
theendofthelecture.

ResultsfromtheMann-WhineytestalongwiththedescriptivestatisticsareshowninTable6.TheMin,Max,
MeanandMedianshowsimprovementfrompre-testtopost-testinnumbercorrectanswersinthetest.There
isatendencyforahighermeanvalueforthelecturewithKahoot!quizcomparedtothelecturewiththePaper
quiz,butthedifferenceisnotstatisticallysignificant.

Table5LearningOutcomefromPaperQuizvs.Kahoot!quiz
Treatment N
Min
Max
Mean Median UA
Z
P
Paper
127
0
7
3.669 4
5655.5 -1.05
0.1469
Kahoot!
82
1
6
3.817 4

5. Conclusion
In this article, we have presented an experiment to investigate how the usage of quizzes in review-lectures

affects motivation (RQ1), enjoyment (RQ2), engagement (RQ3), motivation (RQ4), and the learning outcome
(RQ5). Our experiment revealed that students using a game-based student response system compared to
paper forms and a simple non-game-based student response system were more engaged, motivated and
concentrated, and enjoyed it more. The statistically significant differences were only related to activity of
doing the quiz itself, and not the lecture in general. Regarding the learning outcome, the results from the
experiment did not show any statistically significant differences between the quiz methods (only paper and
Kahoot! was tested). In future studies, we will investigate more thoroughly whether the learning outcome
variesbythemethodorbythequiz-toolusedforalongerperiodoftime.







References
Anderson,J.andM.Barnett(2011)."Usingvideogamestosupportpre-serviceelementaryteacherslearningof
basicphysicsprinciples."JournalofScienceEducationandTechnology20(4):347-362.


Basili, V. R. (1992). Software modeling and measurement: the Goal/Question/Metric paradigm, University of
MarylandforAdvancedComputerStudies.


Bessler,W.C.andJ.J.Nisbet(1971)."Theuseofanelectronicresponsesysteminteachingbiology."Science
Education55(3):275-284.


Burnstein,R.A.andL.M.Lederman(2001)."Usingwirelesskeypadsinlectureclasses."ThePhysicsTeacher
39(1):8-11.

Caldwell, J. E. (2007). "Clickers in the large classroom: Current research and best-practice tips." CBE-Life
SciencesEducation6(1):9-20.


Carnevale, D. (2005). "Run a class like a game show:‘Clickers’ keep students involved." Chronicle of Higher
Education51(42):B3.


Carr, D. and T. Bossomaier (2011). "Relativity in a rock field: A study of physics learning with a computer
game."AustralasianJournalofEducationalTechnology27(6):1042-1067.


CarverJr,C.A.,etal.(1999)."Enhancingstudentlearningthroughhypermediacoursewareandincorporation
ofstudentlearningstyles."Education,IEEETransactionson42(1):33-38.


Casanova,J.(1971)."Aninstructionalexperimentinorganicchemistry.Theuseofastudentresponsesystem."
JournalofChemicalEducation48(7):453.


Coca,D.M.andJ.Slisko(2013)."SoftwareSocrativeandSmartphonesasToolsForImplementationofBasic
Processes of Active Physics Learning in Classroom: An Initial Feasibility Study With Prospective Teachers."
EuropeanJournalofPhysicsEducation4(2).


Cutts,Q.I.,etal.(2004).MaximisingDialogueinLecturesusingGroupResponseSystems.CATE.

Gee,J.P.(2003)."Whatvideogameshavetoteachusaboutlearningandliteracy."Comput.Entertain.1(1):
20-20.


Gruenstein, A., et al. (2009). A self-transcribing speech corpus: collecting continuous speech with an online
educationalgame.SLaTEWorkshop.


Judson, E. (2002). "Learning from past and present: Electronic response systems in college lecture halls."
JournalofComputersinMathematicsandScienceTeaching21(2):167-181.


Kay,R.H.andA.LeSage(2009)."Examiningthebenefitsandchallengesofusingaudienceresponsesystems:A
reviewoftheliterature."Computers&Education53(3):819-827.


Kebritchi, M., et al. (2010). "The effects of modern mathematics computer games on mathematics
achievementandclassmotivation."Computers&Education55(2):427-443.


Kirriemuir,J.andA.McFarlane(2004)."LiteratureReviewinGamesandLearning."


Lepper,M.R.,etal.(2005)."Intrinsicandextrinsicmotivationalorientationsintheclassroom:Agedifferences
andacademiccorrelates."Journalofeducationalpsychology97(2):184.


Liao, C. C., et al. (2011). "My-Mini-Pet: a handheld pet-nurturing game to engage students in arithmetic
practices."JournalofComputerAssistedLearning27(1):76-89.

Lucas, A. (2009). "Using peer instruction and i-clickers to enhance student participation in calculus." Primus
19(3):219-231.


Malone, T. W. (1980). What Makes Things Fun to Learn? Heuristics for designing Instructional Computer
Games. The 3rd ACM SIGSMALL symposium and the first SIGPC symposium on Small systems. Palo Alto,
California,UnitedStates,ACMPress.


Miller,L.,etal.(2002)."Teachingneurosciencethroughwebadventures:adolescentsreconstructthehistory
andscienceofopioids."TheNeuroscientist8(1):16-21.


Papastergiou,M.(2009)."DigitalGame-BasedLearninginhighschoolComputerScienceeducation:Impacton
educationaleffectivenessandstudentmotivation."Computers&Education52(1):1-12.


Pintrich,P.R.(1991)."AmanualfortheuseoftheMotivatedStrategiesforLearningQuestionnaire(MSLQ)."


Prensky, M. (2005). "Computer games and learning: Digital game-based learning." Handbook of computer
gamestudies18:97-122.


Rosas, R., et al. (2003). "Beyond Nintendo: design and assessment of educational video games for first and

secondgradestudents."ComputerEducation40(1):71-94.


Schell, J., et al. (2013). "Catalyzing learner engagement using cutting-edge classroom response systems in
highereducation."Cutting-edgeTechnologiesinHigherEducation6:233-261.


Sellar,M.(2011)."Polleverywhere."TheCharlestonAdvisor12(3):57-60.


Sharples,M.(2000)."Thedesignofpersonalmobiletechnologiesforlifelonglearning."Comput.Educ.34(3-4):
177-193.


Squire, K. (2005). "Changing the game: What happens when video games enter the classroom." Innovate:
Journalofonlineeducation1(6).


Tüysüz, C. (2009). "Effect of the computer based game on pre-service teachers’ achievement, attitudes,
metacognitionandmotivationinchemistry."SciResEssays4(8):780-790.


Wang,A.I.,etal.(2007).LectureQuiz-AMobileGameConceptforLectures.IASTEDInternationalConference
onSoftwareEngineeringandApplication(SEA2007).Cambridge,MA,USA,ActaPress:6.


Wang,A.I.,etal.(2008).AnEvaluationofaMobileGameConceptforLectures.Proceedingsofthe200821st
ConferenceonSoftwareEngineeringEducationandTraining-Volume00,IEEEComputerSociety.

Wu,B.,etal.(2011).ImprovementofaLectureGameConcept-ImplementingLectureQuiz2.0.Proceedingsof
the3rdInternationalConferenceonComputerSupportedEducation.