Design Thinking and AI: A New Frontier for Designing Human-Centered AI Solutions

<span class="text_page_counter">Trang 1</span><div class="page_container" data-page="1">

New Frontier for DesigningHuman-Centered AI Solutions

<small>by Martin B€ockle PhD, Iana Kouris PhD</small>

<small>Martin B€ockleLead StrategicDesigner, BostonConsulting Group,BCGX, London,United Kingdom</small>

<small>Iana KourisManaging Director,Boston ConsultingGroup, BCGX, London,United Kingdom</small>

In recent years, design thinking (DT) has become a pervasive innovationapproach in the managerial community as a structured way of solvingchallenging problems through a human-centered perspective. The approachfosters lateral thinking and has a huge impact on the organizational cultureand the core elements of the innovation process. At the same time, thenumber of organizations trying to solve problems through the application ofartificial intelligence (AI) is constantly growing, although more than 80% ofAI projects never reach deployment due to the wrong strategic approach,poor data quality, or a lack of AI awareness among employees—and thosethat do remain below profitability expectations. The aim of this paper istherefore to connect the AI and DT communities by proposing a firstframework supporting the development of AI solutions in a human-centeredway. The contribution of this paper is thus twofold: First, we investigateways in which design thinking adds real value along the design process ofintelligent solutions, thereby enriching the present body of design

management literature. Second, we propose a first framework by linking coreelements of both worlds to support strategic designers and data engineers indesigning such applications. Finally, we propose a research agenda that canserve as a basis for future research directions in order to advance

contributions at the intersection of the two fields.

Key words: artificial intelligence, design thinking, human-centered design

I

^{n recent years, design thinking (DT) has attracted significant interest}and increasing attention from both industry and academia as a novelproblem-solving methodology. The community is constantly growing,

</div><span class="text_page_counter">Trang 2</span><div class="page_container" data-page="2">

recently by management scholars(Liedtka, 2014; Verganti, 2020b)who are focusing on design methodsand how they are applied toinnovation challenges, since the waysand forms in which design canimpact the first phase of newlyemerging technologies is an

unexplored field (Verganti, 2020b).Although the term design thinkinghas been widely discussed inprominent product managementjournals (Di Benedetto, 2012; Seideland Fixson,2013) such as theAcademy of Management(Dunne andMartin, 2012) as well as in businesspublications like The Economist,Harvard Business Review(Liedtka 2018), The Wall StreetJournal(e.g., Wladawsky-Berger,2015), or The New York

Times(e.g., Parker-Pope, 2016), agenerally accepted definition is stilllacking.

This is not surprising, sinceeven the research stream aroundproduct design, which consists of abody of literature with a higherlevel of maturity, still suffers from alarge number of differing definitionsintroduced over time (Liedtka,2014). Nevertheless, one of themost widely known definitions,introduced by Tim Brown, theCEO of the innovation consultingfirm IDEO, which focuses onproduct development and, morerecently, also on service and strategydesign, defines design thinking as a“human-centered approach toinnovation that draws from thedesigner’s toolkit to integrate theneeds of people, the possibilities

of technology, and therequirements for businesssuccess” (Brown, 2014).

Generally, this definitionhighlights the need to connectdesigners, principles, approaches,methods, and tools in a problem-solving context, while describingthe three spaces of innovation:First, inspiration, which refers tothe actual problem and opportunitythat aims to motivate the searchfor solutions; second, ideation, theprocess through which ideas aredeveloped and tested; and third,implementation, the phase in whichthe level of maturity is reached andthe developed artifact turns itsfocus towards the market (Brownand Katz, 2011; Liedtka, 2014).Brown and Katz (2011) alsomention that for any organization,these skills need to be dispersedand moved up to the executivelevel, to support but also informthe strategic decision-makingprocess. Over time, differentmodels of design thinking havebeen proposed, for instance byIDEO or the Stanford DesignSchool (e.g., empathize, define,ideate, prototype, and test), whichrepresent iterative cycles ofexploration, idea generation,prototyping, and experimentation.Moreover, Kimbell (2011)introduces different ways ofdescribing and clustering designthinking, for instance throughcognitive styles, theory of design,or as an organizational resource.

These characteristics of designthinking also relate to other

emergent approaches like agileproduct development, in whichiteration and experimentationrepresent key components, or thelean startup approach, whichhighlights rapid iteration and thedevelopment of a minimum viableproduct (MVP) to validate andtest the product with end users atan early point (Micheli et al.,2018).

Similarly situated, the process ofdesigning artificial intelligence (AI)solutions highlights the need for amore human-centered thoughtprocess, since more than 80% of AIprojects never reach deployment, duethe wrong strategic approach, poordata quality, or a lack of AIawareness among employees—andthose that do remain belowprofitability expectations(Chawla, 2020). Although thecurrent body of literature does notprovide a widely accepted process fordesigning human-centered artificialintelligence (HCAI) solutions, webelieve that the intersection ofdesign thinking and AI provides avaluable contribution to bothcommunities. Consequently, we aimto answer the following researchquestions:

RQ1: How can design thinking beenabled towards a human-centeredAI approach?

What are the main challenges for thedevelopment of intelligent solutionswithin organizations, and how canthey be tackled through HCAIdesign practices?

</div><span class="text_page_counter">Trang 3</span><div class="page_container" data-page="3">

Thus, the aim of this paper is toconnect the AI and DT communitiesby proposing a first framework thatinforms the design process ofintelligent solutions in a morehuman-centered way. The mainfocus is placed on challenges like datacollection, the intersection of userneeds and AI capabilities, and theexplanation (xAI) of results to keystakeholder groups. The contributionof this paper is thus twofold: First,we investigate ways in which designthinking practices can be enabled andshifted towards a more human-centered AI approach along thedesign process of intelligent solutionsand thereby enrich the present bodyof design management literature.Design methods and skills arementioned by Borja de Mozota andWolff (2019) as criteria of designmanagement, which leans moretowards co-creation with otherexperts in project teams (e.g., datascientists) when designing intelligentsolutions. Furthermore, there is adiscussion around the role andpotential of applying design thinkingin user-centered AI scenarios, acrucial element of design

management tools (Borja de Mozotaand Wolff,2019). Second, wepropose a first framework linkingcore elements of the two worlds tosupport strategic designers and dataengineers in designing such

applications and overcomingchallenges within organizations.Finally, we propose a research agendathat can serve as a basis for futureresearch directions in order toadvance contributions at the

intersection of the two fields withinthe field of design management.

<small>Research background andrelated work</small>

<small>Human-Centered ArtificialIntelligence</small>

Many different research streams arerelated to the domain of human-centered AI. First, ethicallyresponsible AI (Xu, 2009), whichhighlights the objective of avoidingdiscrimination and achieving fairnessand transparency of the intendedsolution. Second, the well-knownresearch stream of explainable AI(xAI) aims to design explainable,usable, and useful AI solutions(Xu, 2009; Riedl, 2019), since theseaspects have not been considered inthe past due the strong technology-driven focus. The application of xAIpractices is becoming more relevantas AI is increasingly applied indifferent end user applications, whilemost end users, specifically thosewith a limited technical background,perceive intelligent systems as a blackbox and view them with a low levelof trust. Consequently, the design ofAI solutions needs to consider howto present the output to differentuser groups in order to increase thelevel of trust. Questions like “Whydid you succeed or fail? Why didyou do that? Why is this the result?When can I trust you?” (Xu, 2009;Riedl,2019; Preece,2018) reveal thechallenge of a one-size-fits-allapproach when interpreting theoutput of such systems. Recent work

shows the potential of classifying theexplanations of AI solutions fordifferent types of users—such asdevelopers, AI researchers, anddomain experts—by consideringtheir needs, objectives, and goals indifferent application contexts(Ribera, 2019). Furthermore, theframework developed by

Wang (2019) includes relevanttheory on human decision-making,for instance how end users shouldreason in order to inform the xAItechniques (Wang, 2019) byhighlighting a taxonomy of

questions:“What is explained?” (e.g.,data or model), “How is it

explained?” (e.g., direct/post-hoc,static/interactive), and “At whatlevel?” (e.g., local versus global)(Arya,2019). Such works show thatAI policy and governance play animportant role in the future ofintelligent solutions.

<small>Design Thinking and HCAI</small>

The current body of literaturebuilding a bridge between designthinking and AI is limited, but thefew studies that have been identifiedreveal the potential of a meaningfulintersection of the two communities,although the question of how AIimpacts design thinking practices inthe form of tools has not beenconsidered. While design thinking isdiscussed within the design

management literature as a way toaddress new problems in

organizations (Cooper et al.,2009),the link to AI has been discussed verylittle, although it has been considered

</div><span class="text_page_counter">Trang 4</span><div class="page_container" data-page="4">

as a future involvement of designers(Borja de Mozota and Wolff,2019).

Relevant work on how to look atdesign in the age of AI has recentlybeen proposed by Verganti

et al. (2020a,2020b), highlightingthat design work for AI solutionsshould not focus on the ideation ofproducts to be commercialized atscale, but rather on the design ofproblem-solving loops, described ashuman-capital-free design systemsthat replace people with computingpower, which then will develop thespecific solution and deliver it tothe end user.

The authors also mention thatAI does not undermine the basicprinciples of design thinking butrather reinforces them by overcomingpast limitations (Verganti

et al.,2020b). Intelligent solutionsare much more user-centered due thehigh level of granularity and

the provision of unique userexperiences (UX) for every singleuser of the application. A recentstudy by Iansiti and Lakhani (2020)reveals that scale, scope, and learningare highlighted as key differentiatorsin designing such solutions (Vergantiet al. 2020a), described through theterm AI factory, a software that, forinstance, runs millions of daily adauctions at Google or Baidu, ordecides about the availability of rideson Didi, Grab, Lyft, and Uber(Iansiti and Lakhani,2020). Thesealgorithms also set the prices ofproducts on platforms like Amazon.However, at their heart, they useinternal and external data forpredictions, insights, and choices:

• Scale and standard designprocesses produce solutions thattarget several users of predefinedcustomer segments or averagearchetypes, which have beenconsidered through “personas”along the design thinking process.AI capabilities remove this scalelimitation in design, sinceintelligent solutions embeddesign rules that are inherentlyuser-centered, for instance in thecase of Netflix, leveraging a richstream of data on each individualuser (Verganti et al.2020a) todevelop a specific user experience.The development of theseexperiences becomes verypowerful as the number of usersand the complexity of insightsgrows.

• Scope and abduction refers to thefact that, in traditional designpractices, products and servicesare designed for a specificindustry, with little flexibility toapply the developed solution to adifferent application context. Thedesign of AI solutions removesthese limitations by reframingdesign artefacts in differentdomains. Verganti et al. (2020b)refer to the example of reusingdeveloped experiences in Netflixfor AirBnB.

• Learning and iterations representa limitation in standard designprocesses to a certain degree,once the developed product islaunched. Input for futuredevelopments and revisedversions are informed by productusage, while in intelligent

solutions the algorithm directsthe learning strategy towards abetter experience, with significantimplications on innovation(Iansiti and Lakhani, 2020). Endusers always experience the bestsolution, which evolves over time.Consequently, in AI factories the“design-build-test “learning loopis fully automated, which meansthat a new version of the productis released once the end useraccesses the service (Vergantiet al., 2020b).

In addition to the academic bodyof knowledge, resources like GooglePAIR (2021) inform towards apractice-driven approach byproviding design guidelines fordeveloping such intelligent solutions.Similarly, the contribution ofMicrosoft (Amershi et al.,2019)provides such guidelines for human-AI interaction in the categories“initially, during interaction, whenwrong, over time.” We stronglybelieve that these practices provide astrong contribution towards ourproposed framework.

<small>Research design</small>

To answer the defined researchquestions and build on a solidbaseline, we conducted a structuredliterature review (SLR) with thekeywords “design thinking AI” and“design thinking artificial intelligence”and analyzed the existing literature atthe intersection of the two researchcommunities. The literature reviewaddresses general and specialized

</div><span class="text_page_counter">Trang 5</span><div class="page_container" data-page="5">

scholars within the domain of AI anddesign thinking who are interested innovel AI and design thinkingresearch from various domains. Inthe conceptualization phase of thereview, an initial explorative searchusing Google Scholar was conductedto conceptually identify currentapproaches and trends. Furthermore,to identify core challenges in thedevelopment of AI solutions withinorganizations, we carried outfivesemi-structured interviews withselected data scientists who

experienced the end-to-end process,starting from data collection to theactual business application.

Demographic information revealsthatfive out of five participants weremale data scientists, two of them witha senior profile and three within aleadership role in the age group 26–35. All participants had a master’sdegree and were based in France. The

semi-structured interviews werecarried out by two strategic designersfor a duration of one hour.

Based on the developed insights,we created afirst framework that wasdesigned in an iterative process byadding veins of knowledge from thedesign thinking community to guidedata scientists as well as businessprofessionals along the process ofdesigning AI solutions. The definedresearch challenges, which inform theproposed research agenda, werederived from the structure of theproposed framework.

organizations should aim to design adeployment process in whichtechnology is improved continuouslyby providing the right quality oftraining data to the developedalgorithms, while products andbusiness processes are adapted at thesame time.

Results were collected from twoworkstreams. First, from thestructured literature review, andsecond, from the semi-structuredinterview sessions. The frameworkproposed in Figure1 is organized inthree layers, where the top onefollows the design thinking modelproposed by the d.School (HPI).This layer aims to connect towards ahuman-centered design approach by

<small>FIGURE 1. Design Thinking/AI Process Framework. [Colorfigure can be viewed at wileyonlinelibrary.com]</small>

</div><span class="text_page_counter">Trang 6</span><div class="page_container" data-page="6">

proposing the second layer, a strategicHCAI design layer that highlights anAI-informed, end-to-end designprocess. The bottom layer highlightschallenges identified within

organizations that inform the layersdescribed above.

In this phase of the strategic designlayer, the framework starts with thepillar of the AI baseline, whichfocuses onfinding the intersection ofuser needs and AI strengths. Theseopportunities can be identifiedthrough a process of mapping theexisting workflow (Google PAIR).Consequently, the list of

improvements needs a decision as towhere and how AI might add uniquevalue (Google PAIR). Generally,there are tasks where AI can add realvalue, for instance by recommendingcontent to different user groups orpredicting future events, but tasksthat might require completetransparency are still a hugelimitation of the technology andshould be considered in the designprocess.

From a design thinking andproduct development perspective, theaim is to understand the target groupby analyzing their needs, goals, andobjectives through close interactionwith them. Depending on thecontext, Liedtka (2014) proposesethnographic research, a qualitativeapproach to develop an

understanding by observing andinteracting with them in their dailyenvironment. Methods include

participant observation, interviews,and customer journey mapping inwhich identified problems and painpoints are highlighted. By mappingexisting workflows, AI opportunitiescan be identified, especially those thathave not been articulated.

Quantitative approaches are helpful ifexisting data is provided to

understand historic behaviors of theend user group. The AI canvas(Maillet,2019) provides a toolbox tobetter understand the intersection ofuser needs and AI strengths, but alsoto place the outcomes of the

abovementioned methods. Generally,the aim of this tool is to find aconsensus between data scientists,designers, and key stakeholders of theintended AI efforts. The proposedcanvas includes topics like thecollaboration between human andmachine activities, benefits forhumans, and considerations andimplications, but it also covers changemanagement activities. The bestpractice, provided by Google PAIR,also emphasizes whether AI is beingused to automate a task or toimprove a person’s ability to carry outthe task themself (Google PAIR).The research activities describedabove should conclude on this topicwithin the “empathize” phase.

Organizational challenges at thebeginning of an AI initiative are notlimited to understanding thetechnological landscape or

the availability and quality of data,although it is important that keydecision makers comprehend thetechnological shift that might beneeded as part of the future solution.

Very often, challenges are linked toan absence of the AI mindset neededto develop such solutions. Forinstance, AI needs to be embedded inthe company’s strategy, vision, andpurpose in order to develop a clearperspective of where AI will drivebusiness outcomes. Furthermore, AIdevelopment needs to be treated as abusiness transformation, whereredesign is collaboratively organizedwith business owners, includingiteration and improvement, based onthe human-AI learning process.

The strategic design layer proposesthe identification and clear definitionof the intelligent behavior and whatAI is expected to do. This exercise isin line with the second designthinking phase, where the aim is todefine a clear problem statement.Guidance is provided by GooglePAIR, which suggest the definitionof the reward function used todetermine“right” versus “wrong”predictions. The design of thesefunctions aims to be a collaborativeprocess across disciplines, where UXdesigners, product designers, andengineers share their perspectives. Asimple template for defining truepositives and false negatives is helpfulfor the final definition of theintelligent behavior (e.g., our AImodel will be optimized for{precision/recall} because {userbenefit} (Google PAIR).

The ambition for value creationthrough AI needs to be clearlyarticulated from an organizational

</div><span class="text_page_counter">Trang 7</span><div class="page_container" data-page="7">

perspective. This also includes settingthe focus and identifying the desiredAI outcome at the department levelto support the corporate strategy(Ransbotham et al.,2019), forinstance in marketing (e.g., cross- andupselling, churn and retention, andnext best product and action) or thesupply chain (advanced forecasting,simulations, and route optimization).Challenging choices within the“define” phase include the definitionof performance metrics, whichrequires an understanding of how theperformance frontier may be evolvingwithin the domain in which theintended AI solution might bedeveloped. This is very challengingdue the fast-moving capabilitiesof AI.

Other well-known

organizational challenges reveal theavailability of data, which includesthe cleanliness and understanding ofthe source data and quality.

Organizations may lack, for instance,the setup of an internal data

economy, such as when datascientists hoard data with noorganizational support for ease ofpublishing. Data access is oftenlimited, orflexibility of accessmanagement may not be guaranteed,including strong data complianceguidelines that make the

development of AI solutionschallenging. Finally, individualswithin organizations may not beaware of what data is available tothem or where tofind it. In most ofthe cases, the data quality does notmeet certain thresholds such as thoseregarding completeness and accuracy.

Consequently, before starting todevelop models in the process, it isnecessary to assess the maturity ofdata capability and understand theorganizational limitations. Designthinking activities that supportorganizations within this step includethe application of the data landscapecanvas in order to understand andcategorize data assets in terms oftheir source and ownership (Wirthand Szugat, 2020).

The business process comes into playin this phase. Based on the designprinciples proposed by Google PAIR,the decision to use AI to automate atask or augment a person’s ability tocarry out the task is crucial at thisstage, since the different ideas mightconsider certain ways in which AIcan solve the problem and supportthe end users in accomplishing theirgoals (Google PAIR). Newlydeveloped ideas might focus onincreasing efficiency or reducingtedious tasks, while others mightconsider a higher level of involvementby the end users, where AI augmentstheir existing abilities. A successfulaugmentation usually increases theend user’s enjoyment of a task orincreases their responsibility andcontrol.

Thus, to tackle this challenge,previous insights from the

“empathize” phase might be helpfulin understanding the end users’motivation of their daily tasks,combined with analyzing currentbusiness processes in order to assess

where AI can create the greatestvalue. This activity is usually donethrough ideation sessions led bydesigners, where several solutions arecreated and discussed with datascientists and engineers. The

identified opportunities should createclarity on how AI will be used tocompete long-term in the economy.

In this case, the delta modelproposed by Hax and Wilde (1999)provides three different strategicoptions to define the scope of theintelligent solution. Thefirst is to bethe best product on the market,either through low cost or throughdifferentiation. The second is acustomer solution that might satisfya wider group of people, with a focuson the customer rather than on theproduct economics (Hax andWilde1999). The third is having asystem lock-in that covers all theplayers that contribute to

the creation of economic value whileproviding the widest scope of allthree options through bonding.

Furthermore, in the process ofcreating an idea andfinding the focusof the developed solution, designthinking provides a meaningfulframework for choosing and rankingdeveloped ideas. The well-knownDesirability, Viability, Feasibility(DVF) approach is a valuable tool inthe innovation process to move to amore structured level of assessmentand sophistication with the plannedbusiness model.

With a clear vision throughideation, organizations often struggleto integrate a clear data strategy thatembodies the baseline for designing

</div><span class="text_page_counter">Trang 8</span><div class="page_container" data-page="8">

and developing the intended AIsolution. In the ideation phase, theoverall data strategy needs to bestrong and have a high level ofmaturity before starting the softwaredevelopment process. This meansthat it has to enable more clarity andunderstanding of data, as well as dataproduct discoverability, for instancethrough a data marketplace. Researchshows that every AI strategy needs tobe complemented by a data strategy.Recent research shows that three outof four organizations declare AI as acore component of their

transformation plans, although only11% of them have started

implementing a solid data strategy,i.e., designing effective data platformsand processes that enable effectivemachine learning approaches (Kironand Schrage, 2019). Furtherelements of a proper data strategyinclude evaluating the data and thecollection method in terms ofwhether they are appropriate for theintended project, including

the documentation of contents anddecisions made during the datacollection process (Google PAIR).

This section emphasizes thesoftware development approacharound the intended solution. TheAI technology on the strategic layerhighlights clarity regarding theselected data strategy to follow. Oneof the most critical issues is howdata is labeled (e.g., data, time,content description) since data thatis not properly tagged comes with

certain limitations attached. Bestpractice shows that the moremetadata is available, the moreoptions there are for developing asuccessful model, which also impactsthe quality of the user experience(UX) (Google PAIR) in thesubsequent steps. For instance, forsupervised learning, one of the initialsteps is creating a labeled data set,to be divided into training andvalidation (Verganti et al., 2019).The company Netflix is using thisapproach for recommendationsbased on labeled data sets (e.g.,watched and liked movies), where alarge group of user choices can leadto effective recommendations. Inthis case, design thinking could beused to identify opportunities forincentivizing end users or trainedsubject matter experts to participatein labeling activities. The design ofthe incentive approach, the labelerexperience, diversity, as well asexisting tools need to be taken intoconsideration. Generally, approacheslike gamification have the potentialto motivate the end user to carryout certain tasks (B€ockle, 2017;B€ockle, 2018); such approaches areconnected to either intrinsic orextrinsic motivational theories.

The decision on the softwaredevelopment approach has hugeimplications and a large impact onthe business outcomes. There areseveral reasons to pivot in thedevelopment of AI solutions, forinstance when a new source of dataincreases accuracy or lowers thecomputational costs, when a newchannel offers the same service, or

when an updated version of softwaredevelopment tools (e.g., PyTorch)requires a code reassessment. Thedevelopment process thus requires ahigh level of flexibility, combinedwith an adaptive design process.Solutions developed by any

organization always deal with certainlimitations when it comes to AI. Oneof them is the lack of generalization(e.g., face recognition) or the bias ofdata to be used to train the

algorithm. Explainability is anongoing issue that is being researchedheavily under the term xAI.

Furthermore, unintended behaviors(e.g., output of a GPT3-basedchatbot or self-driving cars) need tobe taken into consideration as well.

At this stage, design thinking hashuge potential to intervene with morecreative approaches such as designingcommunication patterns that enablethe right type of information to bepresented to the right end user in theexpected format, based on the user’sneeds.

In the testing phase of the developedmodel, the strategic design layerrefers to the user experience (UX) ofAI. From early on, it is necessary toreceive qualitative feedback through adiverse group of end users.

Dashboards and customer datavisualizations, which are also part ofhuman-centered design practices,enable the UX quality of thedeveloped system to be monitored.Testing and tuning is an ongoingprocess for adjusting the ML model

</div><span class="text_page_counter">Trang 9</span><div class="page_container" data-page="9">

(Google PAIR) and needs to beconsidered carefully.

Trust plays a central role intesting the developed AI solutionswith the end user group. If thedesigned user experience does notconvey trust instantly and

consistently over time, end usersmight quickly stop using the productor service. Thus, to increase the levelof trust between the end user and thedeveloped intelligent solutions,certain design best practices havebeen proposed. These include how tomeaningfully introduce the end userto the AI system and how to setexpectations for adaptation andcreate effective mental models, whichneed to be considered in the testingscenarios.

Furthermore, guidelines showthat feedback is crucial to developingtrust in AI-enabled user interfaces, asis explaining how to review, collect,and connect implicit and explicitfeedback to inform and enhance theuser’s product experience (GooglePAIR). The Google design teamproposed further guidelines on howto define errors and failure byproviding a path forward fromfailure, since AI capabilities canchange over time. As explainability isconsidered one of the major driversfor increasing trust in these systems,there are best practices that explainhow AI systems work by connectingexplanations to the end users’interactions with the AI system. Theproposed guidelines follow a human-centered design approach and shouldbe considered in the developedtesting scenarios:

• Mental model: This set ofguidelines concerns the end users’understanding of how AI systemswork and how their interactionsaffect the interface. Generally,mental models aim to setexpectations about functionalitiesand communication limitations.• Explainability and trust: These

guidelines address how the enduser receives an appropriate levelof explanation regarding how thesystem works and the degree ofconfidence in its output. Afterdeveloping a clear mental modeland awareness of the system’soverall capabilities, theseguidelines help end users learnhow and when to trust theunderlying system.

• Feedback and control: This set ofguidelines concerns the designof feedback and controlmechanisms that provide ameaningful end user experience(UX) when suggesting

personalized content. Thesemechanisms can also be used toimprove the underlying AI modeloutput.

• Errors and graceful failures:These guidelines help identifyand diagnose AI context errorsand communicate the wayforward. Context errors includefalse starts, misunderstandings,and edge cases that cannot beforeseen within the developmentprocess. Google suggests thatthese errors should be seen asopportunities to correct the enduser’s mental model, encouragethe end user to provide feedback,

and enhance the overall learningprocess through experimentationand error resolution processes.

These best practices provide asolid baseline for designing andtesting a meaningful user experiencewhile at the same time fostering ahigh level of trust. Organizationsneed to have a clear plan for theirtesting strategy; for instance, whatdoes the plan for early testing of themodel look like? Are the users diverseenough? Which metrics might beuseful to measure whether the tuningprocess is successful? Since designthinking provides a large repositoryof methods for testing, the

application of structured solving approaches enables thequality of the developed model to beimproved.

The result of the present work aimsto support researchers and designersin developing human-centered AIapplications by applying a structureddesign thinking AI process

framework to increase the level ofmaturity and success of the intendedAI solutions. Generally, the

developed framework can be used asa tool to guide and control the designprocess of such solutions, since thereis currently no structured processthat applies patterns of human-centered design principles. To answerthe first research question, westrongly believe that the developmentprocess of AI solutions is lackingcreative approaches that actually have

</div><span class="text_page_counter">Trang 10</span><div class="page_container" data-page="10">

the potential to increase thelikelihood of success. We thereforesuggest that these approaches have astrong impact at the beginning of thedesign process, where baselineelements like the vision or mission ofthe intended solution are still unclearor very loosely defined. Furthermore,design thinking evolves its potentialin the right selection of the intelligentbehavior through the definition of aclear statement, while considering theusers’ needs at the same time. Sincethe development of such solutions isclosely linked to the user needs andthe overall organizational design,there is a need to closely connect AIwith design thinking due the natureof the structured problem-solvingapproach. Consequently, by

intertwining design thinking practiceswith phases of the AI developmentprocess, we propose thefirst

structured approach of an AI-enableddesign process by shifting thestandard design thinking approachtowards a more human-centered AIapproach. Therefore, for RQ1, webelieve that design thinking adds realvalue in thefirst two phases

(empathize, define), but also in thetesting phase where the user

experience (UX) of the proposed AIsolution needs to match the end user

needs. Regarding RQ2, the identifiedchallenges were not limited to dataquality but more importantlyincluded a lack of collaborationbetween departments includingfrictions in sharing data on theoperational level or the overallmissing AI mindset on the

organizational level. We believe thatdesign thinking is very well suited fororganizational change in the long runbut might be limited to having a largeimpact on the operational level.Therefore, we suggest the applicationof behavioral design approaches suchas gamification, which are verypowerful in enabling behavior change.A crucial element is the definition ofthe data strategy, which is often oneof the reasons why AI projects fail inthe first run. We identified severalmeaningful points of contact betweenthe design thinking approach and theAI development process and believethat this first framework serves as abaseline for further revision andimprovements.

<small>Limitations and future researchdirections</small>

The present paper is also subject toseveral limitations. First, the currentbody of literature investigating the

connection between design thinkingand AI is limited. This is also one ofthe results of our structured

literature review. Consequently, thedeveloped framework did not receivestrong input from the theoreticalperspective but rather presents afirstapproach where existing practiceshave been merged with a non-structured AI development process.Second, most of the best practiceexamples were selected from GooglePAIR, which provides a rich

repository of examples for developingsuch solutions. Third, since we onlycarried out five interviews with datascientists and AI experts, thepresented results are limited tothe organization types (in terms ofsize, structure, existing IT

architecture, etc.) where theparticipants provided their insights.Fourth, the developed framework hasnot been tested and validated in areal-world scenario and thus providesafirst approach to showcase theconnection between the existingliterature in design thinking and AI,mixed with insights from practice,highlighted through organizationalchallenges. While there are manyunexplored issues in this domain, westrongly believe that this paper makesa valuable contribution to the design

<small>TABLE 1 Proposed research challenges</small>

</div>