Abstract
Purpose
Numerous design methods are available to facilitate digital innovation processes in user interface design. Nonetheless, little guidance exists on their appropriate selection within the design process based on specific situations. Consequently, design novices with limited design knowledge face challenges when determining suitable methods. Thus, this paper aims to support design novices by guiding the situational selection of design methods.
Design/methodology/approach
Our research approach includes two phases: i) we adopted a taxonomy development method to identify dimensions of design methods by reviewing 292 potential design methods and interviewing 15 experts; ii) we conducted focus groups with 25 design novices and applied fuzzy-set qualitative comparative analysis to describe the relations between the taxonomy's dimensions.
Findings
We developed a novel taxonomy that presents a comprehensive overview of design conditions and their associated design methods in innovation processes. Thus, the taxonomy enables design novices to navigate the complexities of design methods needed to design digital innovation. We also identify configurations of these conditions that support the situational selections of design methods in digital innovation processes of user interface design.
Originality/value
The study’s contribution to the literature lies in the identification of both similarities and differences among design methods, as well as the investigation of sufficient condition configurations within the digital innovation processes of user interface design. The taxonomy helps design novices to navigate the design space by providing an overview of design conditions and the associations between methods and these conditions. By using the developed taxonomy, design novices can narrow down their options when selecting design methods for their specific situations.
Keywords
Citation
Liu, X., Werder, K., Maedche, A. and Sun, L. (2023), "Situational selections of design methods for digital innovation processes", International Journal of Innovation Science, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/IJIS-02-2023-0046
Publisher
:Emerald Publishing Limited
Copyright © 2023, Xuanhui Liu, Karl Werder, Alexander Maedche and Lingyun Sun.
License
Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode
1. Introduction
The continuous and rapid development of digital innovation (Recker et al., 2023) — for example, in game design (Werder et al., 2020), in the design of virtual worlds (Seidel et al., 2023) or in healthcare (Satwekar et al., 2023) — leads to new design-related challenges (Kohli and Melville, 2019) when translating novel concepts into valuable products (Mamasioulas et al., 2020). Digital innovation — that is, “the carrying out of new combinations of digital and physical components to produce novel products” (Yoo et al., 2010) — relies on the digital encoding of information. Digital innovation also empowers companies to effectively address the rising demand for manufacturing customized products on a large scale (Mourtzis, 2021), to provide customers with real-time and individualized product recommendations (Recker et al., 2023) and to implement product-based learning (Werder et al., 2020). Consequently, organizations improve operational best practices, product performance and resiliency (Mourtzis, 2021), and they design digital innovation that has a positive impact on the environment, society and the economy (Ghobakhloo et al., 2021).
Innovation teams are often diverse, with members stemming from multiple disciplines, making it difficult to reach consensus (Chamorro-Premuzic, 2017). For example, individuals with robotics backgrounds may seek to provide solutions in terms of mechanical properties, potentially ignoring human behavior. Yet, human behavior is essential for user interface (UI) design, where methods such as storyboarding, cognitive walkthrough and personas are offered. While innovation teams have different specialties and varying levels of prior experience, they are often design novices for their limited design expertise. Although professionals benefit from a combination of knowledge and experiences (Woolrych et al., 2011), novices are left in awe. Rather, design novices need more guidance in selecting design methods to reach a consensus.
Innovation processes have constraining conditions such as limited time and accessibility of resources that trigger experienced designers to be creative and explore new ideas and solutions (Acar et al., 2019). When exploring new ideas and solutions, experienced designers rely on their expertise to balance multiple constraining conditions and flexibly use design methods in bundles when producing innovative designs (Ahmed et al., 2003). In contrast, novice designers have limited design knowledge, which poses a challenge for them when selecting design methods while accounting for situation-specific conditions (Chen et al., 2022). Therefore, more research is needed to guide design novices in selecting design methods during digital innovation processes.
The objective of this study is to develop a taxonomy that supports design novices with the expertise to select an appropriate design method for specific UI design project situations. Through this taxonomy, design novices are expected to improve their selections and use of design methods during digital innovation processes. In a sense, the taxonomy helps design novices to navigate around existing conditions that arise from their project situations. For example, time and resource constraints may limit the applicability of certain design methods. In fact, we suggest that existing conditions affect not only the selection of a single design method but rather affect the selection of multiple design methods. Therefore, we examine their combinatorial selections to identify specific configurations that design novices prefer and formulate the following research question: How to support design novices to select design methods during digital innovation processes of UI design?
To answer this question, we first identified the dimensions of design methods by reviewing a total of 292 design methods, which we then synthesized in multiple iterations. We interviewed 15 experts to evaluate and optimize our taxonomy. Second, we identified configurations of the dimensions included in the taxonomy. We conducted focus groups with 25 design novices and applied fuzzy-set qualitative comparative analysis (fsQCA) to identify necessary and sufficient conditions (i.e., the taxonomy's dimensions) for each configuration. As a result, we present a novel taxonomy of design methods for digital innovation processes and develop recommendations for their bundling based on different project conditions.
Overall, the present study makes four main contributions. First, we develop a taxonomy by identifying similarities and differences between design methods (Cornelissen, 2017). By systematically analyzing and categorizing these methods, we provide a framework that enhances understanding and facilitates effective decision-making in design processes. Second, we delve into the prioritization of individual dimensions within the taxonomy and identify optimal configurations of dimensions. This research addresses the need for exploring the sufficient conditions within the taxonomy, responding to calls for further investigation into the beneficial combinations of conditions (Nickerson et al., 2013). Third, novices can navigate the complex landscape of design methods more effectively. We offer insights that can guide design novices in selecting the most appropriate design methods for specific projects and contexts, extending previous research on selecting methods based on classified attributes (Jiang et al., 2008). Fourth, we extend prior research on the management of digital innovation (Nambisan et al., 2017; Satwekar et al., 2023) by taking a more nuanced perspective. Our research focuses on the particular task of selecting design methods when designing digital innovation, whereas prior research on digital innovation management has often focused on the overarching business process.
The following section introduces related studies on the differences between design experts and novices, as well as conditions of selecting design methods. Section 3 describes the research approach, which includes classifying design methods by following the taxonomy development method, evaluating the taxonomy’s dimensions by using expert interviews and identifying design method configurations by using focus groups. Section 4 presents our findings, including the conceptualization of the taxonomy and the combinations of its dimensions. In Section 5, we discuss our contribution, limitation and future work before concluding our study.
2. Background
During the digital innovation process, design and evaluation activities are often conducted in parallel and studied jointly because they form an iterative process (Brhel et al., 2015; Zobel et al., 2019) and often require different tools to effectively support the innovation process (Mourtzis, 2020). Design activities aim to create products that meet user needs and expectations, for example, by drawing storyboards (Henrikson et al., 2016) or sketches (Tohidi et al., 2006). In contrast, evaluation activities aim to assess the effectiveness and efficiency of products, for example, by using A/B testing (Kohavi et al., 2007) or Wizard of Oz (Porcheron et al., 2021). As Figure 1 presents, design and evaluation activities are complementary and mutually reinforcing, which gradually meet user needs and expectations through continuous iteration. From here on, when we use the term design methods, we refer to both design and evaluation activities.
2.1 Differences between design experts and novices
When faced with complex design situations, experts often combine multiple design methods to achieve design goals (Gray, 2016a). For example, sketching, cartooning and role-playing may be combined to explore ideas (Harrison et al., 2006). However, this is often tacit knowledge that an expert designer acquires after many years of professional training. The ability to use design methods creatively and flexibly while taking into account the mismatch between research and practice (Roedl and Stolterman, 2013) requires years of training and experience (Rivard and Faste, 2012; Zhang and Wakkary, 2014). Compared to design experts, novices’ limited design background knowledge and experience lead to different design habits (Björklund, 2013). Design novices typically cannot frame design problems as effectively as experts (Kim and Ryu, 2014) due to their lack of training and prior experience (Oleson et al., 2020). Design experts use abstract experiential knowledge (Stolterman and Pierce, 2012), whereas novices invoke concrete, relevant and previously mapped solutions (Ball et al., 2004). Design novices often face the challenge of selecting appropriate design methods, particularly in complex design situations.
2.2 Conditions of selecting design methods
Different conditions apply when selecting design methods, making the innovation process even more complex (Sarbu, 2022). For instance, trends such as Industry 4.0 enable the servitization of entire industries and demand more integrative approaches when designing digital solutions (Benitez et al., 2020; Ibarra et al., 2018). Previous studies identified different conditions for selecting design methods in the form of categories that focus on specific goals. For example, Vermeeren et al. (2010) categorized design methods for user experience evaluation; IDEO (2015) categorized design methods for human-centered design; and Sanders et al. (2010) focused on participatory design. As digital innovation processes of UI design include multiple aspects such as user involvement, ideation and evaluation, design methods from different domains can be adopted. A review of these categories reveals both similarities and differences between design methods. The existing categories and examples of subcategories are shown in Table 1. Among the existing classifications, design phase is a widely accepted category (e.g., Alves and Jardim Nunes, 2013; Martin and Hanington, 2012); more than half of them agree on participants as an important category; and one-third of them have the categories of duration, evaluation type and resources (e.g., IDEO, 2015; Rivero and Conte, 2017). These categories are valuable references for identifying the conditions of design methods for digital innovation design. However, common limitations of these classifications include:
they were developed with a narrowly defined purpose in mind; and
the categories are isolated from each other without any details on how their categories can be combined when selecting design methods.
Guidance for the situational selections of design methods should be supported by joint consideration of conditions, especially for design novices.
3. Research approach
We developed and evaluated our taxonomy in three phases, as shown in Figure 2. In phase 1 (steps 1–6), we followed the taxonomy development method (Nickerson et al., 2013) to classify design methods, combining conceptual-to-empirical and empirical-to-conceptual approaches. In total, 292 design methods were reviewed, and an initial version of the taxonomy was developed with five dimensions and 15 characteristics. In phase 2 (steps 7 and 8), to evaluate the identified dimensions and characteristics, we applied in-depth expert interviews (Schultze and Avital, 2011). We interviewed 15 experts and calculated the precision of the taxonomy. We added a dimension and four characteristics after analyzing the interview transcripts. In phase 3 (steps 9 and 10), we conducted focus groups with 25 design novices and analyzed the data using fsQCA (Ragin, 2008) to identify configurations of dimensions that lead to confident decisions about design methods. There are two findings derived from phases 2 and 3, in which we conceptualize the taxonomy and present the combinations of its dimensions.
3.1 Classifying design methods
The taxonomy development method requires the identification of subsets of objects (in this research: design methods) so that a taxonomy can be developed iteratively by expanding it as new data sources emerge (Nickerson et al., 2013). Given the constant emergence of new design methods, obtaining a definitive list of all available design methods can be challenging. Therefore, we combined multiple data sources and different means of data collection to identify a comprehensive data set. In the following sections, we describe our steps in detail.
At first, we determined the meta-characteristic and purpose of the taxonomy (step 1 in Figure 2), that is, to guide design novices to select design methods based on different design situations. Therefore, we established that the dimensions needed to reflect different design situations, and we decided that the taxonomy should not add much cognitive load to novices.
In the second step, we defined the ending criteria (Table 2) that were used to terminate the iteration in the taxonomy development procedure. Specifically, we distinguished between subjective and objective ending criteria to guide the development process by preventing endless iterations during development (Nickerson et al., 2013). When a given data set provided a new design method that could not be assigned to existing dimensions or characteristics, a new dimension or characteristic was added. Objective-ending criteria tested whether each dimension had mutually exclusive and collectively exhaustive characteristics. Subjective ending criteria checked whether the identified dimensions and characteristics are useful. Since a single person judging the ending condition may introduce a bias, two of the authors jointly decided on the ending criteria. Potential disagreements were discussed and resolved.
In the conceptual-to-empirical approach (steps 3a–5a in Figure 2), we reviewed existing classifications and identified commonly accepted dimensions and characteristics (in Section 2.2). The purpose was to identify prospective dimensions of design methods, examine them and create dimensions and characteristics for further iterations.
In the empirical-to-conceptual approach, in step 3b, we reviewed 292 design methods [1] and identified a subset (68 design methods) to develop the taxonomy. The subset includes design methods that clearly described a set of steps so that they could be easily understood and applied by design novices (Barreto et al., 2015). During the iterations, we constantly compared newly added methods with already identified methods to merge them. In such cases we made the following decisions: First, we selected the method with a more precise name that explained it well (e.g. concurrent think-aloud and retrospective think-aloud were chosen instead of think-aloud). Second, if none of the names was considered more precise than the other, we selected the most frequently used design methods (e.g. affinity diagramming was chosen instead of the Kawakita Jiro method). In step 4b, we summarized the descriptions of each design method into different characteristics by applying open coding (Corbin and Strauss, 2014). The two researchers who evaluated the ending criteria conducted open coding. For example, if the method’s description indicated that it could be used in several hours or a couple of days, we coded it as “short-term.” In step 5b, all the characteristics were grouped into five dimensions: design phase, time dependency, duration, participant and evaluation type.
3.2 Evaluating the taxonomy with expert interviews
Interviews can generate rich data and provide insights into people’s experiences (Schultze and Avital, 2011). Therefore, we interviewed 15 experts [2] from industry and academia, which allowed us to evaluate whether the dimensions and characteristics were consistent with their experiences (step 7 in Figure 2). Design experts were also uniquely positioned to evaluate the content of the taxonomy based on their education and experiences (Kitchenham et al., 2005; Schultze and Avital, 2011). We adopted a framework for the evaluation (Szopinski et al., 2019). We conducted face-to-face interviews with design experts who had previous experiences. Based on the evaluation purpose, we prepared an interview protocol with both appreciative and laddering questions (Schultze and Avital, 2011). The evaluation was based on design situations that the participants had experienced in real design projects. Each interview consisted of two parts: First, we investigated the design situations by asking experts to describe the selected design methods and the considered conditions. In the second part, we presented our taxonomy to collect experts’ feedback.
Design practitioners generally include people from multidisciplinary backgrounds (supplemental material A) to get an overview of how they conducted design activities in innovation processes and test whether people from various domains could understand the taxonomy. All interviews were audio-recorded after requesting permission from experts. During each interview, additional notes were taken. Each interview took 20–45 min. All interview recordings were transcribed.
The transcripts were analyzed using closed and open codes (Myers, 2009) and QDA Miner 5 [3]. The interviewer conducted closed and open coding. The closed codes were the dimensions and characteristics of our developed taxonomy and were extended through open coding. A coding guideline was developed to increase the validity of our coding results. A second coder was trained using the coding guideline (Myers, 2009). We used Krippendorff’s alpha to measure the agreement between both coders (Krippendorff, 2004). The percentage of the code co-occurrence was 89.8%, and Krippendorff’s alpha was 0.765 (Cohen’s kappa: 0.764; Scott’s pi: 0.765), which is considered acceptable reliability (Krippendorff, 2004).
We followed Fawcett (2006) to calculate the precision of our proposed taxonomy (step 8 in Figure 2). First, we matched the responses by the experts with our proposed taxonomy. Table 3 displays six characteristics that were not initially included in the first version of the taxonomy but were mentioned by experts during the interviews. We categorized them into three dimensions: participant, project type and setting. Second, the following formula was applied to measure the precision – namely, the proportion of correctly predicted positive results (also called true positives) over the sum of all correctly and incorrectly predicted positives:
True positive is the number of confirmed characteristics from the interviews, and false positive is the number of additionally identified characteristics. The precision of the taxonomy is influenced by the frequency of characteristics mentioned by the experts that align with our taxonomy. Third, we observed large precision (93.78%), indicating that most experts provided characteristics that were already included in the initially proposed taxonomy.
Based on the expert interview analysis, we enhanced the taxonomy by adding a new dimension setting, extending the participant dimension and improving the wording of the previously generated taxonomy.
In the conceptualization phase, we defined the dimensions and characteristics for a better understanding and application of the taxonomy (Figure 3). The classified design methods can be found in supplemental material B.
3.3 Design method configurations
To gain a better understanding of the possible configurations of dimensions, we conducted focus groups with design novices and used fsQCA (Ragin, 2008) to analyze the result as part of the configuration phase.
3.3.1 Focus group with novices.
We organized five sessions of focus groups with 25 university students (step 9 in Figure 2; see also supplemental material C) consisting of nine activities (Table 4). Each session took around 60 min and contained five participants. The first author acted as a moderator in each focus group session.
We used 6-3-5 brainwriting as a running example for a design method during the focus groups. As a specific method of brainstorming, compared with Kano analysis and repertory grids, 6-3-5 brainwriting is more closely relevant to participants’ everyday lives. Thus, we used it to make sure design novices understood how a design method could be applied in a design process. After introducing the dimensions, participants were asked to fill out an online questionnaire to evaluate each dimension’s importance when selecting appropriate design methods (seven-point Likert scale: 1 = not important at all; 7 = very much important). At the end of the questionnaire, participants responded to three questions on the confidence of the helpfulness of the evaluated dimensions when selecting appropriate design methods (seven-point Likert scale: 1 = not at all; 7 = very much). The measurements (Cronbach’s alpha = 0.8) were adapted from Barden and Petty (2008). Overall, the participants had high confidence in their evaluation (M = 5.12, SD = 0.67).
3.3.2 Fuzzy-set qualitative comparative analysis (fsQCA).
When selecting a design method, multidimensional design situations need to be considered. Identifying a suitable design method is not an issue that can be easily solved using a single dimension. The dimensions should be combined to reflect the design situations at hand. Therefore, we opted for fsQCA as a method that reflects the combinations of dimensions into configurations (step 10 in Figure 2). We analyzed and compared multiple configurations to find beneficial combinations of dimensions when selecting design methods (Park et al., 2020). In fsQCA [4], the combination of conditions (i.e. dimensions) leading to the desired outcome is called a configuration. Configurations can be used to explain how bundles of conditions achieve an outcome (Rihoux and Ragin, 2008). Furthermore, fsQCA enables the analysis of the absence of conditions when predicting an outcome (Misangyi and Acharya, 2014).
FsQCA has been applied in various domains, such as information systems, where it is used to explain the role of business intelligence and communication technologies in organizational agility (Park et al., 2017); when analyzing human behavior, it is used to investigate the influence of social endorsement on customer–brand relationships (Thai and Wang, 2020); and to examine the antecedents of users’ intentions to use smart retail carts (Fazal-e-Hasan et al., 2020). Accordingly, fsQCA enabled us to describe configurations of dimensions, leading to confident decision-making concerning design methods. Specifically, we used fsQCA 3.0 software (Ragin, 2017) for the analysis.
Before generating the configuration, we calibrated the data. We used the collected data about the importance of dimensions (i.e. conditions) and confidence in selection (i.e. outcome) to derive set membership scores. To do so, we used the direct method (Ragin, 2008) of calibration to transform the data based on three qualitative thresholds: full membership, the crossover point and full nonmembership. The fuzzy membership scores range between 1 (full membership) and 0 (full nonmembership) (Ragin, 2008). We used the maximum, mean and minimum values of the conditions and outcome during calibration as the three thresholds.
We used the truth table algorithm (Ragin, 2008) to identify the configurations that lead to confidence in selection. The truth table presents all logically possible combinations of relevant conditions. We set a minimum acceptable frequency of cases at one, the cutoff for raw consistency [5] at 0.85 and the cutoff of proportional reduction in inconsistency [6] at 0.75 (Ragin, 2008). We then used the truth table algorithm to reduce the numerous combinations into solutions that included configurations of conditions. The analysis resulted in three solutions: complex solution [7], parsimonious solution [8] and intermediate solution [9]. When a variable exists both in the configuration of parsimonious solution and intermediate solution, it is a central condition. When a variable only exists in the configuration of an intermediate solution, it is a peripheral condition.
We judged the quality of a configuration based on its consistency and coverage. Consistency refers to how many cases share a configuration in displaying the outcome (Ragin, 2008), whereas coverage refers to how much of the outcome is explained by a configuration of the overall solution. We distinguished between raw coverage and unique coverage. Raw coverage indicates the proportion of memberships in the outcome that can be explained by each configuration in the solutions, which includes cases that can be covered by multiple configurations. Unique coverage indicates the proportion of memberships in the outcome that are only associated with an individual solution (Ragin, 2008).
4. Findings
In the findings, we conceptualized the dimensions and characteristics in detail and presented the combinations of the taxonomy’s dimensions based on the fsQCA result.
4.1 Taxonomy for design method selection
We developed and evaluated a taxonomy for the situational selection of design methods within digital innovation processes. In the following, we describe each taxonomy’s dimension and its characteristics in more details. The taxonomy is presented in Figure 3.
Duration describes the time length of an iteration cycle of a design process. Long-term studies need to be conducted over a long period to identify trends in users’ satisfaction (e.g. collecting users’ day-to-day experiences by applying long-term diary study). Short-term studies can be used to conduct design activities or data analysis in a rather limited period of time (e.g. using bodystorming to quickly evaluate low-fidelity prototypes).
Time dependency discriminates between immediate affective feedback and retrospective cognitive feedback based on memory. The collection and analysis of real-time feedback reveal real-time use and users’ affective change when interacting with UIs (e.g. using the eye-tracking method to analyze users’ reading habits). Retrospective feedback is collected based on users’ memory of their experience in a previous episode that happened immediately or early before (e.g. using a service experience tracker to gather user feedback periodically).
Design phases are conducted in an iterative cycle within design processes. In the planning phase, the design methods used in this phase should include generating design ideas, understanding the use cases and planning the following steps. For example, design methods are applied to analyze design ideas (e.g. using territory maps to visualize shared focuses of a UI team) and to collect and analyze context and users (e.g. using a touchpoint matrix to analyze the connected UIs). In the low-fidelity prototyping phase, designers use design methods to create low-fidelity prototypes (e.g. applying collaborative sketching to create prototypes with collaboration) and to make a series of assessments to compare the prototypes (e.g. using the repertory grids to reveal which paper prototype satisfies them). In the high-fidelity prototyping phase, design methods focus on the in-depth optimization and evaluation of design work and the modification of the prototype to get a stable version with basic functions. The design methods should be used to analyze detailed features and collect user feedback (e.g. using concurrent think-aloud to detect usability problems). In the release phase, the design methods should collect data comprehensively from a pilot version, an internally released version or a publicly released version to further develop the designs and set up the next iteration (e.g. using web analytics to understand web usage better to improve the design).
The dimension participant is used to decide what roles need to be included when applying specific methods. When real users are involved in the design process, designers can observe, analyze and predict how well their designs fit users’ expectations (e.g. involving users to create low-fidelity prototypes by using flexible modeling methods). Except for users, in some situations, stakeholders (i.e. customers, managers, front-line employees, engineers) need to be involved when using a design method (e.g. using a business origami to understand users’ workflow). Furthermore, in some occasions, when specific expertise is needed, using design methods needs the participation of the product team (e.g. creating personas by summarizing the data from user research).
As the design process is iterative, evaluations do not take place only at the end of the design process but also run throughout. The evaluation type prescribes how to evaluate the outcome of a design activity. For example, designers use the design method scenario to create narratives, exploring how people will interact with a UI. The narratives help a design team when discussing a design’s version and goal. When a design method is used within a team to share ideas or make a decision, we assign this method to group discussion (e.g. applying stakeholder walkthrough to evaluate low-fidelity prototypes). The other four characteristics of evaluation types are questionnaire, interview, experiment and observation. The questionnaire enables the evaluation of users’ goal achievement and satisfaction (e.g. collecting users’ emotional feedback by using the “3E” method to create a questionnaire). Interview generates rich data that reveal users’ experiences and expectations (e.g. using photo-elicitation interviewing to evoke conversation and recall users’ experiences). Experiment reflects user behavior in a specific setting through concrete data (e.g. testing whether users are willing to do things in a new way by using the Wizard of Oz method). Observation includes observing how users perform in their daily life and whether UIs fulfill rules to achieve high usability (e.g. using the fly-on-the-wall observation method to monitor users without interference).
Setting indicates the location as a precondition when applying a design method. Some methods need to be conducted in the field (e.g. using fly-on-the-wall to observe people’s behavior while interacting with the environment). Other methods require controlling the environment or using specific equipment that is only available in a lab (e.g. using co-discovery in a laboratory to observe how two people explore a new UI or using eye-tracking equipment to measure users’ eye movement). Furthermore, some methods, such as actor mapping and affinity diagramming, have no special needs for the environment and can be conducted in regular office environments.
4.2 Combinations of taxonomy’s dimensions
The fsQCA results shed more light on the combinations of taxonomy’s dimensions. Table 5 graphically presents the configurations of dimensions to be sufficient for confident selection.
The configurations in the columns associated with confidence in selection resulted in an overall solution consistency of 0.91. The central conditions in configuration 1(a, b, c) suggest that the combination of the presence of design phase and time dependence leads to confidence in selection. More precisely, configurations 1a, 1b and 1c share the same presence of central conditions and only differ in their peripheral conditions. Configuration 2 suggests that the combination with central conditions of the presence of design phase, duration, time dependency and evaluation type, as well as the peripheral condition of the presence of setting, leads to confidence in selection. Unlike configurations 1(a, b, c) and 2, configuration 3 indicates the absence of design phase as a central condition with the combination of the presence of duration, time dependency, setting and evaluation type (peripheral condition), leading to confidence in selection. As such, configuration 3 suggests that, except for design phase, all other dimensions (evaluation type as a peripheral condition) need to be considered when selecting design methods.
The absence of confidence in selection contains two configurations with an overall solution consistency of 0.96. Configurations 4a and 4b have the same central conditions of the absence of design phase, duration and setting, leading to the absence of confidence in decision-making. The absence of design phase is also a central condition in configuration 5, with a combination of the absence of time dependency and the presence of participant sufficient to predict not confidence in decision-making.
A comparison between the configurations in the solutions of confidence and not confidence in selection indicates that each dimension has its role in combination with other dimensions. Therefore, all dimensions support the selection of design techniques. The absence of design phase is a central condition in all configurations, leading to the absence of confidence in selection. Only when all other dimensions are considered, without considering design phase, can we achieve confidence (3). Our results suggest that design phase is an important dimension when selecting design methods. The combination of design phase and time dependency are central conditions for achieving confidence in selection (1 and 2). Thus, time dependency also plays an indispensable role. Since configuration 5 presents time dependency as a peripheral condition leading to not confidence, time dependency needs to be considered in combination with design phase. The quotation analysis of focus groups further strengthens the result (supplemental material D).
5. Discussion
This study developed a comprehensive taxonomy for design methods and identified important configurations, that is, combinations of multiple design methods based on particular conditions. The study differs from existing research in three important aspects. First, prior classifications were developed with a narrowly defined purpose in mind, whereas our taxonomy provides a comprehensive view that accounts for various project situations. Second, prior classifications present dimensions independently from one another (e.g., IDEO, 2015; Rivero and Conte, 2017), while we establish connections between dimensions in the form of configurations. Third, existing research provides design strategies that focus on experts. In contrast, our focus is to guide the selections of design methods tailored to specific conditions for novices that lack experience and expertise with different design methods (e.g., Acar et al., 2019; Chen et al., 2022). Therefore, our study helps novices to understand conditions and select design methods for their project situations.
5.1 Contributions
This study makes four important contributions. First, we add important dimensions and characteristics to the nomological network of similarities and differences among design methods (Gregor, 2006). As such, the taxonomy contributes to the body of knowledge and provides a foundation for further theorizing about design method selections. The proposed dimensions are important factors in selecting appropriate design methods while considering different design situations. Researchers identifying and developing new design methods can benefit from our proposed dimensions and characteristics because they can serve as a tool to communicate and position the novelty of their proposed design method. Therefore, researchers can use the taxonomy to clearly articulate the conditions and the intended design situations of their methods.
Second, we extend the development of the taxonomy by analyzing the configurations of the dimensions, which complements the limitation of isolated dimensions in the taxonomy (Nickerson et al., 2013). Previous scholars suggest the importance of considering different aspects within the innovation process, but research often focused on individual methods when investigating situational needs. We address the common limitation of isolated dimensions in the taxonomy (Bailey, 1994) by proposing key dimensions that collectively guide the selections of design methods. The configuration of dimensions advances the theory about the selections of different design methods, taking into account individual project needs and different design situations.
Third, the developed taxonomy answers the call for more research that supports the selections of design methods based on specific design situations (Maguire, 2001). There has been skepticism about classifications of design methods in terms of their practical use, and the use of appropriate design methods has been viewed as an opportunistic combination of resources (Gray, 2016a, 2016b). However, codified knowledge of design methods provides an overview and gives design novices a tool to compensate for their limited design knowledge and experience. Thus, professionals from other fields can greatly benefit from the codified knowledge of the design community. The identified dimensions can help scholars investigate the antecedents of the successful application of design methods for the innovation process. Therefore, we also extend previous research on the selections of engineering methods based on their classified attributes (Jiang et al., 2008).
Fourth, we extend and contribute prior research on the management of digital innovation (Nambisan et al., 2017; Satwekar et al., 2023). Prior research on digital innovation management often focused on the overall business process of digital innovation from ideation to productization. Our research takes a more nuanced approach by focusing on the particularly task of design method selection, given its pivotal role in shaping the design and success of the innovation. Therefore, our research results provide practitioners with an important design tool that helps novice designers in optimizing their available resource for productivity gains.
5.2 Limitations and future work
Our study also has its limitations. First, our taxonomy does not include all possible dimensions that have been suggested by previous research, such as the origin of a design method and its content (see for example Martin and Hanington, 2012). Although these dimensions might be helpful when learning and understanding design methods, our focus was on the selections of design methods by design novices. Future research can investigate how individuals can best develop a deep understanding of design methods to become proficient. Such dimensions can also be combined with our existing dimensions to improve the application of design methods.
Second, the applied taxonomy development classified design methods into mutually exclusive and collectively exhaustive characteristics. As a result, the taxonomy meets established quality requirements (see Nickerson et al., 2013) and limits the possible application scenarios for a given design method to its primary application. A design novice can thus benefit from limited design choices because less information is easier to process and remember. However, a design method can also be assigned to more than one characteristic. Therefore, future research could be devoted to the implementation of probability scores. For example, a design method that could be applied to two characters would receive two probability scores indicating the probability that it was applied to each of the two characteristics, for example, 90% and 10%.
Third, we used fsQCA to analyze and identify existing configurations across the identified dimensions (Park et al., 2017). While the analysis allows us to identify the combination of dimensions for the successful selections of design methods, the analysis and results need to be interpreted with care. The results may seemingly present some dimensions as more important when compared with others. However, the analysis is not designed to make such claims, and future research could adopt methods that identify the weight factor of each dimension objectively.
6. Conclusion
We developed a new taxonomy that offers a comprehensive overview of design conditions and the associated design methods in innovation processes. The taxonomy enables design novices to navigate the complexities of design methods needed to design digital innovation. Instead of the isolated dimensions in existing research, we analyzed the configuration dimensions to support the situational selections of design methods. By understanding the conditions, design novices can make more informed decisions regarding their selections of design methods.
The study improves our understanding of similarities and differences among design methods by examining crucial dimensions and characteristics. By developing a taxonomy that considers these factors, the study provides a foundation for further theoretical advancements in design method selections. The proposed dimensions can assist researchers in communicating and positioning the novelty of their design methods as well as articulating the conditions and intended design situations of their methods. Furthermore, the study addresses limitations in prior studies by analyzing the configurations of dimensions, which facilitates a more comprehensive approach to selecting design methods. The developed taxonomy supports the selections of design methods based on specific design situations by design novices. In addition, the identified dimensions can aid in investigating the factors that contribute to the successful application of design methods in innovation processes.
For professionals, the taxonomy presents a tool that can guides them in their selections of design methods. This is particularly important for professional in other non-design-related fields who have limited design knowledge and experience. They benefit from the taxonomy as a decision aid that helps them to select the right design methods for their design situations to improve the performance of the design process and the quality of the design outcome.
Figures
Categories of design methods in prior classifications
| Categories | Example sub-categories | Studies | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | Count | ||
| Design phase | Concept, functional prototype | x | x | x | x | x | x | x | x | x | 9 |
| Participant | Users, UX experts | x | x | x | x | x | 5 | ||||
| Duration | Minimum, maximum | x | x | x | 3 | ||||||
| Evaluation type | Observational, participatory | x | x | x | 3 | ||||||
| Origin | Industry, adapted, traditional | x | x | 2 | |||||||
| Data type | Qualitative, quantitative | x | x | 2 | |||||||
| Resource | Pen, notebook, post-it notes | x | x | x | 3 | ||||||
| Location | Lab, field | x | x | 2 | |||||||
| Application | Web service, hardware design | x | 1 | ||||||||
| Content | Attitudinal, behavioral | x | 1 | ||||||||
| Purpose | Making, telling, enacting | x | 1 | ||||||||
1 = (Roschuni et al., 2015); 2 = (Kumar and LaConte, 2012); 3 = (Martin and Hanington, 2012); 4 = (IDEO, 2015); 5 = (Rivero and Conte, 2017); 6 = (Sanders et al., 2010); 7 = (Vermeeren et al., 2010); 8 = (Alves and Jardim Nunes, 2013); 9 = (Curedale, 2013)
Source: Authors’ own creation
Ending criteria for identifying dimensions and characteristics
| Ending criteria | Steps |
|---|---|
| Objective ending criteria | |
| No design methods can be merged or split | 3b |
| At least one design method is classified under each characteristic | 4a, 4b |
| Each characteristic is unique and cannot be repeated (no characteristic duplication) | 5a, 4b |
| Each dimension is unique and cannot be repeated (no dimension duplication) | 5a, 5b |
| No new dimensions or characteristics can be added in the last iteration | 5a, 5b |
| No dimensions or characteristics can be merged or split | 5a, 5b |
| Subjective ending criteria | |
| Concise: the number of dimensions is not unwieldy or overwhelming | 5a, 4b, 5b |
| Robust: enough dimensions and characteristics are available to classify design methods | 5a, 4b, 5b |
| Comprehensive: all identified design methods are classified within the taxonomy | 5a, 4b, 5b |
| Extendible: a new design method, characteristic and dimension can be easily added | 5a, 4b, 5b |
| Explanatory: the dimensions and characteristics can explain design methods | 5a, 4b, 5b |
Source: Authors’ own creation
The characteristics mentioned by the experts that matched the proposed taxonomy
| Class | Confirmed characteristics/true positives (number of observations) | New characteristics/false positives (number of observations) |
|---|---|---|
| Dimensions Characteristics |
Participant User involved (14) Without user (11) |
Participant Stakeholder (3) |
| Design phase Planning (15) Draft prototyping (14) Detailed prototyping (14) Launching (5) |
Project type Complex project (2) Detail project (1) New project (1) |
|
| Duration Long-term study (4) Short-term study (11) |
Setting Field (1) Lab (2) |
|
| Time dependency Real-time (11) Retrospective (7) |
||
| Evaluation type Questionnaire (4) Interview (11) Experiment (5) Observation (12) Group discussion (13) |
||
| Total | 151 | 10 |
The numbers in brackets refer to each term’s occurrence when mentioned by the experts
Source: Authors’ own creation
The process of each focus group discussion
| Phase | Activity | Actor | Time |
|---|---|---|---|
| 1 | • Welcoming participants and introducing the group discussion | Moderator | ∼7 min |
| • Self-introduction | Participants | ||
| 2 | • Introducing the tasks in the group discussion | Moderator | ∼12 min |
| • Introducing 6-3-5 Brainwriting as an example of design methods and when the method can be used • Showing how to narrow down the search scope of design methods using the taxonomy • Introducing dimensions of design methods |
|||
| 3 | • Filling out a questionnaire on evaluating dimensions | Participants | ∼37 min |
| • Discussing the anonymized results of the questionnaire | |||
| 4 | • Summarizing the discussion | Moderator | ∼ 4 min |
Source: Authors’ own creation
The sufficiency of the dimensions for the confidence in selection
| Confident in the selection | Not confident in the selection | |||||||
|---|---|---|---|---|---|---|---|---|
| Configurations | 1a | 1b | 1c | 2 | 3 | 4a | 4b | 5 |
| Taxonomy dimension | ||||||||
| Design phase | ● | ● | ● | ● | ×○ | ×○ | ×○ | ×○ |
| Participant | • | • | ⊗ | ● | ⊗ | • | ● | |
| Duration | • | • | • | ● | ● | ×○ | ×○ | • |
| Time dependency | ● | ● | ● | ● | ● | • | ⊗ | ×○ |
| Evaluation type | ⊗ | ⊗ | ● | • | ⊗ | • | ||
| Setting | ×○ | ×○ | • | ● | ×○ | ×○ | • | |
| Consistency | 0.94 | 0.96 | 0.92 | 0.91 | 0.95 | 0.95 | 0.98 | 0.97 |
| Raw coverage | 0.36 | 0.38 | 0.38 | 0.25 | 0.2 | 0.31 | 0.25 | 0.21 |
| Unique coverage | 0.02 | 0.04 | 0.02 | 0.03 | 0.04 | 0.14 | 0.04 | 0.04 |
| Overall solution consistency | 0.91 | 0.96 | ||||||
| Overall solution coverage | 0.53 | 0.43 | ||||||
“●” represents the presence of central conditions; “×○” represents the absence of the central conditions; “•” represents the presence of peripheral conditions; and “⊗” represents the absence of the peripheral conditions
Source: Authors’ own creation
Notes
usability.gov is a leading resource for usability practices and introduces 32 design and evaluation methods; servicedesigntools.org is a research conducted by the Research & Consulting Center of Domus Academy, which contains 36 methods, techniques and tools for service design; allaboutux.org provides a list of 82 evaluation methods for user experience (Vermeeren et al., 2010); Service Design (Curedale, 2013) is a book that includes 250 methods, techniques and tools for service design; and Universal Methods of Design (Martin and Hanington, 2012) is a book that includes 100 methods and techniques for widespread use in product and service design.
As the interviews were used for the evaluation of the taxonomy but were not the only data source for developing it, we considered 15 experts to be sufficient to evaluate it (Dell and Kumar, 2016; Vigo et al., 2014).
QDA Miner 5 is a qualitative data analysis tool (https://provalisresearch.com/products/qualitative-data-analysis-software/).
Detailed explanations and guidelines of fsQCA can be found in works such as those of Ragin (2008), Rihoux and Ragin (2008), Park et al. (2020) and Fiss (2011).
Raw consistency indicates the degree to which the membership is a consistent subset of membership in the outcome (Ragin, 2017).
Proportional reduction in inconsistency measures how much a condition or a configuration is a subset of the outcome and the negation of the outcome (Li and Ma, 2019; Park et al., 2017; Ragin, 2017).
Complex solution: no remainders and no counterfactuals (Ragin, 2017). Remainders are logically possible configurations with no existing case (Misangyi and Acharya, 2014).
Parsimonious solution: any remainder that helps generate a logical solution is included; “easy” and “difficult” counterfactuals are used (Ragin, 2017). “Easy” counterfactual indicates that a redundant causal condition is added to a set of conditions leading to the outcome; “difficult” counterfactual indicates that a condition that is assumed redundant is removed from a set of causal conditions leading to the outcome (Fiss, 2011).
Intermediate solution: remainders with “easy” counterfactuals are used (Ragin, 2017).
Supplementary material
The supplementary material for this article can be found online.
References
Acar, O.A., Tarakci, M. and Van Knippenberg, D. (2019), “Creativity and innovation under constraints: a cross-disciplinary integrative review”, Journal of Management, Vol. 45 No. 1, pp. 96-121, doi: 10.1177/0149206318805832.
Ahmed, S., Wallace, K.M. and Blessing, L.T. (2003), “Understanding the differences between how novice and experienced designers approach design tasks”, Research in Engineering Design, Vol. 14 No. 1, pp. 1-11, doi: 10.1007/s00163-002-0023-z.
Alves, R. and Jardim Nunes, N. (2013), “Towards a taxonomy of service design methods and tools”, in Falcão e Cunha, J., Snene, M. and Nóvoa, H. (Eds), Exploring Services Science, Vol. 143 Springer, Berlin Heidelberg, Berlin, Heidelberg, pp. 215-229, doi: 10.1007/978-3-642-36356-6_16.
Bailey, K.D. (1994), Typologies and Taxonomies: An Introduction to Classification Techniques, Sage Publications, Thousand Oak (Calif).
Ball, L.J., Ormerod, T.C. and Morley, N.J. (2004), “Spontaneous analogising in engineering design: a comparative analysis of experts and novices”, Design Studies, Vol. 25 No. 5, pp. 495-508, doi: 10.1016/j.destud.2004.05.004.
Barden, J. and Petty, R.E. (2008), “The mere perception of elaboration creates attitude certainty: exploring the thoughtfulness heuristic”, Journal of Personality and Social Psychology, Vol. 95 No. 3, pp. 489-509, doi: 10.1037/a0012559.
Barreto, F., Benitti, V., Campus, S.C., Sommariva, L. and Uruguai, R. (2015), “Evaluation of a game used to teach usability to undergraduate students in computer science”, Journal of Usability Studies, Vol. 11 No. 1, pp. 21-39.
Benitez, G.B., Ayala, N.F. and Frank, A.G. (2020), “Industry 4.0 innovation ecosystems: an evolutionary perspective on value cocreation”, International Journal of Production Economics, Vol. 228, p. 107735, doi: 10.1016/j.ijpe.2020.107735.
Björklund, T.A. (2013), “Initial mental representations of design problems: differences between experts and novices”, Design Studies, Vol. 34 No. 2, pp. 135-160, doi: 10.1016/j.destud.2012.08.005.
Brhel, M., Meth, H., Maedche, A. and Werder, K. (2015), “Exploring principles of user-centered agile software development: a literature review”, Information and Software Technology, Vol. 61, pp. 163-181, doi: 10.1016/j.infsof.2015.01.004.
Chamorro-Premuzic, T. (2017), Does Diversity Actually Increase Creativity?, Harvard Business Review.
Chen, H.-J., Chen, Y.-T. and Yang, C.-H. (2022), “Behaviors of novice and expert designers in the design process: from discovery to design”, International Journal of Design, doi: 10.57698/V16I3.04.
Corbin, J. and Strauss, A. (2014), Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, Fourth. Sage, Thousand Oaks, CA.
Cornelissen, J. (2017), “Editor’s comments: developing propositions, a process model, or a typology? Addressing the challenges of writing theory without a boilerplate”, Academy of Management Review, Vol. 42 No. 1, pp. 1-9, doi: 10.5465/amr.2016.0196.
Curedale, R. (2013), Service Design: 250 Essential Methods, Design Community College, Topanga CA.
Dell, N. and Kumar, N. (2016), “The ins and outs of HCI for development”, Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems – CHI’16, ACM, New York, NY, pp. 2220-2232, doi: 10.1145/2858036.2858081.
Fawcett, T. (2006), “An introduction to ROC analysis”, Pattern Recognition Letters, Vol. 27 No. 8, pp. 861-874, doi: 10.1016/j.patrec.2005.10.010.
Fazal-e-Hasan, S.M., Amrollahi, A., Mortimer, G., Adapa, S. and Balaji, M.S. (2020), “A multi-method approach to examining consumer intentions to use smart retail technology”, Computers in Human Behavior, Vol. 117, p. 106622, doi: 10.1016/j.chb.2020.106622.
Fiss, P.C. (2011), “Building better causal theories: a fuzzy set approach to typologies in organization research”, Academy of Management Journal, Vol. 54 No. 2, pp. 393-420, doi: 10.5465/AMJ.2011.60263120.
Ghobakhloo, M., Iranmanesh, M., Grybauskas, A., Vilkas, M. and Petraitė, M. (2021), “Industry 4.0, innovation, and sustainable development: a systematic review and a roadmap to sustainable innovation”, Business Strategy and the Environment, Vol. 30 No. 8, pp. 4237-4257, doi: 10.1002/bse.2867.
Gray, C.M. (2016a), “It’s more of a mindset than a method’: UX practitioners’ conception of design methods”, Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, Santa Clara, CA, pp. 4044-4055, doi: 10.1145/2858036.2858410.
Gray, C.M. (2016b), “What is the nature and intended use of design methods?”, presented at the Design Research Society Conference 2016, doi: 10.21606/drs.2016.307.
Gregor (2006), “The nature of theory in information systems”, MIS Quarterly, Vol. 30 No. 3, p. 611, doi: 10.2307/25148742.
Harrison, S., Back, M. and Tatar, D. (2006), “It’s just a method!’: a pedagogical experiment in interdisciplinary”, Design Proceedings of the 6th Conference on Designing Interactive Systems, ACM, New York, NY, pp. 261-270, doi: 10.1145/1142405.1142445.
Henrikson, R., De Araujo, B., Chevalier, F., Singh, K. and Balakrishnan, R. (2016), “Storeoboard: sketching stereoscopic storyboards”, Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, pp. 4587-4598, doi: 10.1145/2858036.2858079.
Ibarra, D., Ganzarain, J. and Igartua, J.I. (2018), “Business model innovation through industry 4.0: a review”, Procedia Manufacturing, Vol. 22, pp. 4-10, doi: 10.1016/j.promfg.2018.03.002.
IDEO (2015), “The field guide to human-centered design”, available at: www.designkit.org/resources/1.html
Jiang, L., Eberlein, A., Far, B.H. and Mousavi, M. (2008), “A methodology for the selection of requirements engineering techniques”, Software and Systems Modeling, Vol. 7 No. 3, pp. 303-328, doi: 10.1007/s10270-007-0055-y.
Kim, J. and Ryu, H. (2014), “A design thinking rationality framework: framing and solving design problems in early concept generation”, Human–Computer Interaction, Vol. 29 Nos 5/6, pp. 516-553, doi: 10.1080/07370024.2014.896706.
Kitchenham, B., Linkman, S. and Linkman, S. (2005), “Experiences of using an evaluation framework”, Information and Software Technology, Vol. 47 No. 11, pp. 761-774, doi: 10.1016/j.infsof.2005.01.001.
Kohavi, R., Henne, R.M. and Sommerfield, D. (2007), “Practical guide to controlled experiments on the web”, Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining – KDD ’07, ACM Press, New York, NY, pp. 959-967, doi: 10.1145/1281192.1281295.
Kohli, R. and Melville, N.P. (2019), “Digital innovation: a review and synthesis”, Information Systems Journal, Vol. 29 No. 1, pp. 200-223, doi: 10.1111/isj.12193.
Krippendorff, K. (2004), Content Analysis: An Introduction to Its Methodology, Second, Sage, Thousand Oaks, CA.
Kumar, V. and LaConte, V. (2012), 101 Design Methods: A Structured Approach for Driving Innovation in Your Organization, John Wiley and Sons, Hoboken, NJ.
Li, Y. and Ma, L. (2019), “What drives the governance of ridesharing? A fuzzy-set QCA of local regulations in China”, Policy Sciences, Vol. 52 No. 4, pp. 601-624, doi: 10.1007/s11077-019-09359-x.
Maguire, M. (2001), “Methods to support human-centred design”, International Journal of Human-Computer Studies, Vol. 55 No. 4, pp. 587-634, doi: 10.1006/ijhc.2001.0503.
Mamasioulas, A., Mourtzis, D. and Chryssolouris, G. (2020), “A manufacturing innovation overview: concepts, models and metrics”, International Journal of Computer Integrated Manufacturing, Vol. 33 No. 8, pp. 769-791, doi: 10.1080/0951192X.2020.1780317.
Martin, B. and Hanington, B. (2012), Universal Methods of Design, Rockport Publisher, Beverly.
Misangyi, V.F. and Acharya, A.G. (2014), “Substitutes or complements? A configurational examination of corporate governance mechanisms”, Academy of Management Journal, Vol. 57 No. 6, pp. 1681-1705, doi: 10.5465/amj.2012.0728.
Mourtzis, D. (2020), “Simulation in the design and operation of manufacturing systems: state of the art and new trends”, International Journal of Production Research, Vol. 58 No. 7, pp. 1927-1949, doi: 10.1080/00207543.2019.1636321.
Mourtzis, D. (2021), Design and Operation of Production Networks for Mass Personalization in the Era of Cloud Technology, 1st ed., Elsevier, doi: 10.1016/C2019-0-05325-3.
Myers, M.D. (2009), Qualitative Research in Business and Management, 1st ed., Sage, London.
Nambisan, S., Lyytinen, K., Majchrzak, A. and Song, M. (2017), “Digital innovation management: reinventing innovation management research in a digital world”, MIS Quarterly, Vol. 41 No. 1, pp. 223-238, doi: 10.25300/MISQ/2017/41:1.03.
Nickerson, R.C., Varshney, U. and Muntermann, J. (2013), “A method for taxonomy development and its application in information systems”, European Journal of Information Systems, Vol. 22 No. 3, pp. 336-359, doi: 10.1057/ejis.2012.26.
Oleson, A., Solomon, M. and Ko, A.J. (2020), “Computing students’ learning difficulties in HCI education”, Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, pp. 1-14, doi: 10.1145/3313831.3376149.
Park, Y., El Sawy, O.A. and Fiss, P.C. (2017), “The role of business intelligence and communication technologies in organizational agility: a configurational approach”, Journal of the Association for Information Systems, Vol. 18 No. 9, pp. 648-686, doi: 10.17705/1jais.00001.
Park, Y., Fiss, P.C. and Sawy, O.E. (2020), “Theorizing the multiplicity of digital phenomena: the ecology of configurations, causal recipes, and guidelines for applying QCA”, MIS Quarterly, Vol. 44 No. 4.
Porcheron, M., Fischer, J.E. and Reeves, S. (2021), “Pulling back the curtain on the wizards of Oz”, Proceedings of the ACM on Human-Computer Interaction, Vol. 4 No. CSCW3, pp. 1-22, doi: 10.1145/3432942.
Ragin, C.C. (2008), Redesigning Social Inquiry: Fuzzy Sets and Beyond, University of Chicago Press, Chicago and London.
Ragin, C.C. (2017), User’s Guide to Fuzzy-Set/Qualitative Comparative Analysis, Department of Sociology University of CA, Irvine, CA.
Recker, J., Von Briel, F., Yoo, Y., Nagaraj, V. and McManus, M. (2023), “Orchestrating human-machine designer ensembles during product innovation”, California Management Review, Vol. 65 No. 3, pp. 27-47, doi: 10.1177/00081256231170028.
Rihoux, B. and Ragin, C.C. (2008), Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques, Sage Publications, Sage, Thousand Oaks, CA.
Rivard, K. and Faste, H. (2012), “How learning works in design education: educating for creative awareness through formative reflexivity”, Proceedings of the Designing Interactive Systems Conference, ACM, New York, NY, pp. 298-307, doi: 10.1145/2317956.2318002.
Rivero, L. and Conte, T. (2017), “A systematic mapping study on research contributions on UX evaluation technologies”, Proceedings of the XVI Brazilian Symposium on Human Factors in Computing Systems, presented at the IHC 2017: Brazilian Symposium on Human Factors in Computing Systems, ACM, Joinville Brazil, pp. 1-10, doi: 10.1145/3160504.3160512.
Roedl, D.J. and Stolterman, E. (2013), “Design research at CHI and its applicability to design practice”, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM, New York, NY, pp. 1951-1954, doi: 10.1145/2470654.2466257.
Roschuni, C., Kramer, J., Qian, Z., Zakskorn, L. and Agogino, A. (2015), “Design talking: an ontology of design methods to support a common language of design”, Proceedings of the 20th International Conference on Engineering Design (ICED 15).
Sanders, E.B.-N., Brandt, E. and Binder, T. (2010), “A framework for organizing the tools and techniques of participatory design”, Proceedings of the 11th Biennial Participatory Design Conference on – PDC ‘10, pp. 195-198, doi: 10.1145/1900441.1900476.
Sarbu, M. (2022), “The impact of industry 4.0 on innovation performance: insights from German manufacturing and service firms”, Technovation, Vol. 113, p. 102415, doi: 10.1016/j.technovation.2021.102415.
Satwekar, A., Volpentesta, T., Spagnoletti, P. and Rossi, M. (2023), “An orchestration framework for digital innovation: lessons from the healthcare industry”, IEEE Transactions on Engineering Management, Vol. 70 No. 7, pp. 2465-2479, doi: 10.1109/TEM.2022.3167259.
Schultze, U. and Avital, M. (2011), “Designing interviews to generate rich data for information systems research”, Information and Organization, Vol. 21 No. 1, pp. 1-16, doi: 10.1016/j.infoandorg.2010.11.001.
Seidel, S., Berente, N., Nickerson, J. and Yepes, G. (2023), “Designing the Metaverse”, Proceedings of the 55th HI International Conference on System Sciences, pp. 1-10.
Stolterman, E. and Pierce, J. (2012), “Design tools in practice: studying the designer-tool relationship in interaction design”, Proceedings of the Designing Interactive Systems Conference, presented at the DIS ’12: Designing Interactive Systems Conference 2012, ACM, Newcastle Upon Tyne United Kingdom, pp. 25-28, doi: 10.1145/2317956.2317961.
Szopinski, D., Schoormann, T. and Kundisch, D. (2019), “Because your taxonomy is worth it: towards a framework for taxonomy evaluation”, Proceedings of the 27th European Conference on Information Systems (ECIS), Stockholm and Uppsala, Sweden, pp. 1-20.
Thai, T.D.-H. and Wang, T. (2020), “Investigating the effect of social endorsement on customer brand relationships by using statistical analysis and fuzzy set qualitative comparative analysis (fsQCA)”, Computers in Human Behavior, Vol. 113 No. September 2019, p. 106499, doi: 10.1016/j.chb.2020.106499.
Tohidi, M., Buxton, W., Baecker, R. and Sellen, A. (2006), “User sketches: a quick, inexpensive, and effective way to elicit more reflective user feedback”, Proceedings of the 4th Nordic Conference on Human-Computer Interaction Changing Roles – NordiCHI ‘06, ACM Press, Oslo, pp. 105-114, doi: 10.1145/1182475.1182487.
Vermeeren, A.P.O.S., Law, E.L.-C., Roto, V., Obrist, M., Hoonhout, J. and Väänänen-Vainio-Mattila, K. (2010), “User experience evaluation methods: current state and development needs”, Proceedings of the 6th Nordic Conference on Human-Computer Interaction Extending Boundaries – NordiCHI ’10, ACM Press, Reykjavik, Iceland, pp. 521-530, doi: 10.1145/1868914.1868973.
Vigo, M., Bail, S., Jay, C. and Stevens, R. (2014), “Overcoming the pitfalls of ontology authoring: strategies and implications for tool design”, International Journal of Human-Computer Studies, Vol. 72 No. 12, pp. 835-845, doi: 10.1016/j.ijhcs.2014.07.005.
Werder, K., Seidel, S., Recker, J., Berente, N., Gibbs, J., Abboud, N. and Benzeghadi, Y. (2020), “Data-driven, data-informed, data-augmented: how Ubisoft’s Ghost Recon Wildlands live unit uses data for continuous product innovation”, California Management Review, Vol. 62 No. 3, pp. 86-102, doi: 10.1177/0008125620915290.
Woolrych, A., Hornbæk, K., Frøkjær, E. and Cockton, G. (2011), “Ingredients and meals rather than recipes: a proposal for research that does not treat usability evaluation methods as indivisible wholes”, International Journal of Human-Computer Interaction, Vol. 27 No. 10, pp. 940-970, doi: 10.1080/10447318.2011.555314.
Yoo, Y., Henfridsson, O. and Lyytinen, K. (2010), “Research commentary – the new organizing logic of digital innovation: an agenda for information systems research”, Information Systems Research, Vol. 21 No. 4, pp. 724-735, doi: 10.1287/isre.1100.0322.
Zhang, X. and Wakkary, R. (2014), “Understanding the role of designers’ personal experiences in interaction design practice”, Proceedings of the 2014 Conference on Designing Interactive Systems, ACM, New York, NY, pp. 895-904, doi: 10.1145/2598510.2598556.
Zobel, B., Werder, K., Berkemeier, L. and Thomas, O. (2019), “The role of early user participation in discovering software – a case study from the context of smart glasses”, 14th International Conference on Wirtschaftsinformatik, pp. 1235-1349.
Acknowledgements
This work is supported by National Natural Science Foundation of China (No. 62207024), Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant.