How leaderboard positions shape our motivation: the impact of competence satisfaction and competence frustration on motivation in a gamified crowdsourcing task

Abstract

Purpose

Gamification is a booming motivational approach in information systems. Leaderboards play a key role in gamification; however, there are mixed findings regarding the heterogeneous motivational impacts of leaderboard positions. This study aims to clarify the motivational effects of high and low leaderboard positions by assembling diverse behavioral measures and self-reports. The measures used in this study shed a light on the quantitative and qualitative dynamics of motivation facilitated by leaderboard positions. The authors inspect motivation in relation to satisfaction and frustration of competence need.

Design/methodology/approach

The authors conducted an online experiment set in a crowdsourcing context, asking the participants to compete in an image tagging game. Participants' leaderboard positions were manipulated to be either high or low for five consecutive rounds. The number of clicks, tags, duration of tagging and persistence on the task were measured as indicators of motivation.

Findings

High ranks on leaderboards induced complacent behaviors choosing easy ways to maintain their positions, while low ranks led the participants to stick to the right process of the task with intensified motivation round after round. However, neither of the motivations seemed to be of intrinsic nature.

Originality/value

The present study provides conclusive evidence on the varying motivational impact of leaderboard positions. The authors also demonstrate how the “needs-as-motive” model (Sheldon and Gunz, 2009) applies to gamification. Its implications in self-determination theory and gamification literature are discussed.

Citation

Na, K. and Han, K. (2023), "How leaderboard positions shape our motivation: the impact of competence satisfaction and competence frustration on motivation in a gamified crowdsourcing task", Internet Research, Vol. 33 No. 7, pp. 1-18. https://doi.org/10.1108/INTR-12-2021-0897

1. Introduction

Have you ever run an extra mile to earn a badge on your running app? How about collecting points to get a reward in your favorite store's loyalty program? Or have you spent more time studying to be in first place on your language learning app? If you have done one of these, you are already familiar with gamification. Gamification, a technique to deploy the motivational pull of games on non-game tasks, has become common around us. The gamification market was valued at 9.1 billion dollars in 2020 and is projected to grow up to 30.7 billion dollars by 2025 (Markets and Markets, 2020). Since the COVID-19 outbreak, many aspects of our lives need to be done remotely, so this change of lifestyle is likely to steepen the growth trend. Gamification is “the use of game elements in non-game contexts” (Deterding et al., 2011, p. 10). Its purpose is to infuse motivation into mundane tasks and induce higher engagement and performance. Various industries have adopted gamification to engage their users: government services and public engagement, work, crowdsourcing, commerce, health, education, environmental behavior, marketing, advertising and so on (see Koivisto and Hamari, 2019). Gamification has created a major buzz in academia as well. Many researchers have strived to reveal the psychological mechanisms behind the motivational phenomenon of gamification. Recent research is delving into the effects of individual game elements, as understanding the precise psychological function of each game element will be key to designing gamification effectively (Hamari et al., 2014; Seaborn and Fels, 2015).

One of the most widely used and frequently studied game elements is leaderboard. It displays players' rankings as feedback, showing how one is doing compared to others. Leaderboards have yielded positive results in various contexts. However, there are voices of concern that the effect of leaderboards is not all positive. According to some studies, leaderboards have mixed or negative effects (Domínguez et al., 2013; Hanus and Fox, 2015). An apprehension regarding leaderboards is that while high-rank positions on leaderboards positively affect motivation, low-rank positions produce detrimental effects on motivation (Domínguez et al., 2013). These heterogeneous effects of leaderboard positions may explain the mixed findings of the preceding studies. The current study aims to examine this possibility and clarify how leaderboard positions influence a user's motivation in a gamified crowdsourced task, especially with continuous exposure to positions. We also examine both the qualitative and quantitative facets of motivational change invoked by leaderboard positions. Self-determination theory (SDT) provides a theoretical lens for this research, as we focus on the aftereffects of competence satisfaction and frustration caused by leaderboard positions.

1.1 The motivational impacts of leaderboard positions

Whether leaderboards are solid tools for promoting motivation remains undetermined. The known positive effects of leaderboards include heightening user engagement (Bowey et al., 2015), enhancing performance (Landers et al., 2017; Mekler et al., 2017) and increasing participation (Farzan et al., 2008). On the other hand, some studies have warned of the side effects of excessive competition that leaderboards evoke (Hamari et al., 2014; Hanus and Fox, 2015). These ambivalent empirical results require answers as to “why” and “how” leaderboards fail. One possible explanation is the heterogeneous effects of leaderboard positions on motivation.

The motivational effect of leaderboard positions has mostly been studied in full-fledged game contexts. Butler (2013) manipulated the rankings in a shooting game and compared the fun ratings and replay rates between the high and low ranks. In his study, ranks were not correlated with fun ratings, and players in the lower ranks were more inclined to replay the game. Bowey et al. (2015) measured how positions on leaderboards affect perceptions of success and failure. Leaderboard positions as a within-participant factor, players reported that they felt more competent, more autonomous, more immersed, and found the game more enjoyable when they were in the higher ranks. Sun et al. (2015) found that players who came in 2nd, 4th and 7th rated their satisfaction and enjoyment of the game significantly higher than other positions and were more willing to replay and recommend the game. Their findings implied that satisfaction with the results depends on one's thoughts of “what could have been”.

Gamification research has been paying attention to the role of leaderboard positions as well. Jia et al. (2017) tested the relationship between the Big Five personality traits, gamified domain and leaderboard positions. Although users generally preferred high ranks to middle and low ranks, in the case of extroverted and agreeable people, motivation did not drop significantly even when they found their names on the bottom of leaderboards. Nebel et al. (2016) investigated the impact of leaderboard positions on various aspects of players' subjective experiences and learning outcomes in an educational video game. Notable findings from this study were that learners in the low ranks reported higher competitive effort and displayed better retention than learners who ranked high. The researchers interpreted that the amount of discrepancy between the learners' current position and desired goal made a difference in the amount of effort and learning outcomes. Bai et al. (2021) conducted two studies regarding the effects of top, middle and bottom positions on intrinsic motivation, learning engagement and learning performance; one with an absolute leaderboard and the other with a relative leaderboard. In the absolute leaderboard study, although the top third learners were more intrinsically motivated, this did not lead to higher engagement or better performance. Meanwhile, in the relative leaderboard study, all learners were equally intrinsically motivated and engaged, but the learning outcome was aligned with the leaderboard positions.

Overall, prior research on leaderboard positions is conflicting. The dependent measures used in the studies differed from one another, and they relied heavily on participants' subjective ratings. This may have produced inconsistent results regarding the effects of leaderboard positions. Another overlooked issue is that these studies collected responses after a single presentation of leaderboards. In a gamified system, users are repeatedly presented with leaderboard results, and their motivation varies accordingly. Therefore, to firmly establish the motivational effect of leaderboard positions, objective measurements after repeated exposure to leaderboards are better suited. The current study aims to investigate how leaderboard positions shape motivation in multiple rounds of a crowdsourced task that leaderboards follow after each round. This research uses diverse behavioral measures to examine the impact of leaderboard positions on the amount and quality of motivation.

Previous research has explored possible moderators of leaderboard effects, such as individual user characteristics. Trait competitiveness has been identified as a candidate. Höllig et al. (2020) reported that users' personal development competitiveness positively predicted the perception of leaderboards and usage intention. Landers et al. (2019) hypothesized that trait competitiveness moderates the influence of leaderboards on intrinsic motivation, but their hypothesis was not supported. Amo et al. (2020) observed how leaderboards stimulate users' trait competitiveness in terms of engagement and performance. The researchers learned that leaderboards make highly competitive users engage and spend more time on the task, while low competitive users are driven away by the use of leaderboards. Extant research has looked into the moderation of trait competitiveness on leaderboards without considering the potential heterogeneous effects of leaderboard positions. With this background, we examine the moderating role of trait competitiveness on the relationship between leaderboard positions and motivation.

The outlined studies explored the effect of leaderboard positions by sorting top and bottom positions, some including middle positions. Following this classification, this study manipulates the positions fixed to either high or low throughout the rounds. The positions on the leaderboards would fluctuate in an actual gamified system, however, for the sake of research, we decided to adopt this rather artificial treatment. In this way, we will be able to manipulate users' perceived competence through leaderboards and see the influence of competence satisfaction and frustration on motivation. In the next section, we discuss how leaderboard positions can be a source of competence satisfaction and frustration.

1.2 Need for competence and motivation

1.2.1 Psychological needs and motivation

Many psychological mechanisms are at play in the use of leaderboards. Sailer et al. (2017) name competition as the main principle of leaderboards, while Landers et al. (2017) suggest that leaderboards have a goal-setting function. However, the need for competence most comprehensively explains how leaderboards motivate and engage users. The need for competence as one of human's basic psychological needs originated from Self-determination theory (SDT; Deci and Ryan, 2000). SDT is a well-established macro theory of human motivation, instrumental in unraveling the motivational power of gamification (Deterding, 2015; Seaborn and Fels, 2015). According to SDT, human motivation can be conceptualized into two types: intrinsic motivation, which stems from pure enjoyment of the task itself, and extrinsic motivation, which is facilitated by external pressures such as rewards or punishments (Ryan and Deci, 2000b).

SDT postulates three basic psychological needs as the source of intrinsic motivation: the need for autonomy, the need for competence and the need for relatedness. Autonomy refers to the desire to choose an activity in accordance with one's values and interests, and to act on one's volition (Deci and Ryan, 2000; Sailer et al., 2017). Relatedness can be defined as the desire to emotionally interact and connect with others or a sense of belonging (Deci and Ryan, 2000; Ryan and Deci, 2000b). Lastly, competence is the desire to achieve mastery and feel effective in one's environment (Deci and Ryan, 2000). While the mainstream SDT posits that supporting the psychological needs is the way to arouse motivation, Sheldon and Gunz (2009) suggested that unmet needs also spark motivation and actions to remedy the unfulfilled need state. They expounded that psychological needs are not just requirements to bring about motivation but also autotelic motives that oftentimes elicit behavior. In this sense, a two-process model to harmonize “needs-as-requirements” and “needs-as-motives” perspectives was later proposed (Sheldon, 2011). The majority of gamification research takes on the traditional view of needs as requirements for motivation. However, to better understand the motivational potential of gamification and how the psychological needs contribute to it, the needs-as-motives model should also be taken into consideration. In this study, we aim to explore whether the needs-as-motive approach applies to gamification, particularly with competence needs related to leaderboard positions, and identify the nature of motivation spawned by the positions.

1.2.2 The quantitative and qualitative dynamics of motivation

For gamification to support positive and long-lasting behavioral changes and emotional well-being, we must study both the quantitative and qualitative aspects of its motivational influence. Many theories of motivation have viewed it as a unitary concept, focusing more on its amount or intensity of it (Deci and Ryan, 2012; Ryan and Deci, 2000a). However, SDT theorists opt to focus more on the orientation of motivation, i.e. “the underlying attitudes and goals that give rise to action (Ryan and Deci, 2000a, p. 54)”. They emphasize the reason behind the motivation for they are effective in predicting people's quality of engagement, performance and well-being (Deci and Ryan, 2012). Although both intrinsic and extrinsic motivations promote performance (Cerasoli et al., 2014), only intrinsic motivation predicts meaningful engagement (Walker et al., 2006) and long-lived positive effects (Liu et al., 2019). Therefore, to understand what makes effective gamification, it is imperative to inspect motivation from diverse perspectives.

As motivation, “the psychological force that enables action (Touré-Tillery and Fishbach, 2014, p. 328)”, is an intangible psychological construct, numerous research has endorsed using subjective self-reports as the measurement of motivation. However, when attempting to capture changes in motivation, using questionnaires repeatedly is inappropriate. Touré-Tillery and Fishbach (2014) suggested various behavioral measures to infer motivation which can also indicate its intrinsic or extrinsic nature. They proposed two dimensions of motivation: outcome-focused and process-focused motivation. Outcome-focused motivation is the type of motivation that moves one to concentrate the cognitive and physical resources on goal-congruent activities. Accordingly, a person with an outcome-focused motivation gets a lot of work done with high speed and accuracy. The authors explicated that this pertains to extrinsic motivation, as its purpose is often to gain external benefits. On the other hand, one with process-focused motivation enjoys the process of a task and adheres to the “proper” process. Process-focused motivation encompasses intrinsic motivation and means-focused motivation. Intrinsic motivation is typically measured using the Intrinsic Motivation Inventory (IMI; Ryan et al., 1983) and free-choice, i.e. how much one chooses to persist on a task. Means-focused motivation shares a similar measure, task duration, as one would spend longer time on a task to abide by goal-congruent rules, principles, etc. Of course, it entails high performance outcomes such as high accuracy and a large amount of work done. Observing motivation using the various behavioral measures mentioned above and complementing them with self-reports will allow us to probe into the quantitative and qualitative dynamics of motivation. This study intends to explore the intensity and orientation of motivation sparked by leaderboard positions in order to gain knowledge on how to properly motivate users with leaderboards.

1.2.3 Need for competence and leaderboards

Leaderboards are associated with the need for competence (Sailer et al., 2017). Naturally, receiving positive feedback, i.e. being on top of the leaderboard, will fulfill competence, but negative feedback, i.e. finding oneself at the bottom of the leaderboard, will thwart it. We can surmise how this need satisfaction and frustration will affect motivation from a needs-as-requirement viewpoint; supporting competence increases motivation and general well-being (Ryan and Deci, 2000b; Van den Broeck et al., 2016). Meanwhile, impeded competence need leads to disengagement, undermined motivation and ill-being (Jang et al., 2016; Vansteenkiste and Deci, 2003). Based on this reasoning, it is easy to presume that the high-rank positions on leaderboards would strengthen motivation, and the low ranks would weaken it. However, the needs-as-motives mechanism paints a different picture. In Sheldon and Gunz's 2009 study, the authors demonstrated that those who experienced positive need fulfillment were less prone to pursue further satisfaction of such need. In the case of need frustration, several studies have reported findings that motivation to seek restoration of the thwarted need ensues the need thwarting experience (Fang et al., 2018; Radel et al., 2011; Sheldon and Gunz, 2009).

Regarding the content of motivation facilitated by need satisfaction, need support is known to enhance intrinsic motivation and some autonomous types of extrinsic motivation (see Deci and Ryan, 2008; Vansteenkiste et al., 2009). As per studies on need thwarting and motivation to redeem the thwarted need, the researchers had more focus on the intensity of motivation than its orientation (Fang et al., 2018; Radel et al., 2011; Sheldon and Gunz, 2009). Therefore, it seems premature to determine how high and low leaderboard positions would shape motivation via competence satisfaction and frustration. Rather than hastily developing hypotheses, we present research questions related to the matter.

RQ1.

What are the impacts of high-rank and low-rank leaderboard positions on the amount and quality of motivation in an ongoing task?

Although not widely studied in SDT literature, it is plausible that competitive individuals have a stronger need for competence. Indyastuti et al. (2016) investigated the moderating effect of trait competitiveness on feelings of competence and found that those high in trait competitiveness had feelings of competence in a competitive environment while those low in trait competitiveness experienced the opposite. Therefore, it is likely that individuals with different levels of trait competitiveness will react differently upon exposure to high-rank or low-rank leaderboard results. Landers et al. (2017) failed to demonstrate the moderation of trait competitiveness in the relationship between leaderboards and intrinsic motivation; however, they did not see whether the moderating effect exists on leaderboard positions. Amo et al. (2020), in their study on how people interact with a leaderboard-implemented learning platform, revealed that highly competitive individuals invest more time to dominate the top positions, while those with low competitiveness shy away from participating. Their findings call for a need to further scrutinize the moderating role of trait competitiveness on the relationship between leaderboard positions and motivation. Thus, we propose the second research question.

RQ2.

Does trait competitiveness of individuals moderate the motivational impact of leaderboard positions?

2. Materials and methods

The purpose of this study is to clarify how leaderboard positions affect motivation for an ongoing task. We designed an online experiment in which the leaderboard was manipulated for a participant to receive either high or low-rank results consistently throughout the whole task. As mentioned, this was a measure taken to facilitate satisfaction and frustration of competence. We observed how this changed the participants' motivation. The given context of the task was to generate tags for interior design images. The participants were told that the images were to be used as preprocessed learning data for an interior curator AI. Tags were created by clicking on the images and writing words associated with the images, and participants played five rounds of this so-called image tagging game. Leaderboard positions, henceforth referred to as the ranks in this study, served as a between-participant variable, and the five rounds were a within-participant variable. The dependent variables were the number of input words as tags, clicks made on the images and duration of the time used to generate tags, which we refer to as the number of tags, the number of clicks and tagging duration, respectively. These measurements signify motivation, the rationale for which we will expand on in a later section. To measure persistence on the gamified task, we asked the participants how many more rounds they would like to take on after completing the five rounds but didn't actually make them play. This measure is referred to as the number of additional rounds. Furthermore, we collected survey data inquiring about the participants' subjective feelings of enjoyment, perceived competence, effort and task value to complement the analysis of motivation quality. Questionnaires measuring trait competitiveness were also included as a part of the survey.

2.1 Participants

A total of 125 undergraduate students participated in an online experiment for a psychology class course credit. However, 14 participants were unable to meet the study requirements and were excluded from the analysis. Hence, the final 111 participants were 36 male and 75 female students with a mean age of 21.44 (SD = 2.25). 90% of the participants were of Korean origin and used Korean language as their native tongue. The response data showed that none of the participants had a problem understanding the instructions in the experiment.

2.2 Apparatus

Data were collected via Qualtrics. Since it was an online experiment, the students participated on their personal computers using the link provided. The participants were requested to use their computers instead of other devices, and the specifications of the devices were checked with metadata collected on Qualtrics. The monitor resolution varied from 1180×820 to 1920×1080, all of which were within the range of being able to see and click on the images without experiencing any difficulty.

2.3 Materials and measures

2.3.1 Image tagging task

We introduced the image tagging used by Mekler et al. (2017) as the experimental task. Mekler et al. (2017) analyzed the number of tags generated for abstract painting pictures and evaluated the quality of the tags. Leaderboards were incorporated into their study design and were reported to be effective in increasing the number of tags. Therefore, we believe that image tagging is well-suited for collecting evidence on the quantitative and qualitative motivational changes yielded from leaderboard positions. 15 interior design pictures were presented as stimuli, each image containing a cluster of objects found in households. An example of these images is shown in Figure 1.

2.3.2 Leaderboards

After each round, leaderboards containing the ranks, nicknames and scores of 10 players including the participant were displayed. The high-rank leaderboards had the participants placed in either 1st or 2nd place, and on the low-rank leaderboards, the participants ranked either 9th or 10th place. Each round randomly presented one of the two places. This measure was taken to prevent suspicion that the leaderboards were rigged. The accumulated scores were displayed on the leaderboards, but the score gap between 1st place and the last was maintained between 8 and 12 points. This was to avoid letting the participants give up thinking that there is no room for competition. A sample of leaderboards for the high-rank and low-rank conditions is shown in Figure 2. It was essential that the participants notice their positions on leaderboards, so we asked the participants if they remembered their ranks in the final round as a manipulation check.

2.3.3 Measures

Tags were made of words describing the presented image, and the number of tags split by commas was counted. For the number of clicks, we recorded how many clicks were made on the image during the 10 s it was displayed. Tagging duration was calculated by subtracting the seconds until the first click from the page submission point. In addition, participants were provided with a chance to challenge bonus rounds after the regular rounds and were asked how many more rounds they wanted to add. This was an abridged version of the free-choice method, as the number of additional rounds the participants elected to challenge showed their motivation to persist and engage in the task.

As Touré-Tillery and Fishbach (2014) delineated, behavioral measures such as the amount of work done, accuracy, duration (i.e. time spent on the task) and persistence yield different results depending on the orientation of motivation. As for the number of clicks and the number of tags, they fit as the amount of work done measures. However, these two represent different aspects of motivation, as generating tags requires much higher cognitive effort, e.g. memory activation and verbal elaboration. Plus, since the given task was “image tagging”, writing more tags is abiding by the proper means. Thus, we say that tagging is more means-focused while clicking is more outcome-focused. How much effort the participants allocated to writing tags and clicking tells us about the nature of their motivation. Tagging duration, of course, was used as the duration indicator, and the number of additional rounds was used as the persistence indicator.

In addition to the behavioral measures, we collected the participants' subjective responses regarding the various facets of their motivations toward the task. The questionnaire items were extracted from the Intrinsic Motivation Inventory. The constructs measured by the items were perceived enjoyment (4 items, Cronbach's α = 0.89), competence (4 items, α = 0.82), effort (3 items, α = 0.86) and task value (4 items, α = 0.84). Sample items for each construct were: “This image tagging task was fun to do (enjoyment)”, “I think I am pretty good at this image tagging task (competence)”, “It was important to me to do well in this task (effort)” and “I think this is an important activity (task value).” The items were rated on a 7-point Likert scale. The Trait Competitiveness Questionnaire (Gill, 1986) was used to measure individual competitiveness. The questionnaire included five items (α = 0.81), for example, “I enjoy competing with other people.”, also measured on a 7-point scale. All the questionnaire items were translated and adapted into Korean.

2.4 Procedure

Participants who signed up for the study were provided with an online link to the experiment. Participants were informed that they were taking part in an image tagging game for labeled image data to be learned by an interior design curation AI. The study procedures were approved by the Institutional Review Board of the authors' affiliation.

The participants were given tutorials to learn the rules of the game. The tutorials explained how to click on the objects in the picture, how to input words associated with the image, and that each click and tag will be given a point. The tutorial said that participants would be competing with the records of nine other participants who had already finished the game. There were five rounds in the game, and each round consisted of three different images to be tagged. 10 s were given for the participants to click on the image, and then a text input field was shown with instructions to write tags related to the image. Words separated by a comma were regarded as 1 tag, and 1 point was given per tag. After a round, a leaderboard was displayed with the ranks, scores, and nicknames of the participant and nine other fictitious competitors. Tagging and leaderboard showing were repeated for the remaining four rounds.

After the five rounds, participants were asked whether they wanted to play additional rounds to gain more points, and if they did, how many. A questionnaire regarding the subjective experience of the game was administered. When the participants completed the survey, they were debriefed on the original purpose of the study and were granted course credits.

3. Results

3.1 The number of tags, the number of clicks and tagging duration

Since the study had a 2(rank: high, low) × 5(rounds: 1, 2, 3, 4, 5) design, a two-way mixed-model ANOVA was conducted. Beforehand, natural logarithm transformation corrected the left-skewness of the number of tags and the number of clicks. The distribution of tagging duration was highly left-skewed; thus a common logarithm transformation was implemented to correct the skewness. Mauchly's test of sphericity revealed a violation of sphericity for all three dependent variables; therefore, we report the Greenhouse–Geisser correction results. First, the main effect of rounds on the number of tags was significant, F(2.26, 245.76) = 5.10, p = 0.005, ƞ² = 0.05, while the main effect of ranks was not significant, F(1, 109) = 0.96, p = 0.33, ƞ² = 0.009. More importantly, the interaction between rounds and ranks was significant, F(2.26, 245.76) = 5.29, p = 0.004, ƞ² = 0.05. As for the number of clicks, the result revealed a significant main effect of rounds, F(2.83, 308.09) = 30.52, p = 0.00, ƞ² = 0.22. However, the main effect of ranks was not significant, F(1, 109) = 0.01, p = 0.91, ƞ² = 0.00; and the interaction between rounds and ranks was insignificant by a marginal difference, F(2.83, 308.09) = 2.63, p = 0.06, ƞ² = 0.03. Lastly, tagging duration yielded a pattern similar to the number of tags; significant main effect of rounds, F(2.78, 303.45) = 4.62, p = 0.005, ƞ² = 0.04, no main effect of ranks, F(1,109) = 2.7, p = 0.10 and the interaction of rounds and ranks was significant, F(2.38, 303.45) = 4.96, p = 0.003, ƞ² = 0.05. As shown in the line graphs in Figure 3, the leaderboard positions made a difference in the participants' behaviors and motivation.

We conducted pairwise t-tests comparing the rounds to take a deeper look into the motivational changes depending on the leaderboard positions. First, the change in the number of tags showed a pronounced difference between the high-rank and the low-rank groups. A comparison between round 1 and round 5 number of tags revealed a significant uptrend for the low-rank group, t(56) = 3.55, p = 0.001, whereas the high-rank group did not show a significant change, t(53) = −0.67, p = 0.51. Secondly, the number of clicks took a significant leap from round 1 to round 5 for both the high and low rankers, t(53) = 3.21, p = 0.002; t(56) = 6.70, p = 0.00. In terms of tagging duration, it was notable that the high-rank group spent significantly less time tagging in round 5 than in round 1, t(53) = 3.87, p = 0.00. On the other hand, the low-rank group did not display a significant change in tagging duration, t(56) = 0.39, p = 0.70, despite the rising trend from round 1 to round 4 apparent on the line graph (Figure 3, bottom).

Synthesizing the changes in the number of tags, the number of clicks and tagging duration tells us that the high-rank group, while maintaining their effort in clicking, withdrew from tagging, which is more demanding work. This indicates that the high-rank group was more focused on obtaining easy points than on the task purpose. Contrastingly, the low-rank group kept working harder on both tagging and clicking, which says that they endeavored for both the outcome and the means of the task. Moreover, it is important to pay attention to the downward movements of the low-rankers after round 4, in both tags, clicks and tagging duration, as they may hint that participants in the low-rank condition were about to give up because of the chain of discouraging events.

3.2 The number of additional rounds

First, we checked the number of participants who decided to undergo the additional rounds. 59.3 and 64.9% of the participants in the high and low rank groups, respectively, accepted the additional rounds. However, this was not a significant difference, χ²(1, N = 111) = 0.38, p = 0.54. We then analyzed the number of rounds that the participants were willing to take on. For participants who refused the challenge, the number of additional rounds was regarded as 0. Due to abnormal distribution, we used the Mann–Whitney U test to compare the high and low ranks. The result was insignificant, U = 1,346, N₁ = 54, N₂ = 57, p = 0.22, two-tailed test, failing to prove differences in persistence between the high-rank and low-rank groups. As shown in Table 1, participants in the low-rank group played slightly more rounds on average than those in the high-rank group, but the difference in the median was not statistically significant.

3.3 Moderation by trait competitiveness

We used SPSS PROCESS macro ver. 3.5 (Model 1, bootstrapped samples = 5,000; Hayes, 2013) to analyze the moderation of trait competitiveness. Trait competitiveness was measured on a continuous scale, therefore we eliminated 4 cases above and below 2SD from the mean and centered the trait competitiveness scores. An independent t-test confirmed that there was no significant difference between the high- and low-rank groups in trait competitiveness, t(105) = 0.63, p = 0.53, which established that early exposure to leaderboard positions did not affect the results of the trait competitiveness questionnaire. In a set of regression analyses, rank was fitted as the predictor variable, and the amount of fluctuation in clicks, tags and tagging duration were used as the outcome variables. In other words, round 1 and round 5 number of tags discrepancy (henceforth referred to as R1–R5 tags), number of clicks discrepancy (R1–R5 clicks) and duration discrepancy (R1–R5 duration) were evaluated. With R1–R5 tags, the overall model explained 8% of the variance, R² = 0.08, F(3, 103) = 2.79, p = 0.05. Rank significantly predicted the change in tags, β = 5.71, SE = 2.33, t(103) = 2.46, p = 0.02, but trait competitiveness and rank × trait competitiveness interaction was insignificant, β = 0.83, SE = 1.08, t(103) = 0.77, p = 0.44; β = −2.82, SE = −2.15 t(103) = −1.31, p = 0.19. As for R1–R5 clicks and R1–R5 duration, the moderation model was not significant, R² = 0.03, F(3, 103) = 1.18, p = 0.32; R² = 0.02, F(3, 103) = 0.77, p = 0.52.

We also examined whether trait competitiveness moderated the effect of ranks on the number of additional rounds. The overall model was insignificant, R² = 0.07, F(3, 103) = 2.43, p = 0.07, however, we took a closer look for trends. Ranks, although it was close, did not reach significance in predicting the number of additional rounds, β = 0.63, SE = 0.33, t(103) = 1.91, p = 0.06. Trait competitiveness was insignificant, β = −0.70, SE = 0.47, t(103) = −1.49, p = 0.14. And the interaction between rank and trait competitiveness was insignificant as well, β = 0.55, SE = 0.31, t(103) = 1.82, p = 0.07. Simple slope analysis indicated that for participants with low competitiveness (−1SD), rank did not matter, β = 0.03, SE = 0.47, p = 0.96, 95% C.I. (−0.90, 0.95), but highly competitive participants (+1SD) challenged more additional rounds when they kept getting low-ranks, β = 1.23, SE = 0.47, p = 0.01, 95% C.I. (0.30, 2.15). Despite the trend, trait competitiveness did not significantly moderate the effect of ranks on task persistence.

3.4 Perceived enjoyment, competence, effort, task value

An independent samples t-test was conducted on each subconstruct of intrinsic motivation: enjoyment, competence, effort and task value. The perceived enjoyment of the high and low-rank groups did not differ significantly, t(109) = 1.39, p = 0.17. However, participants in the high-rank group did indeed perceive higher fulfillment of competence, t(109) = 6.39, p = 0.00, which verifies that the rank manipulation successfully induced competence satisfaction and frustration. Moreover, the high-rank group participants believed that they put more effort into playing the game than did the low-rank group, t(102.2) = 2.32, p = 0.02. Despite the different subjective efforts, the high-rank and low-rank groups did not perceive the task value differently, t(109) = 0.69, p = 0.49. Descriptive statistics of the subjective variables are shown in Table 2. Additionally, we administered a correlation analysis to inspect how the behavioral measures and the subjective constructs are related. The change in the number of tags was positively correlated with enjoyment, effort and task value. Clicking was not correlated with any variable. Tagging duration had a positive correlation with enjoyment, while the persistence indicator, the number of additional rounds, was negatively correlated with competence. Lastly, the subjective measures were all significantly correlated with each other. The correlation table is presented in Table 3.

4. Discussion

This study investigated how leaderboard positions change user motivation in consecutive rounds of a gamified task. In five rounds of the image tagging task, the participants consistently received either high- or low-rank positions on the leaderboard. We examined how these positions affect the participants' motivation through behavioral measures and self-reports.

• RQ1. What are the impacts of high-rank and low-rank leaderboard positions on the amount and quality of motivation in an ongoing task?

The high-rank group was less invested in tagging than in clicking as rounds progressed, as shown in the number of tags, tagging duration and the number of clicks. Judging by these measures, the high-rank group leaned towards outcome-focused motivation, preferring ways to speedily achieve a bigger outcome. As for the low-rank group, they continuously increased their effort in both tagging and clicking, with longer duration and more work done round after round. It is fair to say that their behaviors displayed the characteristics of process-focused motivation described by Touré-Tillery and Fishbach (2014). Judging by the performance indicators, the lower positions on leaderboards elevated the level of motivation better than the higher positions did. Our results are in line with Nebel et al. (2016) that the low-rank learners performed better with higher perceived competitive effort. Persistence in the task did not statistically differ in our study, however, the descriptive statistics coincide with Butler's (2013) findings that more low-rankers replayed the game. We should also note that the participants in the high-rank group didn't raise their game after round 2, which is similar to Bai et al.’s (2021) study where the top rankers on the absolute leaderboards didn't show better learning performance in the post-test. It seems that getting top results on leaderboards creates a sense of complacency, thinking that the task is too easy for them (Sun et al., 2015). On the other hand, participants who received low-rank results decreased their effort in round 5 after their continued endeavors until round 4. This hints that efforts to overcome discouraging feedback can only be sustained if one perceives that there is still room for competition (Nebel et al., 2016).

The subjective measures provide more information regarding the nature of motivation. The high-rank group perceived higher competence than the low-rank group. The negative correlation between competence and the number of additional rounds implies that frustration, rather than satisfaction of competence is more pertinent to the motivation to persist on the task, as explicated by the needs-as-motives mechanism (Sheldon and Gunz, 2009). Enjoyment did not differ between the groups, although, according to descriptive statistics, participants in the low-rank experienced lower enjoyment in the task. This could mean that their performance in this task did not derive from intrinsic motivation. Instead, it could have been an ego-involved motivation, a form of controlling regulation (Ryan et al., 1991). Effort was correlated with the number of tags, yet, the high-rank group reported higher effort when they gradually put less effort into writing tags in actuality. Nevertheless, both groups rated their level of effort fairly high. Lastly, the two groups did not perceive the task value differently. It does not seem likely that internalization of regulation was the reason for motivation.

• RQ2. Does trait competitiveness of individuals moderate the motivational impact of leaderboard positions?

Moderation of trait competitiveness was not found for any measure, despite the marginal trend in the number of additional rounds. The achieved effect size for this particular dependent variable, the number of additional rounds, was smaller than we anticipated (Cohen's d = 0.36), and the small effect size resulted in low statistical power (1−β = 0.5). However, with a larger sample size, the moderation of trait competitiveness on the relationship between leaderboard positions and task persistence might still have been statistically proven.

4.1 Theoretical and practical implications

First, the present study contributes to the gamification literature, specifically regarding the effect of leaderboards. This study deployed various behavioral and subjective measures to provide information on the quantitative and qualitative aspects of motivation stimulated by leaderboard positions. Using repeated measures of performance indicators allowed us to capture the dynamic change in the amount of motivation for high and low leaderboard positions. Our study extends from Nebel et al. (2016) and Bai et al.’s (2021) research where the high ranks on an absolute leaderboard were not useful in promoting engagement and performance (Bai et al., 2021), while the low ranks boosted effort and performance (Nebel et al., 2016). The results of this study show that being continuously on the top of the leaderboard entices one to stop putting more effort into the task, whereas the bottom positions stir up increased effort with the motivation to climb up the leaderboard. We also found out that this result stemmed from different qualities of motivation, as high ranks invoked motivation to only secure the outcome, i.e. their positions, rather than focus on the task itself, and the low ranks motivated the players to do the task right by following due process. However, the motivations of the high-rankers and the low-rankers did not seem likely to be intrinsic ones. The high-rankers cared more about extrinsic feedback than their intrinsic satisfaction, and the low-rankers persisted in the task out of ego-involved motivation. The lack of intrinsic motivation from both groups observed in this study adds to the concerns that the motivation gamification brings about may not be intrinsic at all (Hanus and Fox, 2015; Mekler et al., 2017). Another possibility is that leaderboards start with intrinsic motivation and then turn to extrinsic motivation with prolonged play. If this is the case, it would be strongly linked to the inverted U-shape relationship between extended use and performance, a pattern often observed in gamification systems (Amo et al., 2020; Welbers et al., 2019; Yang and Li, 2021).

Secondly, our study demonstrated how the “needs-as-motives” model (Sheldon and Gunz, 2009) applies to gamification. Gamification research has been focused on how game mechanics satisfy the basic psychological needs. However, the results acquired in this study imply that gamification design should also consider how it would induce need deficits. Preceding research points out that need thwarting is followed by restorative motivation to reclaim need satisfaction (Fang et al., 2018; Radel et al., 2011). The majority view on the role of need frustration in games is that it is associated with negative outcomes (Kosa and Uysal, 2022), but when administered at a proper level, need frustration can help players to be more engaged in the game (Abuhamdeh et al., 2015). Fang et al. (2018) illustrated in their study that hindered competence in a prior activity can be succeeded by even stronger motivation in a separate, current activity, if the activity has the potential to revive competence. Our study suggests that such a succession of motivation can happen in the same activity, as long as one perceives that the need impediment can be overcome. We must also be cautious not to overjustify the needs. After experiencing a need-satisfying event, one can become less interested in further pursuit of such need (Sheldon and Gunz, 2009). Therefore, overindulgence of needs leads to complacent behaviors. To ensure optimal functioning, satisfaction and frustration of needs should be appropriately managed.

In that sense, practitioners can consider several measures to introduce the right amount of need frustration with leaderboards. There are many types of leaderboards to be utilized: absolute leaderboards (also referred to as the infinite or global leaderboards) that show the rankings of all players, relative leaderboards (also known as the local leaderboards) that show the rankings of adjacent players, multi-level leaderboards and so on. Jia et al. (2017) proposed a leaderboard model highlighting the dynamic changes in ranks to capture the attention of extroverted users. Relative leaderboards or multi-level leaderboards would be more suitable for such a purpose. Even an individual player leaderboard that lets one compete against oneself can be useful for a user's goal setting. As to how to place the user on leaderboards, Bai et al. (2021) revealed that informing the user of his or her exact position increases competitive effort, whereas hiding it creates a more cooperative atmosphere. Leaderboard designers should choose where to place the user depending on the application context. Refreshing leaderboards regularly can also help keep high-rank users interested in the gamified system. Adapting the leaderboards to user performance will also be necessary.

4.2 Limitations and future research

The current study's context and domain in which gamification was applied potentially limits the generalizability of the results. The laid out context of the experiment was participating in an image labeling task for AI learning data, essentially a crowdsourcing task. Also, this was a one-time experiment in which students participated for a course credit. If the task was personally important to the participants and if the leaderboard results had practical consequences, the motivational patterns might have been different. In other words, the personal importance of a task may moderate the degree of competence satisfaction and frustration stemming from the leaderboard positions. A deeper frustration of competence, in particular, might lead to early resignation from the game. Since the conditions for need fulfillment and thwarting are highly personal and situational (Deci and Ryan, 2008; Ryan and Deci, 2000b), future studies may conduct natural experiments in other domains to look into the motivational differences that leaderboard positions make.

This study focused only on competence need among the basic psychological needs, but investigating the relationship between leaderboard positions and all three needs would have been ideal. Autonomy need is key to predicting intrinsic motivation; therefore, we advise future studies to include measures of autonomy when examining the heterogeneous effects of leaderboard positions. Furthermore, since the current study tested only private leaderboards (i.e. the leaderboard results are not shared with others), studies on how public leaderboards and their positions influence relatedness need are called for. Since basic needs can be conflicted within a game element (van Roy and Zaman, 2019), exploring the relationship between leaderboard positions and the three basic needs is crucial. In addition, we were not able to analyze the accuracy measures as some participants did not speak Korean as their mother tongue. We used the number of tags and the number of clicks differentiation as an alternative, but an accuracy indicator would have provided more definitive evidence regarding the quality of motivation.

The leaderboard implemented in our study was manipulated to maintain an 8 to 12 score gap between the top and bottom positions. This might have had a huge influence, especially on the low-rankers, to think they still have a fighting chance. If the score gap was larger, the results might have been different. Moreover, the high-rank group and the low-rank group were given 1st or 2nd, and 9th or 10th place at random for each round, but being in 1st place and being 2nd makes a difference on the subsequent motivation (Sun et al., 2015). Hence, the places could have had a random influence on the results.

Lastly, we rejected the moderation model of trait competitiveness due to a small effect size followed by low statistical power. However, for researchers who are interested in a small effect size of trait competitiveness as a moderating individual factor, we encourage them to examine it by drawing a larger sample, in consideration of the motivational effects of leaderboard positions.

Figures

Figure 1

A sample of the presented images

Figure 2

Leaderboards used in the experiment (left: high-rank leaderboard/right: low-rank leaderboard)

Figure 3

The mean number of tags (upper left), the mean number of clicks (upper right) and the mean tagging duration (bottom) in line graphs

Table 1

Descriptive statistics for the number of additional rounds

Condition	Bonus round challenge		The number of additional rounds				N
	Accepted (%)	Refused (%)	Mean	SD	Median	Range
High-rank	59.3	40.7	0.74	0.81	1	4	54
Low-rank	64.9	35.1	1.35	2.23	1	10	57

Notes(s): Bonus round challenge rate showing the percentage of participants who accepted or refused the challenge in each condition, the mean, standard deviation, median and range of the number of additional rounds

Table 2

Descriptive statistics for the subscales of the intrinsic motivation inventory

Condition	N	Enjoyment		Competence		Effort		Task value
		Mean	SD	Mean	SD	Mean	SD	Mean	SD
High-rank	54	4.59	0.19	4.65***	0.15	5.73*	0.13	3.87	0.18
Low-rank	57	4.22	0.19	3.30***	0.15	5.23*	0.18	3.69	0.18

Note(s): *Subjective effort differed between rank conditions at p < 0.05

***The high-rank group perceived higher competence than the low-rank group at p < 0.001

Table 3

Correlations between the study variables

Variable name	1	2	3	4	5	6	7	8
(1) R1–R5 tags	1
(2) R1–R5 clicks	0.15	1
(3) R1–R5 duration	0.40**	−0.16	1
(4) The number of additional rounds	0.20*	−0.10	0.06	1
(5) Enjoyment	0.28**	−0.02	0.23*	0.07	1
(6) Competence	−0.04	−0.08	0.03	−0.21*	0.43**	1
(7) Effort	0.32**	−0.10	0.16	0.14	0.64**	0.37**	1
(8) Task value	0.22*	−0.08	0.12	0.02	0.69**	0.40**	0.59**	1

Note(s): *p < 0.05, two-tailed

**p < 0.01, two-tailed