Gamifying recruitment: The psychology of game-based assessment

The development and application of psychological assessment is considered one of the major achievements of psychologists in the last century. When applied appropriately, it can offer behavioural insight in many areas of decision-making and self-understanding otherwise inaccessible to other means.

Psychological testing has its origins in the civil service examinations of the Chinese dynasties as early as 2000 BC. It was World War I, however, that provided modern test development major impetus when America was faced with the task of selecting service members having entered the war without a large standing army. This led to the development of the Army Alpha and Army Beta tests and offered a basic template for the development of subsequent selection measures. Assessment remains to this day an important component of personnel selection across a broad range of job types and industries. Tests of cognitive ability and personality have consistently proven to be useful predictors of job success, and complementary to other selection methods. Combining a measure of general cognitive ability with a well-structured interview for instance can substantially increase the prediction of job performance beyond the power of either method alone (Schmidt & Hunter, 1998).

Technology’s impact

During the past few decades organisations have evolved the way they source, attract and develop talent as they strive to remain both competitive and relevant in ever-changing work environments. Technological advancement has been a driving force for such change, transforming many aspects of the workplace from online job application and applicant tracking systems to high-fidelity learning and development simulation. Bunderson et al. (1989) predicted that assessments of today would also evolve and be radically different from then current tests. Their assertion has in part been correct. During the 1980s, personal computers primarily enabled assessment professionals to automate the scoring of paper and pencil assessments via optical scanners. During the mid-1990s, test developers leveraged the affordability of powerful personal computers and the internet to transmit large quantities of data and start to experiment with video streaming, simulations and real time interactions. Two decades on, in spite of the significant advances in technology, measurement theory, and the eclectic nature of modern-day psychometric practice the vast majority of psychological assessments largely reflect traditional multiple-choice paper and pencil tests. The way in which assessments are delivered has changed, how assessments are constructed has changed much less.

Recent advances in digital experience have created a divide in user expectations and what test publishers can offer as part of their assessments. Everyday individuals now experience and demand dynamic online interactions that are tailored and personalised to their needs. Often these experiences are elegant and simplified to a point that they do not cause any friction or require any training to use. This is in contrast to the primary aim of early academic inquiry to understand equivalence and produce assessments that were as close as possible in nature to their paper and pencil variants.

Gamification and its application to psychological testing is, at least in recent times, a notable exception to this. With technological innovation rapidly advancing and employers adopting latest technology to differentiate positively in their respective markets, test publishers are looking at ways of gamifying their test experience in an attempt to keep pace. Over the past few years, gamification has garnered much attention from business and industry as a means of enhancing user engagement, increasing customer activity and impacting positively on service use.

This is not lost on the psychology community, with members of the Society for Industrial and Organisational Psychology recently including ‘gamification’ high amongst a list of top workplace trends. Gamification can be defined as the process of applying game mechanics in non-game contexts (Zichermann & Linder, 2013). For example, popular game mechanics such as providing progress indicators, clear goals, rules, rewards and instant feedback are engineered into foreign systems (e.g., customer relationship management platforms, training modules) to elicit desired behavioural outcomes.

The rise of gamification

With traditional roots in behavioural economics, gamification is certainly not new. Its application though has mainly been for motivational purposes. Consequently it can be seen in a myriad of activities, from tedious repetitive tasks such as filling in online forms (e.g., LinkedIn) to strenuous exercise programs that encourage better health and wellbeing (e.g., Nike+, Strava, Fitbit). A review of the literature by Hamari et al. (2014) found that gamification can lead to positive outcomes, like increased engagement and compliance, favourable appraisals and ratings, in areas such as commerce, education and learning, health and exercise. Examples abound where organisations have used gamified processes to re-engineer their approach to sourcing, engaging and in some cases selecting applicants. Some recent examples include:

Just like an effective recruitment campaign can impress positively upon applicants, employers are now becoming more aware that the use of innovative or novel personnel selection assessments can further promote an organisation’s attractiveness and technological standing. It can be argued that the rapid transition to online, remote assessment has highlighted this most clearly. Prior to internet testing becoming the default mode of administration, psychological testing generally represented only a minor component of the selection process, often implemented towards the final stages. Its role, therefore, was to assist in selecting an employee from an already pre-screened sample of applicants. This kept administration costs low as only a small number of candidates completed testing. Online, remote administration however has allowed testing to become more easily accessible and affordable for employers. As a result, it is also more readily available as an initial screening tool and can be exposed to a much larger number of applicants.

This has clear implications for both the demands and responsibilities of assessment practitioners and providers alike. Because of this ease in testing, inter-organisational competitiveness has intensified due to the fact that companies are sourcing candidates from a shared pool. The ‘war of talent’ has shifted more towards a focus on employee experience or EX and organisations are cautious not to disengage talent with out-dated selection experiences when trying to attract and source the best and brightest applicants.

The introduction of internet technologies has flattened job application pathways to an extent where this process has become a bidirectional transaction. Rather than being seen as a precursor to the applicant-employer relationship, the selection process may best be viewed as the initial episodes of that relationship where both parties are concerned with courting the other. An applicant’s choice to first continue the relationship beyond an initial recruitment experience and then subsequently behave within that relationship can be greatly influenced by first impressions gathered during this phase (McCarthy et al., 2019).

Gamification vs game-based assessment

Whether it is the growing demand and interest from organisations, motivation for improvement and innovation from test providers or more likely a combination of both, the incorporation of gamification principles in the development of psychometric assessment is becoming more evident. Although at time of writing, it’s still small in scale relative to traditional internet-based assessments, the emergence of several vendors offering products of this nature e.g., Arctic Shores, cut-e, Knack, MindX, Owiwi, Pymetrics, Revelian, attests to this. These products are often promoted as offering distinct advantages to existing instruments such as more positive candidate reactions, attracting greater diversity amongst applicants and promoting a brand of innovation. The extent to which an assessment is game-like however can vary widely from one provider to the next and perhaps rests on two considerations; namely to what extent has the assessment been developed as a game from the ground-up as opposed to incorporating game mechanics to existing item content, and do those who complete such assessments actually describe them using words like ‘gameful’, ‘fun’ and ‘engaging’. Hawkes et al. (2018) offer a more comprehensive taxonomy as a way of determining this. They further propose that each characteristic, if present, can be described as high or low.

Hawkes et al. (2018) provide examples to illustrate how this taxonomy can be applied. A multimedia situational judgement test for instance can be considered high on the fidelity characteristic but low on others, demonstrating the limited game-like nature of this selection tool. A flight simulator on the other hand could arguably be considered high on most if not all of these characteristics and therefore more easily described as game-like. Much like saying not all surveys should be considered useful for psychological measurement, this helps demonstrate that not all gamified assessments should be considered the same. As alluded to previously, this distinction is most notable when comparing an existing assessment which has been gamified to more immersive assessments where game elements are built into the fabric and structure of the assessment, in many ways guiding its design and subsequent scoring and validation. The design of this latter type of assessment, often referred as game-based assessment (GBAs), represents a time and resource intensive enterprise, requiring the interdisciplinary expertise of psychometricians, game designers, software engineers and user-experience experts so as to effectively balance the dual focus on measurement and game development.

Measurement and game design

With respect to measurement, this form of assessment design involves the creation of in-game activities aimed at eliciting behavioural responses that can be scored and converted to meaningful estimates of a construct like cognitive ability, and emotional intelligence. Such in-game behaviours can include interactions with game objects, time spent on tasks and between tasks, mouse clicks and mouse movement. Terms such as trace data, log data, telemetry and digital exhaust (a personal favourite) are used to describe this type of data which is captured every few hundred milliseconds. As you can imagine, just a single playthrough of an assessment a few minutes in duration can amount to many thousands of data points per player. The aggregation of such data is often termed ‘feature engineering’ in machine-learning speak, but corresponds to a psychologist’s understanding of predictor variables. Even when aggregated though the number of such features or variables can easily dwarf those generated from traditional measures, where dichotomously scored right and wrong or likert-scale responses are the norm.

This is where the application of machine learning techniques, or modern prediction methods, is emerging as such techniques can more easily manage these exponentially larger datasets and offer more effective ways of balancing model complexity with parsimony (Putka et al., 2018). An important consideration for models derived from these methods though is their degree of transparency and accessibility. Traditional approaches are often very glass-box in nature, where the underlying algorithms used to generate resulting scores or derive decisions are readily accessible and explainable. Modern approaches can be more opaque and details about which features or predictors are being used, and how they are used, more complex and difficult to interpret. This is particularly relevant in the context of personnel selection where a theoretical basis for prediction is paramount and issues relating to bias and adverse impact are critical concerns (Tippins et al., 2021).

When considering game design, it is important to understand the implications for test publishers who embark on the development process and as such must work closely with multidisciplinary teams. The development of GBAs, like other innovative tools, require all participants to share a fundamental understanding of software engineering to be successful. Traditional psychometric assessment development can be viewed as similar to an engineering approach referred to as waterfall. Waterfall development requires compliance with rigid step-by-step procedures facilitated by extensive planning and documentation. While these approaches may work well with known problems and quantifiable and labelled risks and challenges, they are limited when dealing with novel processes or greenfield projects with many unknowns. For example, waterfall development struggles to account for changes in plans or processes and must return to fundamental stages of decision making. Put simply, waterfall does not accommodate change well.

In recent decades, frameworks such as agile development have arisen in response to the limitations of this waterfall approach. Agile typically consists of small cross-functional teams that work collaboratively on components of the larger development challenge. The teams produce incremental improvements or optimised prototypes which are tested and refined continuously, with iterative changes often guided by user testing and feedback. In game-based assessment design, this may involve the development of a paper-based prototype initially generated from the process of ideation. This idea would then be progressed to a low-fidelity example in which game mechanics data schemas and back-end infrastructures were developed and tested. A final high-fidelity prototype with a more polished user interface and graphic assets could then be produced, ready for formal psychometric validation.

What does the research say?

Although still emerging, a body of empirical evidence demonstrating the relationship between gameplay behaviours and various psychological constructs is growing. Such studies have often used existing commercial and casual games. Quiroga et al. (2015) for instance identified strong correlations between performance on Nintendo Wii puzzle games and a battery of traditional cognitive ability tests. Other efforts have employed games such as World of Warcraft (Short et al., 2017) and Portal 2 (Buford & O’Leary, 2015). More recent investigation has focused on game-based instruments created predominantly for research purposes. In their design of a game-like measure of personality (McCord et al., 2019) displayed moderate-to-strong correlations between responses on a traditional five-factor personality inventory to choices made in-game. Similar supporting evidence is provided by Barends et al. (2021) in their development and subsequent validation of a game-based assessment of the Honesty-Humility personality trait i.e., convergent validity with self-reported Honesty-Humility and divergent validity with other HEXACO traits and cognitive ability. Incremental validity beyond self-reported personality was also shown in the prediction of certain counterproductive behaviours e.g., cheating for financial gain.

Research evidence supporting the use of GBAs in personnel selection however remains small, limiting more widespread adoption and acceptance in organisations. This is due to some extent to the development processes, data and measurement techniques of vendors remaining largely proprietary. Some vendors however have begun to share evidence of the validity and reliability of their measures at recent conferences such as SIOP and IOP. The work by Landers et al. (2017) is an example of this. In their study of college students Landers et al. demonstrated that a composite score derived from a number of commercially developed GBAs measuring broad cognitive abilities was strongly related to a composite score derived from more traditional measures of such abilities. Participants also reported more positive reactions to the cognitive-based GBAs compared to the traditional cognitive test battery. This included motivation reactions e.g., “These tests (games) were fun to do,” perceptions of procedural justice e.g., “I could really show my skills and abilities through these tests (games)” and job relatedness e.g., “It would be clear to anyone that these tests (games) are related to the job.”

Future of gamification

The continued exploration of the utility and limitations of these types of tools will rely heavily on work such as this, where open and transparent collaboration between academics and practitioners takes place. Real partnership between technologists and organisational psychologists is also required. In the absence of this the divide between what is proven empirically and what happens in practice will drastically expand and lessons learnt from previous experience e.g., “the internet testing train has left the station” (Tippins, 2008), will largely go unheeded. The upcoming publication of a special issue of the International Journal of Selection and Assessment titled ‘Game-based and Gamified Assessments: Advances at the Frontier of Psychometrics’ later this year is an encouraging step in this direction. It is through this and other similar endeavours that the psychometric rigour and ethical standards critical to the ongoing utility of psychological assessment can be effectively balanced alongside candidate and market expectations of modern recruitment and selection practices.

Contact the authors: [email protected]; [email protected]