Games Testing Methodologies and Their Impact on Actionable User Requirements

Rollings and Adams (2003) define a user’s experience as the impact three gaming elements – Interaction, Visual and Audio – when used in conjunction, have on their overall satisfaction with a game. While several texts offer consistent and detailed descriptions of what metrics should be measured against in relation to these three elements, little is offered on what methodologies are best for gathering requirements against these metrics, especially in a commercial environment. This workshop is therefore aimed at those with an interest in gaming to explore various approaches to gathering requirements across all three gaming elements and how these approaches might vary across game genres.

Gaming, interaction design, games UI Design, audio feedback, user-centred design (UCD)

1. INTRODUCTION
For commercial games testing, many approaches to user-centred design can be deployed. Reliance on a think-aloud protocol in one-on-one testing may offer excellent feedback on the quality of the interaction. However, is the fact that the user is constantly talking, whilst playing, going to provide fair and accurate feedback on how a game’s audio elements are affecting their experience? Does the mental load of trying to verbalise their thoughts while playing affect their performance and subsequent attitude towards the game? This workshop is designed to address questions of this nature by discussing common and novel approaches to games testing. During the course of the workshop participants will explore: 1) various methodologies or approaches and their suitability for addressing the interaction, visual and audio elements of a game’s design; 2) whether or not these approaches would vary depending upon the genre of game being tested; 3) the commercial feasibility of employing various methods. The end goal of the workshop is for participants to come away with a paradigm for testing user experience across varying game elements and genres by addressing the pros and cons of each approach discussed.

2. DEFINING USER EXPERIENCE
Rollings and Adams (2003) 3 axes of Interaction, Visual and Audio are used to provide a common definition for user experience as a foundation for discussion. 2.1 Interaction Interaction is the ‘feel’ part of the ‘look and feel’ of a game. While it is highly connected to the visual element of a game, it is more concerned with execution. Both the pathways and navigation that is required for successful gameplay and the actual physical controls used to execute moves (Rollings and Adams, 2003). Metrics for assessing interactivity include: how much of what a player wishes to do does the game allow for, how well does the game translate a player’s inputs (e.g. is the game going to make the avatar jump as high as the player wants it to jump), and how well does the game display it’s reaction to a user’s inputs (how does it represent the height and style in which the avatar jumps on screen) (Crawford, 2003).

2.3 Audio
The audio element concerns any aural components of the game: sound effects, music, and verbal dialogue. While audio is often focused upon less than the other two gaming elements, it is an essential component of feedback and creating an appropriate atmosphere. Audio should provide: feedback for distinctive actions (e.g. distinguishing between positive and negative actions), help create pace and provide context (e.g. music speeding up in Tetris to let a player know they’re close to losing the game), and general gameplay enhancement (e.g. each time a particular character comes on screen their theme music plays) (Oxland, 2004).

3. METHODOLOGIES
Currently there are 4 main methods used for testing games: 1) Focus groups, 2) Interviews, 3) Questionnaires/surveys and 4) Ethnographic research (Sweetser and Drennan, 2003). It is feasible that the approaches that come out of the workshop are one, or a combination, of these aforementioned methods. The purpose of this workshop is not necessarily to come up with novel methodologies. Of utmost importance is proposing approaches that are likely to be the most effective depending upon the gaming element and/or genre that is being explored. This could be a completely new approach or a fine-tuning of existing methodologies.

3.2 Commercial constraints
While there are no restrictions on the methodologies and approaches to be addressed, the end result is to discover approaches that are feasible within a commercial design environment. Therefore, of secondary concern are the time it takes to execute an approach (e.g. ethnographic research vs. diary studies vs. one-on-one sessions) and the financial feasibility of a technique: How much is it likely to cost? What equipment would be required?

4. PARTICIPANTS
This workshop is designed for both academics and practitioners interested in games design.

• Practitioners should have experience testing console (e.g. Sony PS2, Nintendo Gamecube, Microsoft Xbox) and handheld games, preferably in a commercial environment.
• Academics should hold an interest user-centred games research – platform independent.

Each participant should submit a position paper about their thoughts on various approaches to user-centred games testing. Positions papers do not have to ascribe to any particular template, but should be maximum of 2 pages and fall into at least one of two categories:

1. Hypothetical: Pros and cons or guidelines about various games testing approaches. This can be as a result of their own personal research or a well thought out literature review of existing methodologies.
2. Practical: General thoughts on games testing through direct personal experience. No definitive guidelines necessary, but the paper should address outstanding issues or learnings from their experience.

4. WORKSHOP PROGRAMME

• Phase 1: Introduction (1 hr) – outlining the days agenda and the rationale for the workshop
• Phase 2: Position papers (1.5 hrs) – participants present their position papers (c. 10mins each) with open discussion
• Phase 3: Game interaction and discussion (4 hr, with 1hr lunch break) – facilitator will demonstrate the 3 games in turn. After each game demo, the facilitator will break the participants up into groups to discuss the best approach to addressing the 3 user experience elements for the game in question. The games will be available for participants to interact with throughout discussion. Games to be explored will be 3 of the following 4 types:
• A first person shooter for the PS2 console; the game relies on handheld controls for input
  
• A battle karaoke game in which players are graded on their ability to hit certain pitches; gameplay is mediated by players singing into microphones
• An EyeToy game - users see a mirror image of themselves on-screen, facilitated by the use of the EyeToy camera attached to a PS2 console, and use their own body parts to dictate gameplay
• an arcade style game for the Sony PSP portable handheld console

• Phase4: Conclusion (1.5hrs) – groups will be brought back together to discuss the pros and cons of each proposed approach for testing each game and their commercial feasibility

REFERENCES
[1] Crawford, C (2003). Chris Crawford on Game Design. New Riders Publishing, USA.
[2] Oxland, K (2004). Gameplay and Design. Addison Wesley, UK.
[3] Rollings A and Adams, E (2003) Andrew Rollings and Ernest Adams on Game Design. New Riders Publishing, USA
[4] Sweetser, P and Drennan P (2003). User-Centred Design in Games. http://www.itee.uq.edu.au/~penny/AGDC%202003.ppt