Game Testing E-Book

Mercury Learning and Information

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 537

Veröffentlichungsjahr: 2024

Beschreibung

This updated fourth edition is your comprehensive guide to video game testing. Whether you're a student aspiring to join the industry or a professional honing your skills, this book covers essential aspects of game testing. It explains how testers fit into development and provides practical knowledge of tools, roles, responsibilities, and metrics for game quality.
The book guides you through test design and QA methods using real game scenarios and interviews with veteran developers. You'll learn the basics of game testing, writing bug reports, and defect tracking. The chapters cover exploratory testing, gameplay testing, combinatorial testing, and regression testing.
Each chapter includes questions and exercises, suitable for classroom or personal study. Companion files with templates and tutorials for creating combinatorial tables and test flow diagrams are included, forming the basis of a robust QA plan. This edition equips you with the skills to succeed in game testing and contribute effectively to any development team.

Leseprobe

LICENSE, DISCLAIMER OF LIABILITY, AND LIMITED WARRANTY

By purchasing or using this book and companion files (the “Work”), you agree that this license grants permission to use the contents contained herein, including the disc, but does not give you the right of ownership to any of the textual content in the book / files or ownership to any of the information or products contained in it. This license does not permit uploading of the Work onto the Internet or on a network (of any kind) without the written consent of the Publisher. Duplication or dissemination of any text, code, simulations, images, etc. contained herein is limited to and subject to licensing terms for the respective products, and permission must be obtained from the Publisher or the owner of the content, etc., in order to reproduce or network any portion of the textual material (in any media) that is contained in the Work.

MERCURY LEARNING AND INFORMATION (“MLI” or “the Publisher”) and anyone involved in the creation, writing, or production of the companion disc, accompanying algorithms, code, or computer programs (“the software”), and any accompanying Web site or software of the Work, cannot and do not warrant the performance or results that might be obtained by using the contents of the Work. The author, developers, and the Publisher have used their best efforts to ensure the accuracy and functionality of the textual material and/or programs contained in this package; we, however, make no warranty of any kind, express or implied, regarding the performance of these contents or programs. The Work is sold “as is” without warranty (except for defective materials used in manufacturing the book or due to faulty workmanship).

The author, developers, and the publisher of any accompanying content, and anyone involved in the composition, production, and manufacturing of this work will not be liable for damages of any kind arising out of the use of (or the inability to use) the algorithms, source code, computer programs, or textual material contained in this publication. This includes, but is not limited to, loss of revenue or profit, or other incidental, physical, or consequential damages arising out of the use of this Work.

The sole remedy in the event of a claim of any kind is expressly limited to replacement of the book and/or files, and only at the discretion of the Publisher. The use of “implied warranty” and certain “exclusions” varies from state to state and might not apply to the purchaser of this product.

Companion files for this title are available with proof of purchase by writing to the publisher at [email protected].

Copyright ©2024 by MERCURY LEARNING AND INFORMATION. An Imprint of DeGruyter Inc.

All rights reserved.

This publication, portions of it, or any accompanying software may not be reproduced in any way, stored in a retrieval system of any type, or transmitted by any means, media, electronic display or mechanical display, including, but not limited to, photocopy, recording, Internet postings, or scanning, without prior permission in writing from the publisher.

Publisher: David Pallai

MERCURY LEARNING AND INFORMATION

121 High Street, 3rd Floor

Boston, MA 02110

[email protected]

www.merclearning.com

800-232-0223

Robert Denton Bryant. Game Testing: All in One, Fourth Edition.

ISBN: 978-1-50152-168-3

The publisher recognizes and respects all marks used by companies, manufacturers, and developers as a means to distinguish their products. All brand names and product names mentioned in this book are trademarks or service marks of their respective companies. Any omission or misuse (of any kind) of service marks or trademarks, etc., is not an attempt to infringe on the property of others.

Cover image from Wildermyth. © Worldwalker Games LLC

Library of Congress Control Number: 2024932259

242526321 This book is printed on acid-free paper in the United States of America.

Our titles are available for adoption, license, or bulk purchase by institutions, corporations, etc.

For additional information, please contact the Customer Service Dept. at 800-232-0223(toll free).

All of our titles are available in digital format at academiccourseware.com and other digital vendors. Companion files for this title are available with proof of purchase by contacting [email protected]. The sole obligation of MERCURY LEARNING AND INFORMATION to the purchaser is to replace the book/files, based on defective materials or faulty workmanship, but not based on the operation or functionality of the product.

CONTENTS

Foreword

Preface to the Fourth Edition

Acknowledgments

Chapter 1: Your Role on the Game Development Team

“You Get Paid to Play Games?”

Elements of a Game Development Team

Programmers

Artists

Animators

Level Designers

Sound Designers

Game Designers

Producers

Testers

Putting It All Together

The Game Production Cycle

Concept Development

Development

Alpha

Beta

Code Lock

Gold

Patches

Updates

Ready, Tester One?

Exercises

Reference

Chapter 2: The Basics of Game Testing

Not All Players Are Alike

Black Box Testing

White Box Testing

Your First Test Suite

Writing Test Cases and Test Suites

The Life Cycle of a Build

The Testing Feedback Loop

The Essence of a Bug Report

Testing “Around” a Bug

Not All Testers Are Alike

Exercises

References

Chapter 3: Bug Report Writing and Defect Tracking

How to Write a Bug Report

Just the Facts, Please

The Brief Description

The Full Description

Great Expectations

Habits to Avoid

Using the Bug Database

Category

Summary

(Full) Description

Severity

Priority

Attachments and Recordings

Other Database Fields

Exercises

References

Chapter 4: How Bugs Happen

Who Cares?

Defect Types

Functions

Assignments

Checking

Timing

Build/Package/Merge

Algorithms

Documentation

Interfaces

Testing Happens

Exercises

References

Chapter 5: The Phases of Game Quality Assurance

Pre-Production

Planning Tasks

Determine the Scope of Testing the Project Will Require

Assign a Lead Tester

Establish Milestone Acceptance Criteria

Participate in Game Design Reviews

Set Up the Defect Tracking Database

Draft Test Plans and Design Tests

Testing Before Testing Begins

Test Kickoffs

Configuration Preparation

Smoke Testing

Alpha Testing

Alpha Phase Entry Criteria

Regression Testing

Beta Testing

Beta Phase Entry Criteria

Design Lock

Living with Bugs (For Now)

“The Swing Set of Death”

Patches, Updates, and Hotfixes

Gold Testing

Last-Minute Defects

Release Certification

Post-Release Testing

Live Teams

Exercises

Reference

Chapter 6: Exploratory Testing and Gameplay Testing

Ad Hoc Testing

Free Testing Comes from the Right Side of Your Brain

“Fresh Eyes”

Directed Testing Makes Order Out of Chaos

If You Are Not Recording, You Are Not Testing

Avoid Groupthink

Testing as Detective Work

The Benefits of Reproduction

The Scientific Method

Gameplay Testing

A Balancing Act

“It’s Just a Suggestion”

Making a Game Easy is Hard Work

External Testing

Who Decides?

Exercises

References

Chapter 7: The Two Rules of Test Management

Rule One: Do Not Panic

Unfamiliar

Unprepared

Under Pressure

Unrested

“Nearsighted”

Rule Two: Trust No One

Balancing Act

Word Games

Last Chance

Trust Fund

Managing Crunch Time

Give and Take

The Rest of the Story

Exercises

Reference

Chapter 8: Combinatorial Testing

Parameters

Values

Defaults

Enumerations

Ranges

Boundaries

Constructing Tables

Combinatorial Templates

Combinatorial Test Generation

Combinatorial Economics

Exercises

Chapter 9: Test Flow Diagrams

TFD Elements

Events

Actions

States

Primitives

Terminators

TFD Design Activities

Preparation

Allocation

Construction

A TFD Example

Data Dictionary

Data Dictionary Application

Data Dictionary Reuse

Data Dictionary Example

TFD Paths

Minimum Path Generation

Baseline Path Method

Expert Constructed Paths

Combining Path Strategies

Producing Test Cases from Paths

TFD Templates

When Should You Create a TFD?

Exercises

Chapter 10: Cleanroom Testing

Usage Probabilities

Mode-Based Usage

Player-Type Usage

Real-Life Usage

Cleanroom Test Generation

Cleanroom Combinatorial Tables

Cleanroom Combinatorial Examples

TFD Cleanroom Paths

TFD Cleanroom Path Example

Flow Usage Maintenance

Flow Usage Profiles

Inverted Usage

Calculating Inverted Usage

Combinatorial Table Usage Inversion

TFD Flow Usage Inversion

Exercises

References

Chapter 11: Test Trees

Test Case Trees

Tree Feature Tests

Test Tree Designs

Exercises

Chapter 12: Defect Triggers

Operating Regions

Pre-Game Operating Region

Game Start Operating Region

In-Game Operating Region

Post-Game Operating Region

The Triggers

The Configuration Trigger

The Startup Trigger

The Exception Trigger

The Stress Trigger

The Normal Trigger

The Restart Trigger

Classifying Defects

Defect Triggers and Test Designs

Combinatorial Design Trigger Examples

TFD Trigger Examples

Exercises

Reference

Chapter 13: Regression Testing and Test Reuse

Regression Testing

The A-B-C Method

Defect Modeling

Time Keeps on Ticking

Expanding Possibilities

Test Reuse

TFD Design Patterns

Looking Back and Forth

Combinatorial Expansion

Exercises

Reference

Chapter 14: Testing by the Numbers

Testing Progress

Testing Effectiveness

Tester Performance

Exercises

Chapter 15: Software Quality

Game Quality Factors

Game Quality Appraisal

Walkthroughs

Reviews

Checklist-based Reviews

Inspections

Game Standards

User Interface Standards

Coding Standards

Game Quality Measurements

Six Sigma Software

Phase Containment

Quality Plans

QA Personnel

Standards

Reviews and Audits

Feedback and Reports

Problem Reporting and Corrective Action

Tools, Techniques, and Methods

Supplier Control

Training

Risk Management

Exercises

References

Appendix A: Odd-Numbered Answers to Exercises

Chapter 1 — Your Role on the Game Development Team

Chapter 2 — The Basics of Game Testing

Chapter 3 — Bug Report Writing and Defect Tracking

Chapter 4 — How Bugs Happen

Chapter 5 — The Phases of Game Quality Assurance

Chapter 6 — Exploratory Testing and Gameplay Testing

Chapter 7 — The Two Rules of Test Management

Chapter 8 — Combinatorial Testing

Chapter 9 — Test Flow Diagrams

Chapter 10 — Cleanroom Testing

Chapter 11 — Test Trees

Chapter 12 — Defect Triggers

Chapter 13 — Regression Testing and Test Reuse

Chapter 14 — Testing by the Numbers

Chapter 15 — Software Quality

Appendix B: Sample Test Suite: RTS Building Checklist

Appendix C: Basic Test Plan Template

Section I: QA Team (and areas of responsibility)

Section II: Testing Procedures

Section III: How Testing Requirements are Generated

Section IV: Bug Tracking Software

Section V: Bug Classifications

Section VI: Bug Tracking

Section VII: Scheduling and Loading

Section VIII: Equipment Budget and Costs

Appendix D: Combinatorial Test Templates

Tables of Parameters with Two Test Values

Tables of Parameters with Three Test Values

Appendix E: Test Flow Diagram (TFD) Templates

Power-Ups

Craft Item

Heal Character

Create/Save

Unlock and Buy Item

Update Song List

Complete a Mission or Quest

Get Weapon and Ammo

Glossary

Index

FOREWORD

Having known Bob Bryant for many years, I am honored to be writing the foreword for this fourth edition of Game Testing: All in One. That the first edition was in 2005 and almost two decades later we are into the fourth edition is remarkable.

With the challenges that Game QA faces today, to put it bluntly, we are long past the point where simply adding more people to our test teams for coverage is the answer.

Then why do we need this book if simply adding more people is not the answer? We still need people, many more people. It is just that the requirements of the role are changing.

Game QA is still one of the best proving grounds for growing the future leaders of tomorrow’s game studios that I know. More so now than ever, it is a stand-alone discipline within game production. Yet many studios continue to underestimate its criticality to not just development, publishing, and launch, but also to the key metric of player retention.

The discipline of Game QA is becoming ever more complex and the ways in which we work are increasingly more sophisticated. It is now its own career path within the game production ecosystem.

Many game studios have dropped the term “Quality Assurance” and now simply use the word “Quality.” A recent sample of testing job roles across the industry included such job titles as quality design, quality verification, quality control, test analysis, quality data analysis, quality engineering, technical quality, compliance, compatibility, release verification, network testing, and user acceptance. The list of terms goes on and on.

The current challenges we face as Game QA are manifold, but can be broken down into two clear areas. The first of these is the volume of content and code that is now generated at an increasingly accelerated pace and the sheer complexity of the game titles that are released.

This can be addressed with automation and tools. Automation will ensure we have coverage over any previously released content and that new content has not broken the “last known good,” to paraphrase a Microsoft term.

Tools help us to work faster and more efficiently. The term artificial intelligence (AI) was coined in the 1950s and is based on the idea of Intelligence Amplification. I believe our current tools future is Augmented AI.

These are the tools we can use to reinforce our decision-making, validate our thinking, and which will enable us to work smarter and more effectively, but not replace us.

Automation ensures that we have not broken what was previously released. However, new content still requires testing by our test teams. More so now than ever, “finding the fun” is a critical part of the Quality role.

Finding exploits is critical: escaped defects and exploits are the costliest bugs a studio can let slip through, with not just revenue implications but also reputational damage and impact to the latest production schedules.

The concept of “Shift Left,” although not new, is also gaining prominence. Quality is becoming about prevention and not detection. Quality becomes an integral part of the design process, quality built in throughout production and everyone being responsible for quality across the studio.

The second challenge is the very nature of “live service” games themselves and the size of the player communities we now serve. Games are no longer shipped with a “fire and forget” attitude. They are products that last years, decades even, building social networks within their communities for our players.

Players put in more play time in the first week, days, hours, or even minutes after launch than the time that we can spend testing the games internally, even with hundreds of testers working for months before launch!

At this juncture, we should consider two questions. What is the role of Game QA? How has this changed?

QA is often referred to as the voice of the player. I believe that in the “games as a service” world, our test teams are the bridge to the player, where at all points in the game production process we are building in quality feedback loops across internal teams and our player communities.

Does the QA role go beyond prevention? Is it also about listening to what our players want, what brings them joy, and what detracts from their experience?

I do not have all the answers to these questions. The customer is usually right, as the adage goes, but reacting to those players who shout the loudest is not the best approach either!

My conclusion, though, is that it is probably more about listening to the voice of everyone in our communities, analyzing what that data is showing us, and making informed quality decisions.

The roles and responsibilities of QA have developed exponentially over the past two decades, the complexities of the games we deal with require many more roles and responsibilities. The basics of game testing, however, have not.

This to me is why this book remains crucial for both entry-level testers, test managers, developers or anyone involved in production, who wants to understand the principles of one of the critical disciplines required for successfully shipping a high-quality game.

Ben Wibberley

Cambridge, England

Ben Wibberley is VP of Game Operations at Jagex and has over twenty-five years of experience in the video game industry. Having started his career as a game tester, he has dedicated his career to championing Game QA. He co-founded Babel Media, the first multi-service game outsourcing provider, in 1998. He also set up a publishing support network at DDM and ran global business development for VMC. He founded DAQA, the Concierge of Game Quality, a boutique game quality studio. He is a co-founder of GameQuality.org and a co-chair of the Qualicon content committee. He will always consider himself to be a tester, albeit not a very good one.

PREFACE TO THE FOURTH EDITION

The first edition of this book appeared almost twenty years ago. (Spacetime is relentless like that.) Since then, the world of video games has both transformed and exploded. More players are playing games than ever before, and there are more platforms for playing on and more ways of making money from games than ever before. Small wonder, then, that some game testers have started to organize and join labor unions in the last few years. This work is skilled labor, and game testers deserve to be adequately compensated for their hard work, enjoying the same benefits and job protections as other members of the game development team.

As we drafted the first edition of Game Testing All in One, Charles P. Schultz and I described the best practices of game testing at a time when most video games in North America and Europe were sold at retail stores and played on computers or game consoles, be they handheld or attached to a television. Now most games are downloaded to smartphones, smart TVs, consoles, and computers. Billion-dollar global companies release years-in-the-making “AAA” games that are epic in their scope. Tiny one-person development teams release quirky little mobile games. Billions of dollars have been spent in the last several years by thousands of companies trying to make a market for virtual reality. Pokémon Go proved the market for augmented reality games in a weekend. The global pandemic that began in 2020 supercharged the growth of video games, as players worldwide turned to titles like Animal Crossing to escape the grubby worries of actual reality.

There have never been more games competing for the attention of more players. Never before has game quality and stability been more important. The process and discipline of game testing is crucial for any development team of any size to learn and adhere to, especially in this new era of constant patches, updates, feature roll-outs, expansions, and DLC releases. It seems today that for so many games, development is never finished. Neither, then, is testing.

Fortunately, the fundamental skills involved in game testing have changed very little. What has changed in many cases is the complexity of the games tested, the number of platforms they are required to be tested on, and the number of ways players can interact with—and spend money in—a game. Although automation and artificial intelligence can help to make the game tester’s job easier, gameplay has to feel right to human players, so human testers will always be needed.

For this fourth edition, we have revised and restructured the book to make it more useful to students, teachers, entry-level testers, and those new to video game development. We have restored and updated material describing the overall game development process and the essential role that quality assurance (QA) plays within it. We have moved to the second half of the book those chapters aimed at test managers and game producers who want to improve the efficiency and efficacy of their testing resources. And we have added a glossary!

Software engineers will continue to make mistakes. Game designers will inadvertently introduce exploits to their carefully crafted gameplay systems. Artists in haste may fail to close vertices or hide seams. It remains the job of the game tester to advocate for the players by “breaking” the game before the player ever gets to play it. It is up to vigilant game testers to save the player from frustration, confusion, and lost time, and thereby help to ensure the commercial and critical success of the game. Like the Night’s Watch in Game of Thrones, testers are the anonymous, out-of-sight, and unsung heroes of the realm of video game development.

Our hope is that this upgraded edition of the book will help you to develop your skills as a tester, or test manager, so that you can remain vigilant. The players, though they may not know it, are ever grateful for your hard work.

Most sadly, this is the first edition of the book to be published without new material from my brother in testing, Charles P. Schultz, who passed away in November of 2022.

This book would not exist without his inspiration, vision, and leadership.

Rest in peace, my friend.

Robert Denton Bryant

Kyle, TX

April 2024

ACKNOWLEDGMENTS

It would be a “show-stopping bug” if I failed to acknowledge the valuable support and input of my colleagues and students at St. Edward’s University in Austin, Texas, most particularly, Assistant Professor of Video Game Development, Jeremy Johnson, PhD.

I am also very grateful to David Pallai at Mercury Learning and Information, Heather Maxwell Chandler, Stefan Seicarescu, and Marius Popa at Quantic Lab, and Terri DePaolo, whose patience is boundless.

Very special thanks to Mirta Schultz, for helping to make this edition of the book possible.

CHAPTER1

YOUR ROLE ON THE GAME DEVELOPMENT TEAM

“YOU GET PAID TO PLAY GAMES?”

The job of testing video games is very hard, very rewarding, and very weird. When you tell friends what you do, they may ask, enviously, “You get paid to play games?”

“No,” you will reply. “Game testing is hard work. It’s a real job.”

The first time you are assigned to test a game, you might play it as a player would, feeling the joy of discovery as you learn the controls, setting, and gameplay loop.

But the fortieth time you launch the same game, it is work.

The discipline of game testing, often referred to as quality assurance or QA, is a crucial aspect of the process of video game development. Game testing is (and should always be) considered an integral part of the wide range of skills needed to produce a successful video game.

Game teams come in different sizes, structures, and locations. They can vary by company, game genre, or platform. The different disciplines required to produce a working video game are often organized into distinct departments within the overall game team. The people on the team must work together to complete the tasks that are needed to get the game done on time and without serious software defects, or bugs. By understanding the different departments on a typical video game project—even though developers may insist that their project is anything but typical—you can begin to visualize your role within the QA department and how it contributes to the success of the project.

ELEMENTS OF A GAME DEVELOPMENT TEAM

Programmers

Programmers make the video game work. They use their programming skills to turn the game design into something you can play on your computer or other device. When unexpected problems arise, they are quick to fix them with their problem-solving skills. They also know the details of the software tools used to make the game, which helps them to improve performance and fix tricky bugs.

The members of this team may call themselves programmers, engineers, coders, scripters, or developers. (This last term is falling out of use, however, since anyone who works for a game development company can be considered a game developer.) They produce the code that the testers must test before the game can be released. Their job involves translating the gameplay, artwork, sound, and story of the game into programming language. This code is subsequently converted into a game program that players run on their devices.

The code has to fit within a certain budget, use a limited amount of working memory, and be able to make things such as user input responses and video frame rate flow smoothly. The game program also has to fit within whatever development framework the team is using. This may include elements such as a game engine, which follows defined rules for automating the processing of certain game elements, and middleware, which provides a common interface to certain game functions so the same code can be moved from one platform to another. Programmers also have to deal with operating systems, device drivers, and communications protocols for multiplayer games, each with their own complexities and challenges.

Programmers may be involved in the porting of game code from one platform to another. Certain portions of the code should remain the same, whereas others must be changed to accommodate differences in the new platform. Porting a game from Windows PC to Xbox may not be difficult because those platforms are similar. However, the PC version needs to account for a variety of screen resolutions, graphics cards, installed memory, audio devices, and input devices. Going from Xbox to another game console would provide a different set of challenges, and porting a game from a game console to a tablet or mobile device may result in a completely new design and development effort, because the hardware configuration might be quite different. In all cases, the port must be freshly tested on the new platform to ensure that the player’s experience is the same as on the original platform.

Artists

Without the art team, we would only be playing text-based games like Hunt the Wumpus or Colossal Cave Adventure. The public’s expectation for the artistic experience of the game is higher than it has ever been. Games based on such transmedia franchises as The Lord of the Rings, Spider-Man, and Star Wars already come with built-in expectations of a visual game experience that parallels what you would see in a movie theater.

Artists apply their skills with drawing, colors, shapes, and lighting by using digital tools to create everything you see in a video game. The art elements in a game are known as art assets. They exist in separate files from the game code, and may be combined or compressed in some way to minimize the amount of memory they use. The art in a game may be provided on a very small scale, such as individual decals to apply to a race car in Gran Turismo® 7, or on a massive scale, such as rendering a planetary landscape in Starfield.

Animators

Animators add realism and motion to the game. Animations need to be smooth and properly scaled, taking gravity into account. An animation is made up of a series of frames. Each frame contains a specific pose, and when the frames are played together, your eyes fill in the blanks between the poses, thanks to a phenomenon known as persistence of vision.

Characters and creatures are believable when they are properly animated. On a large scale, they are animated to walk, run, and jump. Their movement needs to be consistent with their weight, physiology, and the local force of gravity. At a smaller level, facial expressions and body language are animated to communicate emotion.

Explosive and destructive effects provide excitement and urgency to the game. The explosion could come from the end of a gun or from a remote detonation. There will be the central core effect, such as an expanding ball of light, followed by after-effects such as smoke or a concussion wave. The explosion itself can send rocks, vehicles, or the player’s opponents flying. Rocks or walls can break when a player is sent smashing into them.

Animating effects of nature help give a sense that you are part of the game environment. Leaves should spin and float as they fall, rather than drop straight down as if in a vacuum. Environmental interactions may also occur as the result of movement, such as water rippling and splashing when a character steps in or moves around in it.

Level Designers

Level designers define what goes into the various “levels” or parts of the world you explore or inhabit when playing a game. These chunks of content are defined differently based on the particular genre of the game. In a sports game, level designers might be responsible for creating authentic versions of real-life arenas or stadiums. In a racing game, they are responsible for creating authentic or fanciful tracks. For a first-person shooter (FPS) game, a real-time strategy (RTS) game, or a multiplayer online battle arena (MOBA) game, they are responsible for creating interesting and balanced maps for players to do battle. Even a vast, seamless, “open world” such as the Hyrule depicted in The Legend of Zelda: Breath of the Wild and Tears of the Kingdom is composed of regions, islands, zones, cities, and even individual quests, each of which can be thought of as a “level” for production purposes.

The level designer must make each level exciting and unique, but it should also fit into the context and theme of the overall game or world.

Sound Designers

The audio or sound team creates the audible experience of the game. Just like in the movies, a well-crafted sonic experience will make you feel as though you are part of the world. This team collaborates with designers and programmers to design sounds consistent with the visual and gameplay elements.

One of the main functions of a sound designer is to envision and implement an overall sound design for the game and for each level within the game. A sound design includes layered ambient noises (such as water dripping in a cave or rustling trees in a windy forest), triggered sound effects (such as the “ping” when Mario collects a coin), dialog, narration, and music. A player cannot see what is going on in the game beyond what is shown on the screen, but he can hear sounds coming from all directions. Clever game designers work closely with audio designers to create opportunities for players to use their ears to help play the game.

Game Designers

Game designers are responsible for conceiving and defining the gameplay of the game. They are storytellers, entertainers, and inventors. Their concepts are the basis of the game’s worlds, characters, and lore.

The game designer also defines game mechanics that are easy to learn, remember, and access during gameplay. A game mechanic is anything the player can do in the game. Multiple mechanics are sometimes grouped into systems (like a combat system or an inventory system), and sequences of systems are sometimes referred to as loops.

Game design might develop from a top-down approach, where the designers start with a high-level concept and then break it down into greater detail so that the artists, sound designers, and programmers can begin to make the game. Or, game designers might use a bottom-up approach, starting with a few ideas for a fun mechanic, and then working to come up with the story and settings that tie together those mechanics and give the player reasons to keep playing.

Whichever way the game designers arrived at the game concept, they are responsible for producing the game design document (GDD) used by the other departments to guide how they do their parts of the project. This documentation can also be a useful resource for testers to ensure that the designers’ ideas are incorporated into the game properly. Testers should use design documentation as the basis for providing a complete range of tests. From any flow charts or diagrams provided in the game design documentations, lead testers should write tests to cover all of the states and transitions possible in that particular feature or system. The same approach should be used with any screen layouts or menus documented in the game design.

Producers

Producers, often called project managers, work to see that the game gets done on time and within budget. Both the game developer and the game publisher may each employ their own producer. Producers develop a schedule with dates for particular production milestones (or deadlines) by which a set of tasks, goals, or deliverables should be finished. The deliverable provides an indication that the game is making sufficient progress and builds confidence that the game will be released on time.

In addition to developing the milestone schedule, producers may assign certain detailed tasks to specific people or job roles, along with a due date for each task. At any point in time on the schedule, the project manager should know how many people are needed, which tasks have been completed, and which tasks are in progress. Adding up all of the people at each point in time provides a staffing budget.

In addition to staffing, which translates into wages, the project manager may budget monetary expenses for equipment supplies and services. This includes new computers, hardware, and software tool licenses for programmers, artists, and testers. When some aspect of the game seems to be falling behind, producers may request more resources or figure out how to reduce the scope of the game to keep the project on schedule.

Producers rely on game testers to provide them with the best current information about the state of the game. They often determine which bugs will be fixed, in what order, and by which members of the development team.

Testers

Game testers will sometimes be identified by other titles such as test engineer, QA tester, QA analyst, or QA engineer.

There is a paradoxical relationship between programmers and testers that is based on the fact that they both want the same outcome—producing a superb game—but they have opposing roles in making that happen. Programmers want their code to run flawlessly, and will insist that their latest fix “should” work. Testers want the code to run flawlessly as well but will not believe it is true until they have conducted rigorous testing.

The tester’s job is to “break” the game by finding bugs while the programmers are trying to finish the game. Testers focus on what is wrong with the game and programmers focus on making things right. This can create tension within a team and across departments on a big, multi-location project. Both developers and testers should approach their jobs with dedication and energy. However, both groups should work with each other to be better at what they do. Testers can do their jobs better by understanding how the code is designed and produced, and then using that to improve the way they write and execute tests. Programmers can continue to improve their code by learning from what kinds of problems the testers are finding, as well as by using “embedded” testers who work side-by-side with programmers to perform unit testing of new features or bug fixes.

The test team can be dynamic during the course of the game project. In some instances, there is little or no testing at the beginning of the project, but then many testers work on the game at the very end. This is especially true for testers who work for game publishers, because testing is how publishers make certain their investment in the game developer will result in a quality game.

The head of a game team is called the test lead, but they will sometimes be identified by other titles such as QA lead or lead game tester. The test lead plans and orchestrates testing activities performed over the course of the game development project. The test lead is responsible for the on-time delivery of test development and test execution results. Test activities and individual assignments are identified, planned, and adjusted as necessary during the course of the project. The test lead may be helped by one or more primary testers, who act as assistant managers on the team.

The test lead also establishes test procedures and standards. This includes selecting the right test tools and technologies to use for testing the game code. In many cases, test tools need to be supported or otherwise compatible with certain details of the game code. The test lead defines these “testability” requirements for the game and works with the programmers to see that they get properly implemented. On large or complex games, you may see a lead assigned to different game modes or systems, such as multiplayer, story mode, quests, dungeons, mini-games, or the character editor.

On smaller game development teams, the test lead is also responsible for doing some of the testing.

Putting It All Together

Good teams make good games. Testers are an integral and valued part of the game development team. Everyone on the team wants to deliver a good game and has one or more roles to play in making that happen, and the people on the team should be aware of the contributions and responsibilities of the other team members. Testers need to examine the individual work product delivered by each role, as well as their relationship to one another within the context of the game. Any game project documentation generated along the way—from the game design document to individual build notes—should be used by the test team to accelerate test development and improve the effectiveness of the game testing.

Before we delve into the specific phases of game testing (in Chapter 5, “The Phases of Game Quality Assurance”), let’s first examine the life cycle of a typical video game development project.

THE GAME PRODUCTION CYCLE

Some games are developed in a few months. Others take several years. No matter how long projects last, each one goes through well-defined phases that have become standard across the industry. Some game projects may not incorporate all of the phases described below. It is also possible for the activities of one phase to continue while part of the team begins work on the next one:

1. Concept Development

2. Preproduction

3. Development

4. Alpha

5. Beta

6. Code Lock

7. Gold

8. Patches

9. Updates

Concept Development

Concept development is the fuzzy front end of game design. It lasts from the moment someone first comes up with a game idea until the day the game goes into preproduction. The team is very small during this period. It may consist of only the game designer, lead programmer, concept artist, and producer.

The main goal of concept development is to decide what the game is and to write this down clearly so that anyone can understand it instantly. During this phase, you decide on the major gameplay elements, create concept art to show what the game will look like, and add details to the story (if there is one).

It is important to note that no one in the games industry holds a job where they are paid simply to come up with ideas for games. “Game development teams are very collaborative, and if you can’t contribute to a game’s execution, you’re not yet ready to contribute to its conception,” writes veteran Heroes of Might & Magic producer David Mullich (2015).

The documents that come out of concept development are the high concept, game proposal (or “pitch doc”), and concept document.

The High Concept

The high concept is a one- or two-sentence description of what your game is about. It is the “hook” that makes your game exciting and sets it apart from the competition. This is sometimes referred to as an “elevator pitch,” because it is a way to express an answer to the “tell me about your game” question clearly and quickly, as though you had only thirty seconds during a chance encounter on an elevator.

A strong high concept is also valuable during the development phase because it helps you to decide which features to include and which to leave out. If game development is like trying to find your way through a jungle of possibilities, the high concept is a path that has already been cleared so that you do not get lost. Any feature that does not contribute to the game's main focus is a direction you do not need to explore.

The Game Proposal (“Pitch Doc”)

The game proposal is a short handout you speak from during pitch meetings to seek funding for the development of your game. In just a few pages, you must summarize what your game is about, why it will be successful, and how it will make money. This document covers much of the same information as the concept document, but in abbreviated form.

The Concept Document

The concept document is the fully developed version of the pitch doc. It is a longer document that members of the publishing team will not have time to review during a pitch meeting, but will want to read afterward to get a more detailed understanding of your game.

The concept document might contain the following sections:

■ The High Concept

■ Genre

■ Setting

■ Gameplay

■ Key Features

■ Target Audience

■ Target Platform(s)

■ Estimated Schedule, Budget, and Profit and Loss (P&L) Statement

■ Competitive Analysis

■ Important Team Members

■ Risk Analysis

These documents should emphasize the critical points of your game and the ability of your team to deliver a quality product, on time and on budget. As a tester or test lead, you can contribute by providing time estimates for the schedule, test equipment costs for the budget, and identify test risks for the risk analysis.

Preproduction (Proof of Concept)

Preproduction is the “gearing-up” or initial planning time. Your goal is to complete the game design, create the “art bible,” establish the production path, write up the project plan, and create a prototype. This phase is also used to do technical prototyping to demonstrate the feasibility of any new technology you hope to deliver. Preproduction proves both that your team can make the game and that the game is worth making.

The work products of this phase are the game design document (GDD), the art production plan, the technical design document (TDD), and the project plan, which itself is actually a suite of documents. Preproduction culminates in the delivery of the game prototype: a working piece of software that shows why the game is fun to play.

The Game Design Document (GDD)

The GDD exhaustively details everything that will happen to the player in the game. The features in this document become the requirements from which the art production plan and the technical design document are made.

During the development cycle, the GDD should always be the most current representation of everything there is to know about what the player experiences in the game. This should include complete information about the gameplay, user interface, story, characters, monsters, AI, and everything else, down to the finest detail.

Such a document, if committed to paper, would be the size of a stack of several telephone directories, impossible to maintain, read by no one, and almost instantly out of date the minute it was printed. Instead, put it on your internal network as a set of Web pages or use another robust collaborative document or Wiki solution.

Maintaining your documentation online not only has the advantage of keeping the design up to date, but also enables everyone on the team to have easy access to everything at all times. The time savings to the group over the weeks and months of development will be enormous.

Starting the Art

During preproduction, you establish the look of your game and decide how the art will be created. The designer, art director, and concept artist collaborate on developing the visual style of the game. The concept artist makes reference sheets for other artists to work from. Establishing this art bible early on helps to orient new artists coming on to the project and ensures the final product will have a consistent visual style throughout.

Most of this art can take the form of pencil sketches, but it is often useful in selling the game to develop a few glossy pieces that capture the high concept and pack a good visual punch.

In the early stages of the game, you should assemble a visual reference library of images that reflect the direction you want the art to take. These images can come from anywhere—magazines, travel books, movie posters, and classic paintings—as long as they are used only for guidance and do not find their way into the final product.

Establishing the Pipeline

The production pipeline is the process by which you go from concept to reality, from an idea in someone’s head to actual figures and gameplay on the screen. For example, to create a functioning monster in an action game, you must find the most efficient way to move from a designer’s specifications, to a concept sketch, to a 3D model, to a skin for the model, to animation for the figure, to applying behavior scripting to the character, to dropping it in the game and seeing how it works. All the tools you select along the way must be compatible. They must be able to “talk” with each other so that the work you do at one step can be imported to the next step, manipulated, and passed to other team members down the line.

Asset Lists, Budgets, Tasks, and Schedules

The production plan also includes the first draft of the asset list, team task lists, equipment budget, costs, and any other planning documents, including a test plan. Like the GDD, this plan must be updated and kept current throughout the life of the project.

The Technical Design Document (TDD)

The TDD sets out the way your lead programmer plans to transform the game design from words on a page to software on a machine. It establishes the technical side of the art production path, lays out the tasks of everyone involved in code development, and estimates the time to completion of those tasks. From these person-month estimates, you learn how many people you need on the project and how long they will be with you. This, in turn, has a direct effect on your budget.

In addition, the TDD specifies:

■ What core tools will be used to build the game

■ Which tools are already in-house

■ Which tools have to be bought or created

■ What hardware and software must be purchased

■ What changes must be made to your organization's infrastructure (for example, storage capacity, backup capabilities, and network speed) to support development

The Project Plan

The project plan is the road map that tells you how you are going to build the game. It starts with the raw task lists in the TDD, establishes dependencies, adds overhead hours, and turns all that information into a real-world schedule. The final project plan is broken down into several independently maintained documents.

■Personnel Plan: The personnel plan is a spreadsheet that lists all the specific people who will be working on the game, when they will start, and their salaries. It is a road map to determining both the requirements for and costs of human labor on the project.

■Resource Plan: The resource plan calculates all the external costs of the project. It takes from the technology plan the timing of the hardware purchases to support internal personnel, and it estimates when the external costs (such as voice acting, music composition, or motion capture sessions) will be incurred.

■Project Tracking Document: This is where you keep track of whether you are on schedule. Some producers use project integrated management software for this (such as Jira™), some use proprietary in-house tools, and some use linked spreadsheets. The producer will track lists of individual tasks, effort (time) estimates, dependencies, and deadlines using whatever tool is most efficient to keep the team apprised of the day-to-day progress of the game.

■Budget: After applying the overhead multipliers to the manpower plan, you combine these numbers with the resource plan to derive your month-by-month cash requirements and overall budget for the game.

■Profit and Loss (P&L): The original P&L estimate was made during the concept phase. As development progresses and costs become clearer, the P&L statement must be kept current.

■Development Schedule: Many developers would rather avoid creating a firm schedule and committing to a specific release date, but you owe it to yourself and your company to do exactly that. After a release date has been set, the publisher sets their marketing, sales, and distribution teams into motion. The marketing team books advertisements that will appear in the months running up to the release date. The PR department negotiates with magazines and websites for well-timed previews and feature articles. The sales group commits to promotions with retailers. Changing the release date of the software is likely to ruin all the carefully planned efforts of these groups and result in your game selling far fewer units than it might have.

■Milestone Definitions:Milestones are significant points in development marked by the completion of a certain amount of work (a deliverable). These deliverable criteria should be concrete and very precisely defined, with language such as “Concept sketches for fifteen characters, front, side, and back” or “Weapon #1 modeled, skinned, and operational within the game with a placeholder sound effect, but without animations or visual effects.” Avoid vague deliverable criteria, such as “design 25% complete.” The best milestone definitions are binary: a deliverable version either meets them or fails to, with no room for argument in between. Publishers will often instruct their QA teams to test a milestone deliverable against these criteria, which are often documented in the contract and payment schedule.

Prototype

The tangible result of preproduction is the game prototype. This is a working piece of software that captures on the screen the essence of what makes your game special, what sets it apart from the rest of the crowd, and what will turn it into a hit. A prototype (or pre-Alpha version) that contains enough art and audio content to reflect what playing the finished game will be like is sometimes called a vertical slice.

The prototype not only shows your vision for the game, but also establishes that your production path is working. It also gives testers their first look at the game. If the prototype includes working code, this is a good time to try out some of your tests using your test environment, which itself may be a prototype at this point in the project.

Development

Development is the longest and hardest stage of creating the game. Your development schedule may last six months to two years. Some smaller games can be designed, coded, and tested in less than six months. At the other end of the spectrum, games that are longer than two years in development run the risk of going stale, suffering personnel turnover, having features improved upon or rendered obsolete by other games, or seeing technology made useless by advances in hardware. Any of these problems can cause redesign and rework which, in turn, lead to further schedule delays.

The deceptive part of development is how long it seems at the start. You have a good plan, and it is easy to think that everything can be accomplished with time to spare. This phase of the project can be dangerously similar to summer vacation. At the beginning all you see are weeks and months stretching out in front of you, with plenty of time to do everything that you want to do. Then, as the end draws near, you wonder where all the time went and suddenly start scrambling to fit everything in.

The trick to a successful development phase is to break large tasks into smaller, more manageable tasks that are rigorously tracked. You cannot know whether you are behind on a project if you do not track the tasks. This is something you should assess daily, so you can report to your team lead or project manager whether you are on schedule or need help.

If you are an external developer working for a publisher, your progress is tracked for you in the form of milestone deliverables and dates specified in the contract. The incentive to stay on schedule is clear: if you do not meet the deadlines, you do not get paid. Well-run internal groups use the same structure. Milestones are established at the start of development and there are frequent project status meetings in which the producers get together and go over the schedule in detail.

Alpha

Alpha is the first major milestone of the game project. The majority of the game systems should be coded and implemented. Enough of the game’s content should be integrated so that it feels like the game the team conceived those long months ago.

The definition of Alpha may vary from company to company: Generally, it is the point at which the game is mostly playable from start to finish. There might still be a few workarounds or gaps, and all the assets might not be final, but the engine, user interface, and all other major subsystems are complete and working.

As you enter Alpha, the focus starts to shift from building to finishing, from creating to polishing. This is the time to carefully consider the game features and content to decide whether any must be removed to make the schedule. This is also the time when more testers come on the team to start finding bugs.

Beta

Beta is the second major milestone of the game project. At Beta, all content is integrated, all development of new features or systems stops, and the only thing that should happen thereafter is bug fixing (and retesting). Stray bits of art might be upgraded, or lines of text rewritten, but the goal at this point is to stabilize the project and to eliminate as many bugs as possible before release.

The last part of Beta is often called crunch time. During this stage, developers and testers may stay at the office for days at a time, including weekends. While some teams may feel as though crunch time is inevitable as the final deadlines approach, the truth is that working overtime for weeks on end to the point of exhaustion makes for sloppy mistakes in the game, and those mistakes are less likely to be caught by the QA team if they have been working weeks of overtime as well.

Beta Testing

The Beta test phase not only gives developers valuable gameplay and balance feedback, but it is also a great way for the game team to check for defects they may have missed because there simply were not enough testers to execute large-scale multiplayer test scenarios.

Some games have a Closed Beta testing period, where volunteer testers are either randomly chosen or hand-picked based on information they send in when they request to participate. Depending on the project plan or the results from the Closed Beta, there may be a subsequent Open Beta period, which tests the game at an even larger scale. Other rounds of Beta testing may be defined with their own specific objectives, such as testing how well a massively multiplayer online role-playing game (MMORPG) performs with a fully loaded world server.

Compliance Testing

If yours is a console game that is subject to the approval of the console manufacturer, the final weeks of Beta will include submissions to that company so its testers can verify that the game meets their own quality standards.

A PC game can be sent to an outside testing firm for compatibility testing. This should uncover any pieces of hardware, or combinations of hardware, that keep the game from working properly.

A mobile game may need approval from handset manufacturers, wireless carriers, and digital storefronts like the App Store or Google Play. The handset manufacturers may be concerned about the game’s interoperation with a phone's built-in features, while the carriers may want to be confident that the game will not disrupt service on their network.

Code Lock

At the end of Beta, you are likely to be in code lock, when all the work is done and the preparation of a master candidate version begins. In the case of disk-based games, the disks must be tested before they are sent to the manufacturer. In the case of digital-only games, a final version is prepared with all the specifications required for it to be uploaded to the release server. The only changes allowed to the code base are those that specifically address any “show-stopper bugs” that last-minute testing may find.

Gold

This is the last major milestone in the game development life cycle. The game’s released to manufacture (RTM) version has been thoroughly tested and found to be acceptable when measured against all release criteria, whether from the publisher, the platform owner, or the digital storefront. This version, the one distributed to the playing public, is known as the Gold version, and finishing the game is often known as “going Gold.” This term dates back to when all game software was manufactured in quantities and sold in stores. The version sent to be mass produced on CD-ROMs was known as the Gold Master, and versions created during the code lock were known as Gold Master Candidates (GMCs).

The game is finished. (Or is it?) You can finally celebrate. (Or can you?)

Patches

For better or worse, it seems today as though almost every game gets a bug-fixing patch after its initial release. (We will explore why that is, despite months of game testing, later in the book.)

Immediately after the game’s release, these first patches tend to focus on repairs to outstanding bugs, as well as any new bugs that may have been found by players.

Updates

An update is different from a patch. An update contains additional content created to expand the original game. Many game companies conduct seasonal or holiday events for their subscribers, which could include special limited-time missions, crafting options, or item drops. Many competitive games will introduce new cards, new playable characters, or other new gameplay elements that will affect the balance of the game. These are delivered in updates, with further updates sometimes necessary to fine-tune balance.

In any case, an update is a mini-project and needs to be handled like one, with testing, milestones, and all the other factors associated with good software development and testing.

READY, TESTER ONE?

Now that you understand the role of testing in the game development process, in the next chapter we will examine some of the qualities and discipline necessary to become a professional game tester.

EXERCISES

1. The number of testers on a video game project will change over the course of development. Will there be more testers on the team toward the beginning or toward the end?

2. What is the main difference between a patch and an update?

3. Why is it usually a bad idea to print out a game design document (GDD)?

4. Which person is responsible for planning and orchestrating testing activities performed over the course of a game development project?

5. What is the purpose of a game prototype?

6. How can game testers contribute to the management of a game development project?

7. What are the three major milestones of a game development project, and why is it important to have them precisely defined?

REFERENCE

Mullich, David. 2015. “Sorry, There Is No ‘Idea Guy’ Position In The Game Industry.” Last modified November 23, 2015. https://davidmullich.com/2015/11/23/sorry-there-is-no-idea-guy-position-in-the-game-industry.

CHAPTER2

THE BASICS OF GAME TESTING

Developers do not fully test their own games. They do not have time to, and even if they did, it is not a good idea, for reasons that will become clear later in the book.

At the beginning of the video game era, the programmer of the game was also its artist, designer, and tester. Even though games were very small (the size of a modern email message), the programmers spent most of their time designing and programming; they spent very little time testing. If they did any testing, it was based on their own assumptions about how players would play the game. The following section illustrates the type of problem these assumptions can create.

The Player Will Always Surprise You

The programmer of Astrosmash, an arcade-style shooting game released for the Intellivision®