Video Games QA

The Anatomy of a bug: the importance of understanding and the futility of standardization

 

In my decades of working in video games QA I have come to have strong opinions on quite a few things.  None are more important than the importance of thinking about and understanding why you are doing the things you do.  One of the most basic ways this principle can be applied is how a QA tester thinks of a bug.  There is nothing more fundamental to testing video games than finding and reporting a bug, wouldn’t you agree?

Typically, when we start in our first testing role, we are given a form to fill out when we find bugs and are instructed how to fill it out.  At that level, that’s the job you are being asked to do.  But as you grow in your QA career and move up into more senior positions, you’ll often find yourself in a different position; you’ll be asked to create the bug template and process for dealing with them.  This is where it’s critical to understand the “why we do what we do” aspect.

Ask yourself, what purpose does a bug serve?

It is a few important things; it is what gets communicated to the developers who will need to fix them.  It is how we communicate with people who must triage them and decide how they will be prioritized.  It is the digital artifact that goes into a bug tracking system that is used to track a large number of defects all with varying priorities and severities.  A bug needs to satisfy all of those needs and most importantly it needs to serve the organization or product that it is being written for.  And none of those organizations or products are the same, they need bespoke solutions that cater to the actual needs.

The worst thing a QA tester can do is think that there is one solution that fits all, or a standardized bug that meets every need.  There are, however, common threads that can be conceptualized and applied to any tracking system or production process that you are likely to encounter in your career.

Here’s a true story from earlier in my career.  Sometime back in the early 2000’s I was sent to GDC by my employer for the first time.  Of course, the first thing I did was go through the program and find every talk and roundtable that was for QA or even tangentially related.  One of the timeslots I found was a roundtable being hosted by the IGDA QA SIG, the organizers were attempting to collect data on all the fields we in QA were using in our daily work to help them come up with a template for all the things you would want to include in a bug report. 

At that round table, as people took turns going around the circle I listened as people talked about what kind of studio they worked at and what kind of games they were working on; platform first party, Indy studios, cellphone game developers, MMORPGs…the list went on and on.  By the time it worked its way around to being my turn instead of answering I challenged the group with a question.  I said “it sounds like we are all working on very different projects in very different needs and there are a huge variety of bug fields that we track.  Why are we trying to standardize a bug report to satisfy all of us, is that even possible?”. 

As you might imagine, this did not go well.  But in the twenty years since I’ve become increasingly convinced it was the right question to ask.

Since then, I’ve developed a philosophy that a good bug is one that meets the needs of developers, the production process, whatever tracking system you are using and most importantly the game or product you are making.  I’ve distilled this down to what I call the anatomy of a bug. 

In that anatomy there are five things that you need for a good bug:

• Title
• Description
• Steps to Reproduce
• Expected versus actual results
• Meta data

 
It seems like this is something that almost goes without saying, especially for people who have worked in QA for any amount of time.  It is however something that people have a hard time speaking to – and as somebody who interviews game testers from entry level to senior level I find it’s hard to get people to be able to articulate this in an interview.  They know how to fill out the bug forms they have worked with in the past, but they are sometimes hard pressed to explain why these things are important.

I’m here to suggest to my fellow QA professionals that we should all know how to answer the question at a conceptual level and explain why each of these things are important.

The title is a TL;DR of the bug; it should be short, informative and explain what the problem is in a single sentence.  When managing any number of bugs in a game, or trying to work with bug tracking software, you need a good title.

The description is the verbose version of the title.  You cannot always give all the information you need in that short title.  Bugs can be complex; there may be conditions in which it occurs that don’t fit into that summary.  You may want to explain what else you have tried testing so that when the question comes up in a triage process the information is there so it can be judged without having to send it back without questions.  The description is where you have the space to include everything important that won’t fit into a title.

The steps to reproduce are probably the most important part of the bug report.  If a developer cannot reproduce the bug, they can’t be expected to be able to fix it.  Clear and concise steps that walk them through getting the bug to reproduce are key to giving them the information they need to fix it.  Which is important.  Developer time costs money.  Spending time trying to figure the bug out slows down the whole process.

Expected versus actual results is another place where good information can avoid slowing things down.  Some things are so simple that they need no explanation, for example, we all know the game shouldn’t crash.  But not all bugs are that simple and when a bug reaches the triage process which can often include people who are making decisions who do not know the intimate details of the project it can be helpful to explain why something is a bug.  This is particularly true when a bug is a usability issue or a less tangible quality issue than something as simple as a feature just not working or crashing the game.

The final part of a good bug, the meta data is very much a catch all and where a bespoke solution can be crafted for whatever game or project you are working on.  Your mileage is going to vary, there are some things that can be considered standard such as the change list it was found in, the priority or what discipline the bug is going to go to or even what platform the bug occurs on (i.e Xbox, Playstation etc…).  Quite often a good bug needs associated data and media like a screenshot, video or log files that will help explain the bug or provide clues for the developer assigned the bug.  And lastly, the meta data is where we the bespoke solutions for the game or project you are working on can be provided; the list of custom fields I heard described by my fellow QA testers at GDC all live here.

When I interview a tester for a QA role, whether it be a junior or senior tester this is one of the key questions I ask; how would you describe the important parts of a good bug report?

I never expect to get an answer that is exactly how I would answer it.  But by listening to what people say and asking clarifying questions I can often get a good sense of whether the interviewee really understands why we do what we do when we find that bug.  The best testers are usually the ones who have thought about and can speak to the “why”.