Bringing your QA department in-house

When we decided to bring our QA department in-house, we started talking about the values and principles on which to build it. Here’s how we came to the four fundamentals which we wanted to be the foundation of our philosophy on testing, quality, and teamwork.

I’m Andy, and I’m an Agile Evangelist. Being an agile practitioner means that I believe agile is the best way to develop software, and being an evangelist means I’ll tell you about it! I am the agile project manager here at Electric Elephant Games and recently we’ve been working on the large undertaking of bringing our QA department in-house.

Being a project manager rather than a quality assurance expert, my first port of call was hiring an agile QA manager, which got us all thinking about what we wanted our QA department to look like and the principles on which to build it.

This article is the result of those discussions, now established internally in the QA department as our four fundamentals of QA in gaming.

We are all stakeholders

One of the mistakes many businesses make is thinking that the term stakeholder refers only to C-level execs and ‘The Board’. The definition of a stakeholder in this context is interest, investment, and involvement in the project, so in fact anyone that has a stake in the project, including developers, QA, artists, and business analysts, is a stakeholder.

When this interpretation of stakeholder is adopted, magic happens. I’ve seen a business analyst present a great concept for a logo, a developer compose a great idea for a soundtrack, and even a product owner propose a technical solution which was evading the developers. But most strikingly, I’ve seen QA come up with all of these things and more.

This is because in gaming, QA have an abundance of experience in playing games: they’re doing it up to 40 hours a week, and maybe more in their free time. Quality assurance in gaming, if done well, is as much about finding defects as it is about finding improvements taken from the extensive experience of the QA team.

For example, the settings panel for a game I worked on was practically designed by the QA lead using examples from other games, both our own and others they had experimented with, and it really made a difference. The settings panel isn’t the most interesting part of a game, but ours looked really slick. It’s the small touches like this that make the difference between good, and outstanding.

There is a certain artistry to quality assurance in any industry. It isn’t just checking that things work, it’s checking that the product feels good. QA testers and engineers tend to really care about their products, which makes them all stakeholders.

We are part of the process

In the waterfall model, software development proceeds in a fairly linear fashion: requirements gathering, design, implementation, testing, maintenance. Each phase can only really begin once the preceding one has finished. In agile, this is not the case.

Agile software development works in small iterations to deliver product in chunks. The features in these iterations must be potentially shippable, meaning they could theoretically go live, which requires that they have been fully tested by the end of the iteration.

To achieve this, we merge together the waterfall phases of implementation and testing within an iteration. This means that QA is embedded into development rather than after the fact. This has several advantages over the traditional waterfall model.

Firstly, we gain the benefit of the aforementioned ability to use QA experience to suggest improvements during development rather than after the developer has finished. This makes it much more likely for those improvements to make it in to the product rather than being too late in the process.

Secondly, it eliminates a bug fixing phase after development, because all defects and improvements have been identified and addressed during the development phase. Finally, it allows a cross functional team to work together in getting features over the line, rather than getting held up in endless testing.

When I create my project plans, I don’t account for a bug fixing phase after development is completed, because we don’t need it. Our ‘definition of done’ is that the coder finished coding it, the tester certified it as bug free, the artist approved that it was implemented according to the design, and the product owner approved that it was developed according to the specifications.

This means there is no bouncing back and forth between test and development at the end of the project, it’s already done, because QA was part of the process, not after the process.

Diagnostic engineers, not just testers

Part of quality assurance in gaming is testing. This is simply following a test script, checking for the expected outcome, and logging a pass or fail; but this is only the beginning. If a test fails, a QA engineer can diagnose the issue and feed back to the developer with information on why the defect occurred. This, along with an agile testing philosophy, leads to fast turnaround times and less time spent in investigations.

When a player clicks the spin button on a slot game, a whole chain of events is set in motion. The game makes a request to the remote gaming server (RGS) for a spin, which then performs a server-to- server call with the provider to validate the balance. If all is well, the provider responds ‘OK’ to the RGS, the RGS responds to the game with the spin result, and the game can unfurl the spinning reels, the winning fanfare, and the coin explosions to the player.

But what if all is not well, and instead of a big win celebration the game simply displays the dreaded message: an unexpected error occurred (error code 0)?

A major difference between testing and diagnosis is that a lot of testing can be automated by a machine. With technologies like Selenium, a computer can simulate the appropriate key-presses, mouse-clicks, and finger-taps, and then compare the game with a screenshot to verify the expected outcome. It’s even possible to strip the reels from a slot machine and write a program that interfaces directly with the code to run test suites. But a machine can’t really diagnose every single bug that might come up.

A QA engineer sees a slot machine the same way that Neo sees in The Matrix: they see the random number generator underneath the spinning reel, the HTTPS handshake behind the loading screen, the code behind the clicks. They can reproduce the error, produce error logs and debug information, diagnose what the root cause is, and attach all of this information to the defect report before handing it over to the developer to fix. This partnership speeds up bug fixing without slowing down feature development – so you can have your cake and eat it too when QA are diagnostic engineers, not just testers!

We are all in this together

When I’m interviewing QA staff, I like to ask the question ‘how do you QA in an agile way?’ The answer I’m looking for is that QA should happen side-by-side with development, but an even better answer is that QA should be involved from the very beginning of the project.

Slot machines are completely random. Every spin is an independent, standalone game in itself so the chance of a win is the same on every spin, meaning that it is entirely possible to win the grand jackpot twice in two spins, although the odds of this happening are in the quadrillions. This makes it very difficult to test winning a jackpot!

Having QA involved right from the start means they can identify areas like this from the game design document and ask ‘how can we test that?’ Which means that we don’t get into situations where a QA engineer is unable to test a feature and has to ask the developer to create testing tools, delaying the project due to extra, unexpected, un-estimated work.

Then we come to compliance, which is the phase of development where the game is tested by a gambling regulatory authority. The RTP (return to player) of a slot machine is the percentage of the money going into it that gets paid out to the player over time.

The RTP is accurate to four decimal places and to be certified by the regulatory authority they simulate as many as five billion spins on the game. If the RTP isn’t accurate enough, it gets kicked back to us.

Unlike some other industries where QA is the gateway to market release, in gambling the QA department is just the gateway to compliance. This causes a fundamental difference in the way that testers and developers work together in the highly regulated industry of gambling.

The aim is to make the game solid enough to get a legal seal of approval from the regulatory authority, or no-one ever plays it. This barrier to getting our games to our players really enforces the work ethic that we are all in this together.

Bringing your QA department in-house

I’ve seen some industries where QA are seen as the enemy: a developer spends three days working on a new feature, and a QA engineer spends five minutes playing it before they find and log a bug – and they have the audacity to be proud of this!

So far, I haven’t seen a developer punch a tester because they found a bug in their code, but this apparent dynamic of developers creating features and QA finding bugs in them can sometimes feel like it’s them against us.

All of the preceding principles are shot through with a common thread that blends them all together: it’s teamwork. If that seems stupidly obvious to you, then you’re correct, because it really is that simple.

If there’s one, single, golden rule that we want our QA department to be based on, it’s that QA aren’t ‘them’, QA are ‘us’.

Andy Kimbrey, agile project manager, Electric Elephant Games


Related Posts