Public Test Servers – Why, When, How

Public Test Servers – Why, When, How

Some game studios use a public test server to generate advance player testing of upcoming patches and game releases. Some companies choose not to use public test servers.

They have names such as Public Test Realm (WOW), Test Center (Marvel Heroes) and even the Public Beta Environment (League of Legends). 

This article discusses the pros and cons of each approach and what an individual designer can do to help their designs if they choose to use a public test server.

First, the pros and cons of each approach.

Advantages of Using a Public Test Server

Designers will have an opportunity to observe player reaction to new content before committing to the final build. Obviously, this can help to avoid launching a live patch with a particular element that players hate, which could have easily been changed if the feedback came before the final build. This includes balance changes that occur with a patch (also known as nerfs, buffs or “tuning”).

Players will feel more involved with the development process if they see their test server feedback being appreciated and acted upon.

Unforeseen behaviors can be discovered and addressed on a public test server. Even with a huge, highly-skilled internal testing group, thousands of simultaneous players will always find a way of dealing with content that was not anticipated or tested internally.

Assuming that you’ve copied player data to the test realm, test server results can find technical issues that QA testing may not find. “Real” player states or player inventories can have unexpected technical difficulties that a standardized QA pass may not find.

Players can stress test certain multiplayer and public game elements of the patch that QA can’t, simply by providing large numbers of players hitting a certain public area or logging in simultaneously. Some QA tools allow this to be simulated,but that’s not always a feature of those tools.

A public test server can help QA do its job. It’s impossible for QA teams to test absolutely every interaction and behavior that players can create. Test servers can turn up issues that QA can then quickly repro and even use as a starting point to find other issues. Done right, this can be a huge advantage for QA.

Disadvantages of Using a Public Test Server

Serious cost – both in money and manpower. The server upkeep cost, the admin cost, the community management cost and the cost to designers. A successful test server run includes preliminary patch notes and should include at least some interaction from design and community about the things that are on the test patch. There is an argument that it’s better to pay the costs of public testing up front, which could be much cheaper than the cost of releasing something questionable in a live patch.

Spoilers are at risk. You may want to save a story surprise for the live server, or keep an item/zone/enemy hidden until the big reveal actually experienced by the player in the live game. In some ways, discovering a surprise on the test server can be decent promotion for the game and can work to get more players excited about logging in to the live servers to experience the new content for themselves, so the “spoiler” issue can be both an advantage and a disadvantage.

Unfinished business. Test build timing and scheduling may create situations that allow players to experience unfinished or untuned content on public test servers. This can lead to an situation where the “first impression” of new content is negative. Keep that in mind if an when you release unfinished content for public testing. It’s particularly bloody when something appears to be “overtuned” on the test server and gets “nerfed” when it’s launched for the final public build. Angry Reddit posts are sure to follow.

Sales. In a few cases, particularly with free to play games, your studio may be uncomfortable with allowing players to experience content for “free.” The concern may be: Allowing players to use through the content for free means they won’t buy that content on the live server. I’ve heard this concern a few times, so I’ve included it, but I do think it’s a minor issue and only applies to a subset of games – and even then the number of players who use a test server tends to be tiny percentage of the overall player population.

 

So, Should You Use a Test Server?

In my experience, it’s usually better for the players and the development team to have access to a regular public test server, and therefore I recommend it for most studios. It’s definitely not mandatory, but usually the upside is much greater than the downside.

However, if you have a very large internal test team (that includes QA and representatives from other disciplines), you could get most of the benefit of a public test server without the downside of spoiling content. You can still use most of the advice in this article while working with an internal test team.

 

Test Servers and Communication

Here are a few suggestions when it comes to communication about test servers:

• Have patch notes immediately available when you put up a patch on the test server. It’s important that players understand what has changed and what new content is available. You want designers to put the changes in their own words, with their perspective on the changes before the public gets into the public test server. You don’t want the one loudest, most negative player to start the conversation about any changes, nerfs, or new content on the test server. The community manager or actual designer of the content should create a clear and forthright post about the content and time that post to launch just before the test server is open.

• Create a dedicated forum section and a dedicated Reddit thread to discuss the content on the test server. You don’t want dozens of new threads popping up about the test server. You want everything in one place so community, QA  and Design can review everything in one place. You usually don’t want the test server conversation to overshadow the regular day-to-conversations that take place. (There can be exceptions where you want the conversation to center around the test server content because it’s so positive and exciting that you want players to be focused on it).

• Don’t bullshit. Ever. Understand and assume players are attentive and they will clearly see any nerfs or “player unfriendly” items that are included in the test server. Don’t hide from it, otherwise it will fester and become the only narrative. Ensure controversial issues are addressed directly in the test server patch notes and in a dedicated test server forum post by the community manager or designer before the test server opens.

• Explain clearly why changes are being made, with honest upsides and downside and invite feedback from players in a designated area. Ask forum moderators to direct any feedback into those threads and make sure the threads are being monitored and updated whenever players make a fair point or changes are being made. Don’t give the perception that you are ignoring feedback. Explain changes you’re making based on feedback or explain why you’re sticking to your guns and why it’s good for the game.

 

 

Test Center Duration

Games obviously differ widely in scope. Something like World of Warcraft or Marvel Heroes or Destiny has a ton of moving parts, game modes, loot tables, thousands of items, and a variety of enemies, dungeons and raids. Even seemingly small changes can have wide-ranging repercussions.

Something like Overwatch is a more contained and requires less test time for a new map or hero. A bug with a new hero in Overwatch or a new map in Call of Duty will require less cleanup and places less pressure on extended test server periods.

Here’s an important thing to remember – Plan enough time to allow at least twofull iterations of your test server content and enough time to test the contents of those iterations. Your testers will feel much more appreciated if they see results of their testing work and you will have a better final build of the patch when it eventually goes public.

The scope of the patch content will also determine test server time. If you are testing a single new Crucible map in Destiny, you can do that in a week or two.

If you are testing a new hero in Marvel Heroes, you probably want to allow two weeks minimum, for enough iteration time to occur, and two or three total test server patches with adjustments to that hero included.

For a new raid in WoW, The Division or Destiny, you probably want to allow a solid three weeks of testing, because there are many more things that can go wrong with environments, mobs, powers, raid grouping, raid UI, loot and difficulty tuning.

For full-size expansions or large new zones of the game that are populated by enemies, new loot, new quests, and so on, you definitely want a generous amount of testing time in the master production schedule. I would recommend four weeks as a minimum, depending on how much polish has already been applied after internal QA testing.

 

 

Test Server Balance / Tuning

One of the biggest test server issues, besides bugs, will obviously be game balance.

The most loyal players will often be the most passionately vocal critics of buffs, nerfs and tuning changes. Go into that situation with eyes open and take it very seriously.

The higher quality your gameplay has, the more players will care about every slight change to balance. It’s a good thing when players care enough to criticize your decisions.

I’d like to write an entire article about translating player communication, but basically, you need to have a very thick skin when reading test server feedback. Read it carefully, with an open mind. Take it personally and seriously, but don’t be defensive about it. When there are tuning complaints, take them seriously and discuss them with the appropriate subgroup of designers within the studio.

Strongly consider a small group of “expert” players to consult about tuning feedback as well. A trusted group of analytical players that know their opinions matter will usually offer great advice and analysis of tuning changes. Keep the list pruned to ensure you always you always have 5-15 active players in the group that will test all types of content, weapons, and classes. Allow them to present arguments among themselves if they disagree with each other. A private forum section or discussion group is ideal.

Scheduling Test Server Events

Here’s a secret weapon for getting great testing: Announce specific times for players to test specific features with the dev team and make a big deal of it. Have a few designers or other staff participate in the test and heavily promote it.

Point the players to a specific feedback thread to discuss their findings. Have a designer, community manager or producer get ready to gather feedback and be absolutely sure to followup with a “thank you” to players after the test session, so they feel good about it and will participate in future events.

This method works very well and will ensure that you get a decent number of players testing the content you need feedback about.

Sample communication:

“From 4:00 pm-6:00 p.m. server time, we invite all players to test our new PvP game mode. Designers and other staff will be playing and we hope to see you there. We’ll collect feedback on the new mode on this forum…”

 

Tips and Reminders

• Create designated forums and threads for test server feedback before the test patch goes up.

• Have complete and clear patch notes available just before test server launches.

• Tell the players what you would like them to test.

• Have one or more designated test server communicator from community or design.

• Absolutely review test server feedback daily and ensure feedback is communicated to the right people immediately using an organized, prearranged method.

• Allow enough time for two full iterations of test server content, possibly more.

• Schedule and promote times to test specific content to ensure critical mass.

• Be incredibly gracious and thankful toward test center players. They are doing work that saves you time and makes your game better.

 

In Conclusion

Test servers are great and will help most studios that don’t have the luxury of a large QA group working in concert with a robust internal testing group. However, they must be approached in a highly-organized, responsive manner to offer great results.