Playtest blog

VR Playtest Best Practices Featuring Schell Games


This post orig­i­nally appeared on the Oculus Blog on July 13, 2020. You can view it here.

The team at Schell Games has been providing education and enter­tain­ment game devel­op­ment since 2002, and more recently have released titles like I Expect You To Die and Until You Fall. Today we hear from Alexis Miller, Director of Product Manage­ment, and Tera Nguyen, Producer, as they provide a number of learnings and insights from their expe­ri­ence playtesting 2D and VR titles.

If you’re inter­ested in learning more on this subject, be sure to check out the recently published Oculus Playtesting Guide, which includes collab­o­ra­tion with the team at Schell.

We’ve done a lot of live moderated playtests over the years, and we recently found success with remote unmod­er­ated playtests and remote live (video) playtests as well. The tips shared here are the result of a lot of shared wisdom from our past and present colleagues at Schell Games. Each method involves different planning and pros and cons, but all methods can provide valuable feedback.

All Methods

All methods of our testing require the following:

  • A recruit­ment plan for finding and sched­uling testers.
  • A technical plan for how your players will get access to the game.
  • A way for players to sign or agree to your company’s non-disclo­sure agreement.
  • A checklist and/​or script that includes infor­ma­tion about the specific equipment and systems, ways to make your playtesters comfort­able with the equipment and with sharing their feedback, and plans for data gathering.
    • A list of questions (prepared in advance) to ask the player about their expe­ri­ence playing your game is really important. While impro­vising some questions can be good, don’t improvise all the questions.
    • The fewer surprises and frus­tra­tions the player has during the setup process, the better the expe­ri­ence and feedback will be.
Moderated In-Person

If players are coming to our office or other location, they’ll need a lot of details that one might take for granted but are important to them and will make it a better expe­ri­ence for them. These details include:

  • Where to park and/​or what public trans­porta­tion is nearby.
  • What time to arrive and what time they should expect to be done.
  • If they need to check in at a front desk or security.
  • If they will be paid for their time and how much.
  • If you will be providing any food for them (we always ask if they have food allergies).

We’ve recruited testers from our own marketing email lists, friends and family of the studio, and by reaching out to local high schools, colleges, or univer­si­ties with which our company has rela­tion­ships.

Remote Unmod­er­ated

For unmod­er­ated, remote playtests, we recently started to use the platform UserTesting. UserTesting records the tester’s screen — either on a PC, tablet, or smart­phone. This platform gives us access to nearly 2 million people in the US, Canada, and other countries around the world (they don’t all have VR equipment though). To test on this platform, there’s a lot of upfront, pre-test planning work, but we save time by being able to reuse our test questions and tasks on future tests and not having to spend our time moder­ating each test indi­vid­u­ally.

First, we launch a recruit­ment test to find testers in the UserTesting panel who have the right hardware (like an Oculus Quest) and can demon­strate their ability to cast their Quest on the UserTesting platform — either to their phone or a PC. This 10- to 15-minute test doesn’t require them to download our game yet. It’s a way to make sure the real playtest will go more smoothly from a technical stand­point since we aren’t there in-person, or even virtually, to help them. During recruit­ment, testers also have the option to provide their email address. This is used for subse­quent playtests, as we give them access to our game through our Oculus devel­op­ment release channel.

In order for testers to access our games that are in devel­op­ment, we created a playtest release channel on the Oculus Dashboard and added our testers’ email addresses whenever we launch a remote unmod­er­ated test.

In these unmod­er­ated playtests, we walk testers through each step one at a time via written instruc­tions. There are important, but basic, tips to remind the player of in an unmod­er­ated, remote session. These are a few sugges­tions:

  • Make sure the tester’s headset and controllers are charged before starting the test.
  • If they are casting from Quest to their phone for the test, it’s polite to remind them to turn off push noti­fi­ca­tions for privacy reasons.
  • Turn up the volume on their Quest so that the UserTesting screen recording picks up the in-game audio.

We want to make sure that we have a diverse group of playtesters, so one thing that’s really useful is collecting demo­graphics of our testers and asking them a few questions related to acces­si­bility. We recently started asking players which hand is their dominant hand (or if they’re ambidex­trous) and if they identify as having any disabil­i­ties or acces­si­bility needs.

Most players can cast from their Quest to the Oculus App if given the right instruc­tions and common trou­bleshooting tips. These are some of the tips that we share with testers that seem to help:

  • Make sure Bluetooth is turned on in your headset and phone.
  • Make sure you’re signed in with the same account/​email address on the Oculus App and your headset.
  • Make sure your headset and phone are using the same WiFi network.

With this mode of testing, we learn a great deal from players when they think out loud as they play, so at the beginning of the test, we remind them to talk while they perform the required tasks.

Lastly, because these test instruc­tions are being read on the player’s phone or PC, we try to set them up with all the needed infor­ma­tion about the test before they go in their headset to play the game. They only have to answer questions when they’re done playing, so there isn’t a lot of lifting their headset up and down, going between their headset and their phone or PC screen. Besides it being uncom­fort­able for the player, this behavior can also discon­nect casting and cause other technical problems on the UserTesting platform. This also provides a more authentic play expe­ri­ence by not inter­rupting their play.

Remote Moderated (Live)

For remote moderated (live) tests, we have scheduled several 30-minute to 1‑hour long live sessions with our testers over video confer­ence or the UserTesting platform. Our designers act as moder­a­tors to give testers instruc­tions on how to set up casting and how to download the game. They also help whenever testers have technical diffi­cul­ties or get stuck, frus­trated, or unsure what to do to move forward in the game. During these live sessions, designers can ask testers open-ended questions like, What do you think you should do next?” or, Why did you perform x,y,z inter­ac­tions?” in order to gather insights on the testers’ expec­ta­tions.

Remote moderated tests are specif­i­cally useful when a product is still in its early devel­op­ment stage and requires addi­tional guidance or prompting. Similarly, it is also useful for testing later levels or parts of the game that may take a lot of time or skill to play through.

In order to ensure that testers have every­thing they need for the live tests, we follow similar protocols with the unmod­er­ated tests (mentioned in the section above) except that the moderator is reading a script with the instruc­tions out loud instead of the tester reading the instruc­tions on their PC or phone.

It is crucial for us to be able to commu­ni­cate with testers while hearing and seeing their gameplay. Therefore, we usually spend the first 10 minutes of the live tests helping testers set up their casting method and video/​audio settings. The most successful method we have expe­ri­enced is when the tester casts their VR view to the Oculus App on their smart­phone while screen-sharing it to us over a confer­ence video call.

Another easy option is for testers to stream their gameplay privately to a Facebook page; however, if your game has not been announced or released to the public, you may not want to pursue this option. While there is a private” setting, it could be easy to miss this setting and cast the playtest publicly to the tester’s Facebook page.

Test Facil­i­ta­tion

Here’s what we’ve learned from facil­i­tating VR playtests, including some process and tactics that have helped ensure we gain action­able insights.

Moderated In-Person
  • Try to limit the number of people in the room with the playtester and announce when new people enter the room since it can be uncom­fort­able for a VR tester to not know how many people are near and watching.
  • Clean the VR headset materials while the playtester is watching. It’s another way to build rapport and trust.
  • Try to limit developer talking during the playtest, espe­cially if it’s not directed to the playtester.
  • Create an isolated, large testing space with a soft floor so the playtester knows they’re in a safe space for testing.
  • Show the playtester the headset and controllers, and offer to adjust the straps for them.
  • When asking the player questions, wait for their answer longer than feels natural before moving on so that they have time to think and respond. It’s amazing how much more talkative players can be when you give them extra silence to fill.
  • One good way to get more infor­ma­tion is by asking the question, Can you tell me more about that?”
  • If a player becomes stuck to the point where inter­ven­tion is needed, don’t just give them the answer. Try to replicate the players’ thought process as much as possible so they can find the solution on their own. For example: Imagine you notice that the second panel on the left has some strange markings on it.”
  • It can help to remind playtesters that there are no right or wrong answers to questions and that all of their feedback is helpful, even if they feel frus­trated or confused.
  • Show playtesters how to calibrate their screen with the Oculus Home button so they can have a centered view of the game.
  • Make sure that the tester sets up the appro­priate boundary to play the game. In our case, we develop a seated gameplay expe­ri­ence and it is different from a Quest stationary boundary.
Remote Unmod­er­ated
  • One of the great things about this method is that the player is in their home with their own headset, so they are playing in their natural envi­ron­ment.
  • Because questions have to be typed in advance and then read by the player, be sure to keep the questions as open-ended as possible as there is only one shot. Avoid questions that they could answer easily with Yes” or No.” That’s usually not enough infor­ma­tion.
Great questions to ask playtesters:
  • How often do you play video games? What are your favorite games?
    • This question can help determine their expertise and what kind of player they are.
  • Did you feel any discom­fort, physical or otherwise?
    • This question can help catch occur­rences of discom­fort, account for body type, and reveal issues with the content (diversity, repre­sen­ta­tion, etc.)
  • What was your favorite part of the expe­ri­ence?
  • What was your least favorite part of the expe­ri­ence?
  • What was the most frus­trating part of the expe­ri­ence?
  • When did you feel most clever?
    • This is espe­cially useful for puzzle games.
  • What do you think we should change first before we test with others?
  • If you had a magic wand and could add, change, or remove anything, what would you wish for?
  • How would you describe this game to your friends?
    • This question can be useful for making sure design goals are being met. For marketing, it is helpful to hear the words that players use to describe the game. These state­ments can be used to hone store descrip­tions and marketing materials.
Data Analysis /​Post Playtest

Next, we’ll look at the processes, tools, and data points we leverage to analyze playtest results.

Remote Unmod­er­ated

The platform we’re using creates a recorded video of each playtest and their written responses to any post-recording questions we send them. It also gives us demo­graphic infor­ma­tion about each player, including age, gender, profes­sion (if any), household income, gaming genres they like, and if they have any children. It also allows us to create short video clips and highlight reels with tags, which makes it easier for multiple people on the team to review the relevant parts of the videos. For example, when using a tag like #play­erdies, it is easy to search for all the parts in the videos where the player died and analyze any patterns or surprises.

These clips and tags have time stamps too, so we can measure how long it takes players to perform certain actions, like completing the tutorial or success­fully completing a level.

All Testing

We’re working on building a pipeline to receive and analyze results using Unity Analytics. This platform promises to provide data on the most common or least common player expe­ri­ences in the game, such as what the most common reasons were that a player died. We create target behavior ranges in advance and aim for the data to fall within those ranges. Once we have enough data to review, we can make changes to the game and observe the results.

Further Learnings

Don’t be afraid to playtest early, even if the game isn’t totally playable. Most devel­opers find it valuable to gain player insight from proto­types and unfin­ished levels. It can help to remind players that the game isn’t finished so that their expec­ta­tions are realistic.

Make early prepa­ra­tions to include playtesting in the process and to allocate time for people on the team to prepare the tests, do the recruiting, moderate the tests, analyze the feedback, and, of course, to make changes in the game. It usually takes longer than expected, but it is completely worth it.

Thanks to our partners at Schell Games for these insights into their playtesting process. For more best practices, recom­men­da­tions, and example docs to help improve your VR testing process, review our recently published VR Playtesting Guide where we go deep on playtest planning + surveys, as well as facil­i­ta­tion and post-test data analysis.