Game Surveys: 5 tips to get advantage of an underutilized tool.
I have been working in the industry for 8 years now and up until very recently I always felt like surveys were not worth the effort and time, and that the results extracted out of them were very questionable.
I am not going to lie, I am the type of product manager that looks at data, the economy, the hard facts and that plays his game to death to find out insights and reasons why. I’ve traditionally left surveys out of the equation (if anything I’ve taken the ocassional peek at the forums and the discord servers).
But things changed last year, when I ran a quick survey among our players in Bit Heroes. I had to take over a game in flames (while managing my other projects), the data schema was not reliable and I needed quick feedback to make informed decisions. For $1 worth of gems in game, we got insightful answers and ideas from more than 500 qualified players, most of them had been around for 2-3 years in the game.
That first survey encouraged me to do some investigation and to dive into best practices to extract the most value out of each survey (I am an optimizer after all) and that’s what I want to share with you in this post!
1.- (Company) Size Doesn’t Matter.
Whether you are a one-man studio or a huge company, surveys don’t need to be expensive, there are plenty of free solutions like Surveymonkey or Google Forms that you can tap into. Some of these solutions even offer analysis services if you are up for that.
Surveys are flexible and easy to prepare. They can be conducted online and once you master these tips you will be able to have a solid survey up and running in no time.
You can surface them up through different channels (in game with a pop up or a placement in your newsfeed, social media channels, forums, mail lists…) so this shouldn’t be a concern.
Overall, preparing a survey shouldn’t take you more than 10-15 minutes and your community will appreciate that you ask and listen to their feedback.
2.- Be Precise With Your Questions.
Particularly if you want to explore how players feel about your game. Here are some quick tips:
Ask them to mention their top 3 and bottom 3 things in the game and the reason why that’s the case.
Ask them how would they define the game to a friend. This helps them focus on more tangible components.
Ask them objective questions, particularly around usability. For example, instead of asking them if the tutorial level was clear, ask them what they remember learning throughout that level (or at a specific moment).
Only ask one question at a time. You don’t want to overwhelm your participants with several questions simultaenously.
Avoid leading questions. Even if you don’t intend to, certain words or phrases can introduce bias and personal preferences into the equation. Generally speaking, try to provide only as much wording as the participants needs to provide an answer.
3.- Use Point Likert Scale Based Questions.
A Likert scale is essentially a scale used to represent people's opinions and attitudes towards a topic or subject.
Likert scales usually range from 1 to 4, 1 to 5 or 1 to 7.
Ask your players how much they agree with certain statements or claims, for example:
Pros of using a 1 to 4 point scale.
Usually, Likert scales are odd-numbered scales. The 1 to 4 point scale is the exemption to the rule.
This type of scale forces the player to fall into the “Agree” or “Disagree” bucket, while it still provices some level of depth to the answer. There’s no middle point so you should get a direct actionalbe out of it (this can also distort the results, but if you need a Yes or No answer it might be your best approach).
Pros of using a 1 to 5 point scale.
Probably the easiest to understand of the 3.
Tends to produce better distributions of data .
Pros of using a 1 to 7 point scale.
The most accurate of all the scale options presented here. It gives a better reflection of the participants’s evaluation.
It is considered to be the best solution for UX questions.
Typically I would recommend to stay true to one type of scale for each survey for consistency purposes, but there’s nothing preventing you from mixing things up.
4.- Test Your UI Designs
If you want feedback on a new menu or UI, you can present your participants a screenshot with a letter or number associated to each component of that menu and ask them to describe what they believe each element does.
Early UI Survey example from one of our latest games.
This can be somewhat extrapolated to other UI elements, such as icons. Ever had a discussion about whether or not that icon transmitted what you wanted to? Now you have a way to get actionable data: send a survey in-house!
Skill Icons in Ori and the Will of the Wisps. How clear are they?
5.- Structure Matters.
We have already talked about some of these in the other sections, but having a clear short list helps (I guess).
Define a clear goal: What is your intention with this survey? It is not the same to ask for UI feedback than for UX feedback for example. What area of your game are you going to focus on?
Keep the heavy (written) questions for the end: This helps the participants to save mental energy for those questions. Don’t use these too often either!
Don’t make it too long: Try to keep things short, the longer the survey, the more your participants will drop off or leave it for later. Be concise, get to the point (keep your goal in mind).
Incentives work: As mentioned, a little compensation or incentive goes a long way to increase both your number of participants and the number of completed surveys. Whether that is an internal $50 gift card lottery or 100 gems in-game, it is much better than nothing.
Keep it consistent: For example, if you decide to use a Likert scale of 1 to 7, keep using the same one across all your questions. Stay true to your structure and avoid jumping on and off from topics.
Stay away from Yes or No questions: These will hurt the quality of your responses. They are usually biased and they make the participants either agree or disagree without giving him any space to explain why, which is what you should be most interested in.
Only ask one thing at a time. Do not ask for feedback on two different topics in the same question. Keet it consistent and clear so your participants is focused.
6.- Go try it out!
“Hold on! You said 5 tips, not 6!” That’s true, but I wanted to finish this post by encouraging you to start surveying internally and / or externally soon.
We work in games, we are biased by our own experiences and plenty of people and players have feedback or want to say something, but they don’t have the proper channels for that. Surveys are an excellent way to give them the opportunity to do so in an organized manner that is effective and channeled towards certain topics, and most importantly, in a way that is actionable. It is a win - win scenario most of the times and they take very little time and effort to setup.
You might have guessed already that there’s something missing in this post, and that’s analysis. I will work on another post about how to dig into the results of surveys more efficiently (word cloud analysis, distributions, mean and mode…etc.)
