Harness the power of surveys done well
/What do you feel when you hear these words, “Excuse me, but do you have 5 minutes to take a brief survey on….” If you are like most, then you probably grimace at these words. I’m guessing the last time you heard words like this and had been excited was when you wanted to stick it to them because you were unhappy about something.
Most people hate surveys, and survey participation has been going down for years, but does this mean you should not use surveys? Absolutely not! You should stop doing bad surveys!
Surveys are one of the most important research tools. They can provide meaningful qualitative and quantitative data and do so cost-effectively. You just need to make sure you understand when to use them, how to design and execute them, and how they fit with your other research approaches. This article is going to focus on the best practices in using them.
Survey Design
Start with identifying the ‘Why’ survey: Before creating a survey, you should understand the “why’ you are creating a survey. These are the objectives of your survey, and be explicit. There should at least be some hypothesized value.
What will you do with your survey?: Start with a plan on what you will do with the results of your survey that aligns with your ‘why.’ You don’t have to have every item mapped out, but by thinking through some of this prior, you will have a better opportunity to design the right survey.
Design Your Survey with Your Audience in Mind: Your survey should be designed in a way that you gain insights from your target audience in a way that will help you achieve the survey’s objectives and goals. For example, you may be surveying your customers leveraging Net Promoter Score (NPS) to understand how your software and services benchmark to your peers. Alternatively, you may be a services organization that is looking to expand into new service offerings, and you are surveying your audience to understand what pain points exist now to identify if there are new service areas you may be able to help your clients.
Use the Right Question Type for the Job: There are many types of questions that you may leverage in your surveys. The two general types of questions are quantitative questions and qualitative questions. You generally want to have a mix of quantitative and qualitative questions in your survey.
Quantitative questions are questions like multiple choice, Yes/No, and rank the following and similar questions. When evaluating the responses here, you will look at things like the mean, median, mode, variance, and other statistical interpretations. The good news is that depending on how the responses come in, there may be a good deal of confidence that can be interpreted without spending a lot of time reviewing.
Qualitative questions are questions that provide small or long answer textual responses. The best qualitative questions are open questions that encourage respondents to elaborate and share their insights. These open-ended questions are where a lot of the true gold can be gathered from surveys. It takes some time to interpret, but there are tools like sentiment analysis and frequent word usage by question that can be leveraged in some tools, or you can do this analysis yourself.
Throw In Open-Ended Questions for Insights: Open-ended questions can provide some of the best insights you will get from your surveys. Just like good user interviews will help uncover gems through open-ended questions, these similar open-ended questions can lead to insights that you didn’t know to ask.
Keep It as Short as Possible: People are surveyed out. Be an organization that is trusted by keeping your surveys short. The general rule of thumb is that most surveys should be 10 questions or less. There are more complicated surveys that are part of complex research or strong business relationships where there is extrinsic or intrinsic motivation that will offset longer surveys.
Be Honest with the Time it Will Take: There are a lot of surveys that say 5 minutes and, in reality, take 10-15 minutes for most people to take. Understand how long your survey will take and provide the upper range of expectations to respondents sought to be upfront. If they are faster in completing, they will feel better and smarter for doing so and more likely to respond to your survey again.
In order to understand how long a survey will take, you can either have several people take it that are not part of writing the questions or a lot of survey software will provide insight on how long they think it will take to respond.
Use Simple Language: Clear and concise language in your surveys is a must. Don’t make people guess what things mean. This doesn’t mean that you can’t use terms of art for respondents where that term of art is known well.
Use Unbiased Language: Do not use language that will bias your audience. Even using certain words that may relate much more to one segment of your audience than the other can lead to misleading results. Further, if you are leveraging leading questions or examples that prime your audience, then you will have biased responses. If your survey system doesn’t analyze for biased language, then you should have a diverse peer group that can review and help you avoid biased language.
Spell Things Out if in Doubt: In addition to using simple and unbiased language, you want to spell out what is being asked for if there is potential doubt. You might provide an example response or an example scenario that would differentiate what you are looking for and what you are not looking for. The only caution in doing these examples is being careful not to bias your audience by priming them for a response.
Prioritize Your Questions by Building Down, Not Building Up: Start with your most important responses first. You want the most mental attention on the survey spent on the most meaningful questions. Note there is also the potential that people might not complete your survey and leave, so some software will even save those responses even if the user doesn’t finish. Now, if you choose to use these unsubmitted responses, it is another matter.
Only One Question at a Time: Having a compound question means you will have extra response variance and, in turn, less question validity.
Use Smart Branching to Focus the Right Questions with the Right People: Many of the survey providers out there allow for the branching of questions. If a user chooses one response, then the next question can segment the user into another question chain versus another respondent who chooses a different answer. This type of software is especially beneficial in allowing you to submit your survey to a less-known advanced audience.
Segment Your Population Surveyed: If you have several types of customer segments you are surveying, then you can either have a survey designed for each, or you can use smart branching, as mentioned above. The key, though, is really segmenting your audience ahead and understanding what insights you are seeking from each audience type.
Give People an Out: All too often, questions do not provide the user an out. Don’t force an opinion on something if they don’t have one. Having an ‘Other’ or ‘N/A’ option is a wise option.
Ask What People Have Done, Not What They Will Do: When seeking input on what people will do, you understand that by asking questions about what they have done. What have they done to solve a problem? What have they bought in the past? How many times in the last year have they done something? How many minutes do they spend doing a task? All these items relate to items that have been done by the respondent.
Be Consistent with Your Questions: Being consistent in how you present questions to users is valuable. Don’t reverse scales. For example, you don’t have a question 1 (most strongly agree) to 5 (most strongly disagree), and then in the same survey, another question has 1 (most strongly disagree) to 5 (most strongly agree). Don’t change rating scales. For example, you don’t ask one question on a scale of 1 to 5, and then, in the survey, give a scale between 0 and 10 on another question.
Be Consistent Across Surveys: Keep your survey questions consistent over time if you want longitudinal insights. Even slight changes to questions can dramatically increase how people interpret questions and respond. If you change a question and notice an increase or decrease in the next year, then the assumption should be this is likely because of the question wording. Sometimes, changing questions is needed, but you know that if questions are changed, then you are starting your historical data over.
Survey Execution:
How many responses do I need? Whenever determining how many people you need to survey, you need to understand how big of a population you are translating results into, how confident you want to be in your results, and what your expected survey participation rate is.
Population size: This is simply the total size of the population segment you are seeking to understand. For example, let’s say you have 1,000 medium-sized businesses as customers. You could always sample 100%, and that may be the best strategy in some cases, like an annual customer survey. However, if you are doing new product research, for example, then you don’t always want to be hitting up all your customers each time you are doing product research. Instead, you want to identify the sample size so that you can extrapolate the results back on your whole client base with high confidence.
Confidence level: This represents how confident you want to be that the differences in the results are not due to random chance. Most people will use 95% for most research cases but may increase this as high as 99% if they want enhanced confidence.
Survey Participation: We know that survey participation is not going to be 100%. Accordingly, it is good to understand what percentage of people you need to approach or send out your survey to get to your needed survey size. If you don’t know what that percentage might be, then look online at what percentage response rate is typical of surveys like yours. One important note about survey participation size is that low survey participation can often lead to misleading results if viewed from a quantitative lens. However, qualitative learning, even in those low participation surveys, can be beneficial.
There are statistical calculations that are done to determine based on these factors, and good survey population calculators like Survey Monkey and Qualtrics provide. One note is that each of these calculators does not factor in the expected response rate, so I suggest dividing the Sample Size determined by these calculators by the percentage response rate you expect.
Anonymize or Not: If you want the most honest responses, then having surveys anonymized is extremely important. In fact, there is almost no good reason to not anonymize things like consumer surveys and employee surveys. However, when it comes to surveys of business where there is a B2B relationship then there are tradeoffs with anonymizing surveys or not. Having non-anonymized makes sense, and that is in the B2B space where you want to do follow-ups after the survey and/or you want to longitudinally understand how that specific B2B customer views you. Sometimes it is best in B2B relationships to have an extremely brief (oftentimes just Net Promoter Score) anonymized survey. Then, you should have a non-anonymized survey that is more of a longer report-card-like approach where you will have a longitudinal understanding of your customer and be able to follow up.
Get Question Input from Others: Any survey worth sending out is worth getting input from others. Sometimes, this is a colleague with experience as a second opinion, but for larger surveys that are important for organizations, then getting input from a broad group of stakeholders is important. It is easy to discount the outcomes of things you have not participated in developing, so by bringing in stakeholders, you are getting their buy-in also.
Test Your Survey: Always test your survey as a participant prior to sending it out to your survey participants. This includes testing to respond as you think it would work.
Distribution: There are a lot of ways to distribute surveys, from paper surveys in person to emails, text, etc. The key is, again, understanding your survey audience and where they would feel most comfortable getting survey requests. If you regularly communicate with your customers via email, then sending a survey via email with the email providing context makes sense. As part of the distribution, be very upfront on time effort and the ‘why’ behind the survey frame in a way relatable to your targeted respondents.
Follow-Up - Don’t be Annoying: Not everyone wants to respond to your survey. It is best to try between 2-4 times and then stop. Further, if someone asks you to stop after the first time, then you should respect that. As context, B2B surveys where there is not a prior relationship will often be under 5% compared to the typical response rate for surveys, which is roughly 30%, but can vary drastically based on whether B2B to B2C and if there is a prior relationship with the person being surveyed.
Don’t Over Survey: Another way to be annoying is to over-survey your customers. Understand the relationship you have with your customers. If you survey your customer after every customer service call, then that is annoying for most. This doesn’t mean that you don’t provide an option for a person to give feedback if they have it. There are some customer relationships that people have in the B2B or B2C space, though frequent input requests can be valued and strengthen the relationship. In these cases, it is especially important to demonstrate action based on survey responses. This should be done even if anonymized by sending to respondents and non-respondents, so that respondents can understand their insights might have spurred action and non-respondents will be more likely to participate in the future.
Provide Participants a Bonus? Sometimes, it makes sense to use a bonus to thank or entice a potential survey respondent. You must be careful that the primary reason for taking the survey can’t be to receive a bonus, or else your data becomes less meaningful. Further, if you start using a bonus, then you want to consistently use a bonus in similar surveys. One bonus approach that often will help still be enticing but mitigate personal pecuniary bias is donating a certain amount to a charity for each respondent. You may even have several charities highlighted, and the respondent gets to choose which of the charities receives the bonus from the respondent’s participation.
Surveys are a valuable tool that helps you to understand your target audience cost-effectively. Done well surveys will be one of your most valuable research tools. Follow these steps above on designing and executing your survey, and you will be on the right path to a more successful survey.