Markets & Investing
Polling Problems: 5 Types of Bad Survey Questions
02 Nov 2018
Writing a survey is simple, right? Not exactly. The goal of any effective survey is to collect accurate responses useful in future decision-making. For instance, companies may need to collect employee survey responses ahead of policy changes or new initiatives. Educators frequently use surveys to take the temperature on student attitude or retention. In use cases like these, it’s highly important to make sure you’re asking fair and comprehensible questions up front—otherwise the data you collect will be tainted.
Want to avoid common polling problems? Start by avoiding these five types of bad survey questions.
Double-Barreled Questions
In your eagerness to gather information, it’s easy to inadvertently ask more than one thing in a single question, often without realizing you’ve done so. Questions containing multiple inquiries are known as “double-barred questions” within the survey community.
Here’s an example of a double-barreled statement from the University of Colorado Denver: “I feel welcomed by staff and other youth at the center.” As far as true or false statements go, this one actually has two separate components. Respondents may feel welcomed by staff and other youth. They may feel welcomed by staff but not other youth. They may feel welcomed by other youth but not staff. Or they may feel welcomed by neither. This statement is not conducive to isolating respondents’ actual experience.
Break double-barreled questions into multiple distinct questions so you’re only ever asking one thing.
Leading Questions
Survey results are only as useful as they are honest. Asking leading questions destroys the impartiality of a good survey because it skews respondents toward answering in a certain way. Whether intentional or not, leading questions end up sullying the sanctity of survey results.
Here are a few guidelines from Poll Everywhere for asking bias-free questions:
- Ask open-ended questions instead of forcing participants to commit to absolute “yes-or-no” questions.
- Avoid pigeonholing people into selecting one of two answers with strict “or” questions.
- Do not insert any assumptions into your questions.
- Ask just one questions at a time, so responses correlate clearly.
Convoluted Questions
As the preparer of a survey, you have more time to think through questions and digest them. Respondents are reading them for the very first time, either on paper or on a screen. The last thing you want is for your survey to induce the pressure of a standardized test reading comprehension section in which people are trying to read paragraphs very quickly, understand them and respond within the time limit.
There are a number of sub-types of convoluted questions, including trick, confusing and long/disjointed queries. The best rule of thumb here is to always Keep It Short and Simple, or KISS.
Vague Questions
The opposite of a convoluted question is a vague one. Make sure you’re providing people with the information they need to answer with confidence. You want to strike the perfect balance directly between providing too little information and too much.
Non-Exclusive Multiple-Choice Questions
Just as important as the actual questions are the response options you’re providing to participants. Multiple choice surveys are only effective if they encompass every possible answer a respondent could possibly give.
One problematic survey structure is providing a finite number of answers with no “other” or “does not apply” option. Another is providing a set range of answers without making space for the outliers on either side. The data you capture will not be valid if people are forced to choose an answer that doesn’t apply to them.
Surveys are powerful tools in the workplace and classroom, but only if you avoid bad survey questions that end up skewing the data and frustrating participants.
