Center for Applied Research & Evaluation
Resources

How to Create a Good Questionnaire/Survey

Although questionnaires/surveys are one of the most common methods of gathering information for evaluation purposes, it can be tricky to create a good one. The following are some tips to help increase the likelihood that your surveys measure what you intend in a consistent manner across all respondents.

  • Know your purpose and stick to it! Identify one or two things you really want or need to know...and then make sure that ALL questions on the survey are connected to them. Just because something is interesting doesn’t mean it fits the purpose of your survey!

  • Group questions into sections that make sense (e.g., a section on knowledge, a section on attitudes, etc). Even though it may not be necessary to label sections, it can help respondents organize their thoughts and simplifies data entry.

  • Use clear questions. This is much easier said than done but very important. Make sure your questions have one meaning for all people and that they only ask about one thing at a time.

  • Watch out for "double-barreled" questions that ask about multiple issues e.g., "How often do you have episodes of anxiety or depression?" or "How would you rate the professionalism and concern shown to you by staff here?" These should both be at least two separate questions.

  • Watch out for questions that ask about issues that have multiple dimensions e.g., "What is your current capacity to meet your financial obligations?” In this question, "capacity" is something that involves multiple elements (i.e., physical, psychological, financial) as does "financial obligations" (i.e., the person may have many bills but have trouble paying just a few). You'll get a very general answer to this type of question that may not help you pinpoint the issue or where to make improvements.

  • Avoid vague questions that allow for multiple definitions e.g., "How often do you use drugs?" This may seem like a clear question but people will define "drugs" differently (i.e., illegal drugs, prescription or over-the-counter drugs, alcohol) so you won't really know about which type they're responding.

  • If necessary, you can provide a definition of the concept or issue with which people are unfamiliar. This is usually done in a narrative right before the questions. But the questions themselves should be clear.

  • Use "value-free" questions. A question like "How much do you like broccoli?" telegraphs the value of liking broccoli. A better way of asking this would be "How much do you like or dislike broccoli?" or even "How do you feel about broccoli?" which allow for both ends of the spectrum (i.e., hating and loving). In particular, avoid value-laden questions when asking about sensitive or controversial issues.

  • When asking questions that involve scaled (or Likert) responses, avoid using scales with more than 7 points. When a scale is larger than this (say, a 10-point scale), people can no longer make meaningful distinctions between what each point represents.

  • To help people make meaningful distinctions between scale choices, make sure all scales are grounded. This means that each number on the scale should have a description attached to it. Rather than ask people to rate their level of preparedness on a scale from 1 to 4 where 1=Not at All Prepared and 4=Very Prepared, ask them to rate their level of preparedness on a 4-point scale where 1=Not at All Prepared, 2=Somewhat Prepared, 3=Prepared, and 4=Very Prepared.

  • Pay attention to comprehension/reading level. Most surveys should be written at about 5th or 6th grade reading level.

  • You may need to read questions aloud for persons who have difficulty reading or comprehending – but only if this does not embarrass them or compromise confidentiality!

  • Your results can be skewed if you have to provide explanation of your questions. A well-written question at the appropriate reading level shouldn't require explanation.

  • Be aware of social desirability. Social desirability is related to the tendency of people to answer questions based on what they think others consider "appropriate" rather than how they truly think, feel, or act.

  • Asking a question like "how much do you approve or disapprove of hitting another person?" is not a good way to measure attitudes because the socially desirable response is clear...regardless of how a person really feels. A question such as "How much do you agree or disagree with the following statement: There are times when people deserve to be hit?" is a better way of assessing attitudes about violence.

  • Responding with "yes" or "no" or one-word answers should never be an option for open-ended questions (i.e., questions that aren’t multiple choice but require writing). This requires that the questions be carefully written. Rather than asking something like "are you interested in this topic", it is better to ask "what do you think might be interesting about this topic?" People can still provide one-word answers with the second option but it's less likely.

  • Avoid "why" questions as much as possible. "Why" puts people on the defensive. Using "what" is a better way to get the same information (e.g., "Why do you believe this?" vs. "What has influenced your belief in this?").

  • In order to assess whether you have a good survey, you can ask people who will not be taking the survey to review it and tell you what each question makes them think or feel. This is a form of "cognitive interviewing" in which you're not asking people to answer the questions but to tell you what they understand them to mean, what the questions make them think about, and any issues with comprehension or values.


ABOVE ALL ELSE

Make sure that you protect the confidentiality and well-being of those you survey. An ethical survey is one in which people can answer honestly with no concern regarding recrimination, exposure, or psychological harm. In particular, be very thoughtful about how your questions can make people feel. You do not want a respondent to end your survey feeling frustrated, ignorant, scared, or with re-ignited feelings of past trauma.

CARE staff have extensive experience and training in creating good surveys that ask the right questions to get you the information you need. Please contact Dr. Tara Gregory at 978-3714 or tara.gregory@wichita.edu to discuss how we can help you with your survey or questionnaire.