In addition to administering our own surveys, the Office of Institutional Research, Analysis, and Planning supports the Bates community in its efforts to engage in survey research. This support includes consultation about survey design, distribution, and analysis of results. Additionally, the office provides guidance about how to use Qualtrics, the web survey tool that Bates has licensed and made available to all faculty, staff, and students. For instructions about setting up a Qualtrics account, please visit this page.

In order to make sure that our office is able to provide the support that you need, it is strongly recommended sufficient time to plan and to consult the guidelines on this site is incorporated.

A General Guide to Administering Surveys

1. Formulate research questions

Formulating your research questions is a critical step in this process as these questions will guide your research from how you design your study to how you analyze and report your results. Your research questions should be clear and purposeful about what your research is trying to accomplish. Additionally, the rationale for your study should go beyond “I’m interested to know;” there should be practical or theoretical implications for your study. In order to develop strong research questions, it is important to review the literature relevant to your topic. It is likely that researchers have already addressed the topic you wish to study, and their work can inform the theoretical framing and research design that you use. For those with little experience conducting a literature review, consider consulting the Research Services staff at Ladd Library.

2. Consider whether you really need to conduct a survey

Having formulated your research questions, you may find that administering a survey is not appropriate for your study. For example, depending on your research questions, it may make more sense to use constructivist approaches like interviews or focus groups.

If you determine that survey data are the appropriate data source for your study, it is possible that the data have already been collected. For decades, Bates College has engaged in survey research with students, alumni, faculty, and staff. The Survey page on the IRAP website provides information about the surveys that Bates College regularly administers.

3. Establish an appropriate timeline

While setting up a web survey and distributing it via email can be a quick process, the preparations in advance of survey administration can be fairly time consuming. This process can take several months and should involve careful consideration of your target distribution date and the amount of time it will take to accomplish all that needs to be done prior to this date. It is important to give yourself ample time to plan, develop, and test your survey.

Avoiding conflicts with other survey activity

When choosing a target distribution date, it is also important to consider the other things that are may be happening at the time. For example, May is a busy time due to the administration of the Senior Survey and graduation activities, while November and March are particularly busy for student-administered surveys. Other considerations are times that are difficult to attract responses, such as vacations and midterm and final exam periods.

4. Identify your target population

Your choice of the target population of your survey will largely be determined by your research questions. For example, if your research questions address the experience of first-year students, the target population would probably be limited to first-year students (or possibly faculty and staff who work with first-year students). Once you have identified the appropriate target population, another thing to consider is whether it makes more sense to survey the entire population (a census) or survey a sample of your target population. Determining an appropriate sample for your survey is not a service that IRAP provides, though the office can direct you to books and online resources about sampling.

Important note: The best way to reach your population is frequently through the moderated listservs on the Bates campus. For either a census or a sample, you may want the contact information of your target population. This information will only be provided by IRAP or the appropriate administrative office and this determination will be made on a case by case basis. If you do need the contact information of your target population, please complete the Data Request form on our IRAP website. This review process can take several weeks, so please plan accordingly.

5. Design the survey instrument

There is voluminous literature on the art and science of designing a survey, so summarizing this body of work in this space would not do it justice. The following are tips and resources you may want to consider using as you design your survey.

Here are a few general survey design tips:

• Keep in mind that your survey items should align with your research questions.
• Always number your items and keep them in a bold font.
• Avoid leading questions. Keep your questions neutral and make sure they don’t lead respondents to a particular answer.
• Avoid double‐barreled questions. Example: “How satisfied were you with the sound and visuals of the presentation?” It is unclear whether you want them to evaluate the sound or the visuals. Split it into two questions.
• Be mindful of the length of your survey. Shorter surveys tend to generate stronger response rates.
• Limit the number of open-ended questions and place them at the end of the survey. Open-ended questions can be disruptive to the flow of your survey.
• Have someone review your survey and read through the questions. What makes sense to you may not make sense to the audience.
• Always test your survey. Fill it out yourself and look at the results to make sure that it is error‐free.

Helpful resources:

• An excellent place to start is the Qualtrics website. Qualtrics is not only a highly intuitive web survey tool, it also provides excellent support, including these basic survey development overviews.

6. IMPORTANT: Include informed consent and evaluate whether you need IRB review

Some surveys will need to be reviewed by the Institutional Review Board (IRB). The IRB, which oversees research that uses human participants as subjects, aims to protect research participants by ensuring that they are treated safely and respectfully. Federal law requires that all federally funded research involving human participants receives IRB approval. For more information about IRB review and whether it applies to your survey, please visit the Bates IRB website.

Whether your survey requires IRB review or not, ALL surveys at Bates are expected to include informed consent language. Prior to the start of your survey (e.g., in a signed consent form, in the body of an email invitation to participate, at the start of the survey, all of the above), you will need to inform your potential participants about the purpose of the survey and how their data will be used. Examples of the information you provide include how the data will be stored, who will have access, whether you can provide anonymity or confidentiality, and how and for what purpose will the data be reported. Templates and forms for consent can be found at the Bates IRB webpage. If you have any questions about whether you are providing an acceptable level of informed consent, please contact Tom McGuinness (tmcguinn@bates.edu) in the Office of Institutional Research, Analysis, and Planning or a current Bates IRB representative.

7. Consider offering incentives to respondents

If you are concerned about your response rate, you may want to consider offering incentives to your respondents. A common approach is to raffle off gift cards, with the eligible population being those who complete the survey. However, do not expect that offering incentives will guarantee a high response rate for your survey. Research has shown that while incentives have a positive effect on response rates, this effect is quite small (Porter & Whitcomb, 2003).

Keep in mind that compensating individuals for participating in your study, through either payment or a raffle, may have tax consequences and be unavailable to some International students. For additional information, consult IRS Publication 15-B on de-minimis (minimal) payments or email Scot Flink (sflink@bates.edu), Director of Payroll.

Something to keep in mind if you are considering offering incentives is that you will need to gather the contact information of survey respondents. If you intend to make your survey anonymous, it is recommended that you collect the contact information in a separate survey. Qualtrics has a straightforward way of accomplishing this.

• Set up a separate survey in Qualtrics to collect the contact information of those who wish to be entered into your raffle. A title and a text entry form with name and email address is probably all that you need.
• Select “Launch Survey” and then “Activate your survey to collect responses” and copy the anonymous link.
• Go to the edit screen of your original survey and select “Survey Options.” In the “Survey Termination” section of the “Survey Options” menu, there is an option to “Redirect to a URL.” When you select this option, paste in the anonymous URL from the new survey you have set up.
• Now, once your respondents complete and submit the survey, they will be automatically directed to the form to collect their personal information for the raffle.

8. Administer survey and collect data

Crafting an email inviting your target population to respond

When distributing your survey via email, consider how the recipient will react to receiving it. In order to encourage the recipient to open the email and click on the link to respond to your survey, it is important to employ a subject and email language that explains the value of responding to the survey and minimizes the burden of the respondent. Pieces of information that demonstrate the value of responding include the importance of the survey, how the data will be used, and incentives the respondent might receive as part of his or her participation. Minimizing the burden to the respondent includes providing an accurate estimate of the time required to complete the survey, clearly explaining the purpose of the survey, and using discretion when considering approaching your target population again with a reminder email.

Distributing your invitation email

In Qualtrics, there are two approaches you can use to administer your survey and collect data. The first approach is to use an anonymous link and distribute the survey via email to your population. The second is to distribute the survey through the Qualtrics system by setting up a contact list (within “Directories” in Qualtrics). Each of these approaches has its own advantages and disadvantages.

The advantage to the anonymous link approach is that you can guarantee anonymity to your survey population. It is additionally a very straightforward and easy approach. All you have to do is activate the survey, copy the anonymous link, paste it into an email, and send the email. The potential drawback to this approach is that it is impossible to identify your respondents and, when sending reminders, you will not be able to remove any of the individuals in your population who have already responded to the survey.

Using Qualtrics to distribute your survey is a bit more complicated, but there are some real advantages to using the Qualtrics distribution system. Using this system, you pre-load in the information about your population into a Directory list in Qualtrics. The system can distribute a custom email to each person in your population. For example, by including a name and major in your contact list, you can address each individual by name and include his or her major in the text of the email. These customized emails provide a personal touch that may encourage the recipient to respond to the survey. Another advantage to this approach is that it might allow you to shorten the length of your survey. If you have relevant background information about your population (e.g., class year, field of study) you can link the survey response to these data rather than including questions about these characteristics on the survey. Finally, when sending reminders from the Qualtrics system, it automatically removes individuals who have already responded from the reminder distribution, so you do not have to contact them unnecessarily. You can also take advantage of this feature and provide anonymity to your respondents.

The potential disadvantage of this approach is that it requires you to have, at a minimum, the email addresses for all of the individuals in your population. If you do not have this information, your best option is likely to be distributing an anonymous link via the Bates listservs or Bates Today.

Monitoring your survey responses

Once your survey is distributed, make sure to monitor the responses as they come in. The main Qualtrics screen (“Projects”) shows the number of respondents who have started your survey. The “Data & Analysis” tab in Qualtrics provides detail about both the individual responses and the “Reports” tab provides information about aggregate data. You will want to look at the more detailed view in order to determine the number of people who have completed your survey. Most responses come within the first 48 hours of distribution. After your initial distribution, you may want to send a reminder email to your population. Once your responses stop coming in, you will want to close your survey. This can be accomplished by selecting “Close” in your survey menu on the right side of the screen when you are on the “Projects” screen.

What is a good response rate?

Many people wonder what constitutes a good response rate. Unfortunately, there is not a simple answer. The extent to which a response rate is good is dependent upon a variety of factors, including the size of the population and whether your respondents are representative of the population you have surveyed. For more complex surveys and those based on samples, there are statistical procedures and response rate calculators that you can use to make judgments about the number of responses you may need. For simpler surveys, the key question you will want to ask is whether the results are believable and generalizable to your target population, based on the number of responses you have.

9. Analyze and report results

By the time you distribute your survey, you should have a plan for how you will analyze and report the results of your survey. You will want to have a good sense of your intended audience, the data you would like to share with this audience, and the format in which you will disseminate the results. These decisions may be a factor in whether you need IRB review. For more information about this, please consult the Bates IRB website.

For many users, the reports that Qualtrics generates may be adequate for descriptive analyses. Reports can be viewed by selecting the View Reports button on the Projects screen or, when you are working on your project, by selecting Reports at the top of the screen. Each survey has a report, called Default Report, that displays the results of the survey in real time. It is easy to manipulate these results and export the report in a variety of file formats (i.e., Word, Excel, PowerPoint, PDF). For more information about the Qualtrics reporting feature, visit this link.

If you need a more robust analysis, it is easy to export your raw data to Excel or SPSS by selecting the Data & Analysis tab and clicking on the Export & Import button.

At any time, feel free to connect with the Bates IRAP office for further guidance, questions or ideas.