Qualitative vs. quantitative survey: how to classify yours

The methods section of a thesis has a way of forcing the question: is your data collection qualitative or quantitative? Many students build an online questionnaire, add two open-ended boxes and would like to write “a bit of both”. Advisors and IRB reviewers rarely go along with it.

Whether a survey is qualitative or quantitative comes down to standardization, not to the number of open-ended questions: if every respondent gets the same questions in the same order with the same answer choices, the study is quantitative. It becomes qualitative only when wording, sequence and answers stay open and the individual case is interpreted. By the end of this article you can classify your own survey and defend the choice.

📌 Key takeaways

• Standardization decides the label, not the open-ended questions.
• Pew: 43 percent gave answers outside the closed-ended list.
• Opt-in online samples showed 5.8 points average error, panels 2.6.
• Guest, Bunce and Johnson reached saturation within twelve interviews.
• AAPOR: opt-in polls have no valid margin of sampling error.

Qualitative vs. quantitative surveys: the difference

A quantitative survey measures how often something occurs and tests expectations set before data collection begins. A qualitative study reconstructs how individual people make sense of a situation, working with few cases from which new expectations emerge.

The split is less about worldview than about procedure. Everything that follows from it, from sample size to analysis, is downstream of one decision: whether the instrument is fixed before you go into the field or not.

Neither approach is better research, they answer different questions. If you want to know how common an attitude is on campus, take the quantitative route. If you want to understand where it comes from, take the qualitative one. For the terms themselves, see our article on qualitative and quantitative research.

How to spot the degree of standardization

Three checks give you the answer: are the questions fixed in advance, is their order fixed, and are the answer choices fixed? Three yeses mean standardized and therefore quantitative. Three noes mean unstructured and therefore qualitative.

Between those poles sits a third level, and it shows up in more theses than either extreme. All three in turn:

• Structured survey: everyone gets the same questionnaire with set answer choices. Conditions stay constant, which is what makes statistical analysis possible at all.
• Semi-structured interview: an interview guide fixes the topics and their order while answers stay open. The normal case for expert interviews.
• Unstructured interview: only the topic is fixed. The interviewer decides wording and sequence during the conversation. Typical of narrative interviews.

The word interview is the trap, because it stretches across the whole scale. Pew Research Center reports fully standardized data collection as “interviews”, while the same field uses “in-depth interview” for the open end. Never let interview stand alone in a methods section, always add structured, semi-structured, in-depth or cognitive.

💡 Tip

Write the degree of standardization into your methods section instead of the label. “Standardized online survey with closed-ended items and two open-ended follow-ups” survives any defense question. “Qualitative-quantitative hybrid” invites one.

Do open-ended questions make a survey qualitative?

Open-ended questions do not make a standardized questionnaire qualitative. What matters is not the answer format but whether every respondent gets the same questions in the same order. If the questionnaire stays standardized, the open-ended answers are part of a quantitative dataset.

Open-ended items do change what you learn, and Pew Research Center has measured how much. In its guide to writing survey questions, Pew reports that when the economy was explicitly offered as a response, 58 percent of respondents chose it, while only 35 percent volunteered it in the open-ended version. More striking: “fully 43% of those asked the open-ended version provided a response not listed in the closed-ended version of the question”.

That is an argument for using open-ended questions, not for calling your study qualitative. Pew itself describes the standard workflow: a pilot study with open-ended questions to discover which answers are most common, then closed-ended questions built from those results. Both stages stay quantitative. For when each format earns its place, see our comparison of open-ended and closed-ended questions.

⚠️ Careful

Calling a standardized survey a “qualitative study” because it contains a few open-ended questions is the easiest point for a committee to attack. Name the degree of standardization instead, and explain how you coded the open-ended answers.

Why your online sample probably is not representative

Anyone running a self-service online survey recruits an opt-in sample, and US research has quantified what that costs. Opt-in means respondents were not selected randomly from the population but came in through ads, reward sites or self-enrollment in a panel.

Pew Research Center compared both types directly in September 2023. Across 28 benchmark variables, online opt-in samples showed an average absolute error of 5.8 percentage points, about twice the 2.6 points of probability-based online panels. The study also found a likely cause: on the opt-in samples, an average of 8 percent of all adults, and 19 percent of Hispanic adults, answered yes to at least 10 of 16 yes/no questions, which Pew reads as a sign of “bogus respondents”.

The American Association for Public Opinion Research puts the consequence in one sentence: “Currently, it is impossible to develop statistically valid margins of sampling error from nonprobability surveys, such as opt-in, online polls” (AAPOR, 2012). So report your recruitment method and skip the margin of error. Our article on representative surveys explains what you can still claim.

How many participants does each approach need?

Qualitative studies work with few cases, quantitative surveys need enough cases for the analysis you have planned. The number follows the method, and that reasoning belongs in the methods section rather than a round figure chosen for comfort.

For the qualitative side there is a solid reference point. Guest, Bunce and Johnson analyzed sixty in-depth interviews with women in two West African countries in the journal Field Methods in 2006 and tracked when new themes stopped appearing. Saturation occurred within the first twelve interviews, and the basic elements of the metathemes were present after six. For a homogeneous group and a narrow topic, a two-digit number is defensible rather than a compromise.

On the quantitative side no equivalent rule of thumb exists, because the required sample size depends on the analysis and the precision you need. As the Pew comparison shows, how you recruit matters more than how many you reach: a larger opt-in sample does not become more accurate, it just becomes a larger opt-in sample. Say plainly whom your findings describe.

When it makes sense to combine both approaches

Qualitative and quantitative work combine well when each part has its own job and the two run in sequence. Putting them side by side inside one questionnaire almost always produces a standardized questionnaire with comment boxes, nothing more.

The usual order is qualitative first, and Pew describes exactly this pattern for its own instruments: new questions get tested ahead of time “through qualitative research methods such as focus groups, cognitive interviews, pretesting”, and only then go into the field. In a thesis the same logic works with six to ten open conversations followed by the standardized questionnaire that measures how widespread the themes are.

The reverse order works too: a survey throws up an unexpected result, and a handful of interviews explain why. Either way each part keeps its own sampling, its own data collection and its own analysis. For how analysis differs by data type, see our overview of evaluation methods in empirical research.

A comment box does not make a qualitative study. Two data collections with two analyses do.

Common mistakes when classifying a survey

Most classification errors do not come from ignorance. They come from writing the methods section after the survey has already gone out, so the label has to fit whatever happened. A lean standardized survey then turns into a “qualitative pilot study”.

Three patterns come up again and again. All three take minutes to avoid before you field the survey, and a second round of data collection to fix afterwards.

Mistake 1: confusing the answer format with the approach

Open text fields look to many students like proof of qualitative work. The approach depends on the procedure, not on the input box. A standardized questionnaire with three open-ended items is still a quantitative survey whose comments get coded. The consequence is concrete: the methods section names a method that never appears in the results.

Mistake 2: relabeling a low response rate as qualitative

When eighteen replies arrive instead of the two hundred you hoped for, “qualitative study” sounds like an elegant rescue. It is not one. A qualitative study differs by open data collection and interpretive analysis, not by having fewer cases. Naming the low response rate as a limitation is both more honest and academically stronger.

Mistake 3: reporting a margin of error you do not have

Statistics software will happily print a confidence interval for any dataset. AAPOR is explicit that opt-in online polls do not support a statistically valid margin of sampling error, so a number from a self-selected sample describes nothing beyond the people who answered. Report the recruitment method instead. For the rest of the build, see our tips for online surveys.

Conclusion

It comes down to one question: before data collection began, was it fixed what would be asked, in which order and with which answer choices? If yes, your study is quantitative, and open-ended follow-ups do not change that. Anyone who genuinely wants to work qualitatively plans a separate data collection with its own analysis.

Where to go next

• Want the terms themselves? Qualitative and quantitative compared
• Planning a standardized survey? Quantitative surveys explained
• Planning open interviews? Qualitative surveys explained
• Running a survey for your thesis? Surveys for your thesis

Ready to run your survey?

With empirio.ai, an online survey tool from Germany, you build a standardized questionnaire in minutes and analyze the answers right away.