How to Analyze Open-Ended Questions in Online Surveys

Picture two questions from the same customer survey. The first asks “Would you recommend us to a friend?” and only allows Yes or No. The second asks “What could we do better?” and gives people an empty text box.

The first question hands you a clean split, say 82% Yes and 18% No, which you can turn into a chart right away and break down further with a cross tabulation. The second one hands you 140 responses worded 140 different ways: one sentence here, a full paragraph there, and now and then just “it's fine.” Right away, the example makes clear why open-ended responses need a plan of their own. Counting alone won't get you far with them, but in return they're exactly where you find out why your numbers look the way they do.

The best way to analyze free-text answers, meaning the open-ended questions in your survey, is to decide first how closely you want to look. When only a few responses come in, you read every one of them, and when there are many, you group them into categories and count how often each category comes up. The most reliable approach blends the two, so you build categories and still read through all the original responses once.

We show you how to find the right depth for your survey and how to report results that other people can rely on. With empirio.ai, you read open-ended responses in the response list, see them as a word cloud and let the AI summarize them or sort them into themes.

📌 The key points at a glance

• Read a few responses in full; sort large batches into categories.
• A word cloud shows frequent words, but not what they mean.
• AI makes analyzing hundreds of open-ended responses far easier.
• empirio.ai offers you several tools to analyze open-ended responses.
• The empirio.ai AI summarizes responses and sorts them by theme.

Why open-ended questions call for a different kind of analysis

In an employee survey at a regional credit union, staff rate their workplace an average of 3.1 out of 5. That gives you a number, but no explanation. Add the follow-up “What's the main reason for your rating?” and suddenly you're reading lines like “I never hear back about my projects” or “The break room microwave has been broken since Labor Day.” Open-ended questions deliver exactly these reasons, examples and suggestions, because they don't offer any answers in advance and everyone writes in their own words.

How much the format shapes what people say shows up in a well-known experiment by the Pew Research Center. After the 2008 presidential election, voters were asked which one issue mattered most to their vote. Given a list, 58% picked the economy, yet only 35% named it when they had to answer in their own words, and 43% of the open-ended group gave an answer that wasn't on the list at all.

With empirio.ai, you add an open-ended question in the editor just like any other question. For a short, one-line answer, pick the Short text question type, and for a multi-line text box, pick Long text. You can also give a multiple choice question an “Other” field, which turns it into a semi-open question whose typed entries land in your analysis later, too.

As valuable as open-ended questions are, they come with two drawbacks you should know about before you even put your survey together. Both of them affect your analysis directly later on.

Drawback 1: Too many open-ended questions cost you responses

Someone who sees five empty text boxes stacked on one page may still write two sentences in the first. By the third it's just “ok,” the fourth stays blank and before the fifth the browser tab gets closed. Typing simply takes more effort than clicking, which is why people drop out more often when a survey has lots of text boxes. In the GESIS Survey Guidelines on open-ended questions, Cornelia Züll notes that open-ended questions elicit more “don't know” responses or refusals than closed questions do, and she recommends using them sparingly. What that means for you: ask an open-ended question only where you truly need a reason, for example right after an important rating question.

Drawback 2: Unhappy respondents write more than happy ones

The same guideline points to a second effect you'll run into during analysis, namely that dissatisfied respondents are more likely to answer and tend to write longer answers. A diner who waited 40 minutes for a table writes three paragraphs. A diner who had a great evening writes “all good” or nothing at all. Your open-ended responses therefore don't automatically reflect how everyone feels, and often they mostly capture the people who have something to get off their chest. What that means for you: when you analyze, count how many people bring up a topic instead of letting the length of individual responses steer you.

Analyzing open-ended questions: how closely do you want to look?

Megan manages a team of 50 at a nonprofit in Columbus and asked them a single open-ended question. She ends up with 50 responses she can read calmly in one afternoon. Tyler runs customer experience at a home services company, sent a customer survey with three open-ended questions to 800 customers and is now staring at up to 2,400 responses. Both of them want to know the same thing, namely what people really think, but they need completely different ways to get there.

How you analyze open-ended responses mostly comes down to three questions:

• How many responses are there? Multiply the number of open-ended questions by the number of participants. With 50 participants and three open-ended questions, you'll get no more than 150 responses, and with 800 participants it's already up to 2,400.
• Do you need to know every response, or is an overview enough? In a small employee survey, you may want to follow up on each piece of feedback, so you simply read everything, either right on empirio.ai or after exporting to a spreadsheet. If what you need is a sentence like “Most people praise our company culture,” you group the responses into categories and count them.
• Why did you ask? Ideas for the next company cookout call for a different analysis than the reasoning in your master's thesis. Academic work needs a documented coding scheme, while team feedback is often well served by a list you read carefully.

Methods for analyzing open-ended responses

Picking a tool gets easier once you know the methods underneath. Open-ended responses can be analyzed in five core ways, and they combine well:

1. Reading everything: Every response gets read, and anything notable goes into your notes. Precision doesn't get any higher, though it only scales to modest volumes.
2. Coding: You assign each response to a category such as “price,” “service” or “delivery” and at the end count how often each category appears. If you define the categories ahead of time, that's deductive coding. If you develop them from the responses as you read, it's inductive coding. Coding is the classic method of qualitative research.
3. Counting words: Single terms get tallied rather than whole statements. A word cloud or a list of top words quickly shows you which topics come up, but not how they're meant.
4. Gauging sentiment: Each response is tagged positive, neutral or negative, which researchers call sentiment analysis.
5. Summarizing: You condense many responses into a few key statements. It used to be done entirely by hand, and today AI often takes it over while you check its output against the originals.

AI can help with coding nowadays, too. The method stays the same, only the sorting goes a lot faster.

Which method fits your situation

The table shows which method suits you, and the right-hand column shows how to put it into practice with empirio.ai. Treat it as a rough guide rather than a set of hard rules:

Why you should still read everything once

Suppose your categories show “Price: 40 mentions.” Only when you read the responses do you notice that half of them praise the price and the other half complain about it. Categories smooth over differences like that, while the original responses keep them visible. So build categories or let empirio.ai suggest some, but also scroll through every response once, as long as the volume allows. You'll get the numbers for your report and still keep sight of what people actually wrote.

Analyzing open-ended questions with empirio.ai: every feature at a glance

For Megan's 50 responses, the response list is often all she needs, while Tyler's 2,400 responses are where AI pays off. So empirio.ai gives you several instruments for open-ended responses, whichever way you prefer to work: go through every response yourself or hand part of the job to tools and AI. Everything sits in the Results area of your survey:

1. Response list: read and filter every open-ended response
2. Export: analyze the raw data in Excel or as a CSV
3. Word cloud in the charts
4. Text Analysis: top words, AI Analysis and every response to a question
5. Text Labelling: sort responses into themes automatically
6. AI Assistant: ask questions about your whole survey

Response list, export and word cloud need no AI, whereas Text Analysis, Text Labelling and AI Assistant lean on it. Plans compared:

Below, we walk you through each one, with everything you need to get started.

1. Response list: read and filter every open-ended response

The response list is the simplest way to get to your open-ended responses. You'll find it under Results in the Responses tab, and it comes with all of its filters on every plan, the free one included.

Each participation shows up as a row and each question as a column, so your open-ended responses sit right among all the other answers. Long texts expand with Show more. At the top, you narrow down the time frame with From and To and choose between All participations and Completed only. If you only want to know what the unhappy customers wrote, use Filter by answer to show everyone who checked “Dissatisfied” on a multiple choice question, and then you read only their open-ended responses.

For Megan's 50 responses, the response list is plenty. With Tyler's 2,400, though, scrolling tends to leave you remembering the three loudest responses rather than the ten most common ones.

A good fit when …

• you only have a handful of participants.
• you want to see each response alongside the other answers from the same person.

Less of a fit when …

• hundreds of responses have come in.
• you need numbers per topic for a report.

2. Export: analyze the raw data in Excel or as a CSV

If you're writing a thesis or a capstone project, a quick look inside the app won't cut it, because your advisor will want to trace how you categorized every single response. The export exists for exactly that. You start it under Results in the Responses tab with the Export button, available from the Lite plan at €19 a month.

Your download arrives as an Excel or CSV file laid out with a row per participation and a column per question. Responses from an “Other” field get a column of their own. In Excel, add a column for your category next to the open-ended column, fill it in response by response and tally everything at the end with a pivot table. Categorizing itself works the same way as when you analyze a questionnaire: read first, then build categories, then count.

Thesis writers get the cleanest, most traceable route this way. For a quick sense of what folks are writing about, though, typing in categories by hand is a slog.

A good fit when …

• you want to do the coding yourself, for example for a thesis.
• you plan to keep working with the responses in MAXQDA or SPSS.

Less of a fit when …

• you just want to know quickly what people are writing about.

3. Word cloud in the charts

In the word cloud from a boutique hotel's guest survey, the word “breakfast” stands out in big letters. You instantly know what lots of guests are writing about. Whether they loved the breakfast or hated it, however, the word cloud can't tell you.

No upgrade needed here, since every plan includes the word cloud. Open Results and switch to Charts, and next to the bar and pie charts of your remaining questions, each open-ended question gets a cloud of its own. Frequent words grow bigger, rare ones shrink.

Treat that cloud as an early signal of which topics surface in your responses at all. Whether breakfast earned praise or complaints, and why, only becomes clear when you dig into the responses themselves, which is exactly what the response list above and the Text Analysis coming up next are for.

A good fit when …

• a quick first impression is all you're after.
• your report could use an eye-catching visual.

Less of a fit when …

• you need solid statements about praise or criticism.

4. Text Analysis: top words, AI Analysis and every response

When the word cloud feels too coarse, Text Analysis is your next step, an empirio.ai feature designed specifically for open-ended responses. You'll find it as a tile under Results in the Analysis area, right next to Cross tabulation and Text Labelling. Trying it is free, and ongoing use starts with Plus at €39 a month.

Once you click the tile, choose your open-ended question via Select question, then move between three tabs: Word Cloud, AI Analysis and Responses. Here's what each of them does.

4.1 Word cloud with top words

Rather than merely spotting a big “breakfast,” you get its exact count under Top words, for example 37 mentions in 120 responses. On top of that, the Response length distribution shows how many responses are short (under 50 characters), medium (50 to 200 characters) or long (over 200 characters). When many responses run long, a topic usually matters to people.

4.2 AI Analysis: summarize responses in one click

Often there just isn't enough time. You have 300 responses to “What could we do better?” and the meeting where you're supposed to present the key points starts in an hour. Reading everything isn't going to happen. For moments like that, the AI Analysis tab summarizes every response to an open-ended question for you.

First, choose a focus: Identify main themes, Problems & criticism, Praise & positives, Sentiment analysis, Extract suggestions, Action recommendations, Notable quotes or Surprise me. After you click Run AI Analysis, you get a Summary, matching Keywords and, depending on the focus, Positive Statements or Critical Statements along with Recommendations.

Keep an eye on the line above the result, such as “3 of 4 filtered responses analyzed,” because it tells you what the summary is based on. The filters for time frame and participations apply here, too. With Re-run Analysis, you start over with a different focus. Since the AI does its own weighting when it summarizes, read a few original responses for each theme before you present it in the meeting.

4.3 Read every response to one question in peace

In the Responses tab, you see only the responses to the selected open-ended question, listed one below the other, each with a timestamp. Unlike the big response list, there are no other columns to distract you here. After an AI Analysis, this tab is the best place to double-check the result.

A good fit when …

• you need exact counts for individual words.
• you have lots of responses and little time and want to know which themes dominate.
• you want to read all the feedback on one question in peace.

Less of a fit when …

• the responses consist almost entirely of single words.
• the AI result is supposed to feed into a decision unchecked.

5. Text Labelling: sort responses into themes automatically

AI Analysis tells you what people are writing about. Text Labelling tells you how many of them write about each thing. It sorts your open-ended responses into theme groups on its own, with a count per theme. You'll find it under Results in the Analysis area, where you can test it for free and keep using it from the Plus plan at €39 a month.

Open the Text Labelling tile, choose your open-ended question and click Generate labels. empirio.ai then groups the responses by similarity in content, gives each group a name and assigns every response to a label. You set the maximum number of labels with Max. Labels. Afterward, the overview shows the Label distribution, meaning how many responses fall under each theme.

The labels are a suggestion, not a final result. You can rename them, add descriptions or delete them, create your own with New label and move responses to a different label. The Low confidence only filter shows you the responses that only loosely fit their label, so you know which ones to check by hand. When you're done, export the assignment as a table of responses and labels.

Clustering requires at least 20 text responses. The payoff is a line ready for your report, such as “A third of all complaints are about how hard it is to reach us.” If your academic work has to apply a fixed coding scheme, though, you're better off coding by hand.

A good fit when …

• you need a countable result for each theme.
• well over 20 responses have come in.

Less of a fit when …

• there are fewer than 20 text responses.
• a fixed coding scheme is required.

6. AI Assistant: ask questions about your whole survey

Up to this point, each feature has zoomed in on one open-ended question, whereas the AI Assistant takes in your entire survey at once. Think of it as a chat built into your results, reachable under Results in the AI Assistant tab. Testing it costs nothing, and ongoing use comes with Plus.

Questions go in plain everyday language, like “What trends do you see in the responses?” or “What improvements would you suggest?” Because the assistant knows the whole survey, it connects the open-ended responses with your other questions. Ask what the detractors in your Net Promoter Score give as their reason, for example, and it summarizes exactly those responses and suggests concrete next steps.

Once again, watch the basis: the assistant mentions how many completed participations went into its answer and whether filters are switched on. When a report needs exact figures, verify them in Charts or Text Labelling instead.

A good fit when …

• your question goes beyond a single open-ended question.
• you want to turn the results into concrete actions.

Less of a fit when …

• you need exact frequencies for a report.

How to combine the features: a 5-step workflow

Tyler's customer survey shows what this looks like in practice, with 120 responses to the question “What could we do better?” Five steps take him from a rough overview to a number he can back up. The same workflow fits just as well for feedback after a workshop or for Megan's team survey.

1. Skim. In Text Analysis, scroll through every response once in the Responses tab without sorting anything yet. With very large volumes, a sample of a few dozen is enough.
2. Check the top words. The word cloud with its top words shows you which topics jump out. For Tyler, “hold” and “phone” sit right at the top.
3. Find the themes. Run the AI Analysis with the Identify main themes focus or generate labels in Text Labelling. With only a few responses, just create the categories yourself.
4. Cross-check. For each theme, reread a few original responses and review the low-confidence assignments in Text Labelling. If a label's name doesn't match what's inside, rename it.
5. Report numbers and quotes. For each theme, give the number of responses it covers. Add one or two typical quotes alongside.

In the end, Tyler's report contains a sentence like “Of 120 responses to the question about improvements, 38 are about being put on hold on the phone.” The number makes the statement solid, and the quote next to it makes it tangible.

Read first, then sort, then count.

Common mistakes when analyzing open-ended questions

Sorting rarely trips people up, but reporting often does. Five slipups cross our desk again and again.

Passing off the word cloud as the result

The word “service” fills the whole slide, and everyone in the room nods. Whether people praised the service or criticized it, nobody knows at that moment. A word cloud shows you which topics come up, but not yet what people actually wrote about them. Use it as a starting point and as a visual for your report, but not as evidence for a claim.

Taking the AI summary at face value

“Many patients would like the clinic to open on Saturdays,” the AI writes. When you check, it turns out that exactly two of 150 people mentioned it. An AI summary sounds fluent and convincing even when a topic barely comes up. So check every statement headed for your report against the original responses and note how many responses it rests on.

Mistaking the loudest voices for the majority

Two angry paragraphs stick in your head, while twenty short “works for me” replies don't. People who are unhappy often write more, and more sharply. So count instead of going by your gut, and compare the open-ended responses with the results of your closed questions as well.

Breaking anonymity with quotes

“Ever since we moved to Building C, nothing on our team works anymore.” In a department of five, everyone knows instantly who wrote that. Open-ended responses often reveal more than you'd think, through someone's writing style, a project they mention or their own department. If you promised an anonymous survey, only pass quotes along shortened or paraphrased, and leave out names and details.

Passing on only the summary

HR analyzes the employee survey, and in the end the leadership team gets a slide reading “Communication: 23 mentions.” What sits behind that number, say “I hear about big decisions through the grapevine first,” never makes it into the room. With every step from individual response to category to presentation, meaning and substance get lost. Whoever makes the final call often only knows the condensed version.

Our recommendation: read as many open-ended responses as you can yourself, even if you didn't do the analysis. When you pass results on, attach a few typical original responses to each category, anonymized and shortened. Leadership then also gets a feel for what people really think.

Checklist before you report

• Every category and every label has a clear name.
• Each theme shows the number of responses behind it.
• The quotes are typical, not just especially dramatic.
• AI results have been checked against original responses.
• The filters are the same for every analysis and written down.
• No quote lets anyone identify an individual person.
• Whoever receives the results also sees some original responses.

The bottom line

The yes-or-no question tells you how many people are satisfied. The open-ended question tells you why. Your approach to those responses hinges on their volume, the depth you're after and your reason for asking. Our advice: personally read as many as you can, lean on word cloud, AI Analysis and Text Labelling once volumes grow, and close out with counts plus a few typical quotes.

Where to go next

• Looking to word open-ended questions well? Open-ended questions: definition and examples
• Ready to compare groups across your multiple choice questions? How to create a cross tabulation
• Curious what your customers truly think? The customer survey guide

Analyze open-ended responses without reading each one by hand?

With empirio.ai, the AI summarizes your open-ended responses and sorts them into themes. Try it free, with 100% EU hosting, AI included.

Create your free survey