Key points
Decide the purpose and the classification criteria first, clean up the data, and then have AI summarize and classify the responses.
Compare the AI's summary against the original text and check that minority opinions have not been dropped.
How to use the classification results is decided by people, such as a board of officers or the local government.
Updated
Overview
The free-text fields in a survey are where people write the reasons and circumstances that multiple-choice questions cannot capture. On the other hand, as the number of responses grows, reading and classifying them one by one takes time, and the way they are grouped can vary depending on who reads them.
With AI, you can summarize a large number of free-text responses and sort them by theme in a short time. However, an AI summary is "a draft that has been tidied up to be easy to read." People need to go back to the original text and check that nothing not written in the original has crept in and that less common opinions have not been left out.
This page sets out the steps and points to watch when tallying and classifying free-text responses with AI, as a general description of common practice. It also introduces what Plural Reality's Baisoku Survey can do.
Steps
- Decide the purpose and the classification criteria. First, write down "what decision these results will be used for." For example, the criteria for classification change depending on whether you want to set the date of an event or think about the priorities of a policy. Deciding in advance whether to group by for/against, by type of problem, or by reason makes your instructions to the AI less likely to drift.
- Clean up the data. Put the responses into a table with one response per row. Set aside blank responses and responses such as "nothing in particular." Remove or mask statements that could identify individuals, such as names, addresses, and phone numbers, before passing the data to AI. If you collected responses on paper, enter them into the same table and handle them together.
- Summarize and classify with AI. Give the AI the criteria you decided on and have it classify the responses. Also have it produce a summary for each category and the points where opinions agree and where they diverge. Having it also list the "responses that did not fit any category" makes later checking easier.
- Go back to the original text and check. For each category, read a few original responses and see whether the AI's summary matches what they say. Checking is easier if you can trace, one by one, which responses each claim in the summary is based on. Also read the responses that did not fit any category and the categories with only a few responses in their original text.
- Let people decide. The AI's classification and summary are material for a decision. What to prioritize and which opinions to reflect in the next initiative are decided by people, such as a board of officers, a general meeting, or the local government. Recording the reasons for the decision also helps when explaining it to respondents.
| Aspect | What is easy to leave to AI | What people handle |
|---|---|---|
| Volume of reading | Drafting summaries and classifications of a large number of responses | Reading through the original text for each category |
| How to group | Sorting according to the criteria you decided on | Deciding the criteria themselves |
| Minority opinions | Pulling out responses that do not fit any category | Judging whether an opinion should be kept |
| Conclusions | Organizing the points where opinions agree and diverge | Deciding what to do and explaining it |
Points to watch
Check AI summaries against the original text
AI summaries may use wording that is not in the original text or phrasing that is stronger than the original. Match the sentences of the summary to the original text and check for overstatement or misreading. When quoting in a report, quoting the original text rather than the summary is less likely to cause misunderstanding.
Do not drop minority opinions
Because a summary focuses on "what many people wrote," opinions with few responses tend to be left out. Yet a small number of responses may describe issues that many people have not yet noticed. Do not decide weight by count alone; also look at responses that fall outside the categories.
Decide in advance how far personal information may be handled
Free-text responses sometimes mention the respondent, their family, or their neighbors. Decide which items to mask before passing the data to AI, and when publishing the results, check that no statements that could identify individuals remain.
What Baisoku Survey can do
Baisoku Survey is a survey tool that digs deeper into respondents' thinking through a dialogue with AI. It asks "why?" in response to each respondent's answer, so free-text responses come in with the reasons already written.
The collected answers are counted automatically, and a written summary (the overall report) is produced. The report shows "where opinions agree," "where they diverge," and "what to do next." Each sentence of the summary is followed by a code that leads back to the original answer, and clicking it shows that answer. On the Pro plan and above, you can also write the angle you want to look at and regenerate the summary. The summary does not steer toward any particular conclusion; the final decision is made by people.
In Ota City, Gunma Prefecture's program review (10 programs in total), Baisoku Survey was used to collect the views of observers, and a report was produced that also broke down the comments on the evaluation sheets into issues and reasons. For each responsible department, it was set up so that the breakdown of verdicts, how the conclusions were divided, and the original text could be checked.
Results can be saved as print/PDF or CSV (a format that opens in spreadsheet software), and on the Pro plan and above, automatic export to Google Sheets is also available.
Related case studies
Using AI to Overcome the Challenge of Translating Residents’ Voices into Policy
A case where Plural Reality's AI tools were used to "translate" residents' diverse voices into policy.
Read articleBuilding a City Where Children and Young People Can Fulfill Their Dreams, Together with Citizens: Higashiosaka City's Effort to Uncover Honest Views Through Conversational AI
A planning case that used conversational AI to get at the honest views behind surface-level requests.
Read articleFrequently asked questions
- Can I leave the classification of free-text responses entirely to AI?
- It is safer not to leave it entirely to AI. Check the AI's summaries and classifications against the original responses. The conclusions are drawn by people, such as a board of officers, a general meeting, or the local government.
- What should I do if responses include names or addresses?
- Before passing the data to AI, remove or mask any statements that could identify individuals. Decide how far the data may be used within the scope of use you described to respondents in the survey invitation.
- How should minority opinions be handled?
- The fewer responses an opinion has, the more likely it is to be left out of an AI summary. After classification, read the "responses that did not fit any category" in their original text and check whether there are opinions that should be kept.
- In Baisoku Survey, how do you check the basis of a summary?
- Each sentence in the overall report is followed by a code (e.g., U24-Q5 = the 24th respondent's answer to question 5). Clicking the code shows the original answer, so you can compare the summary with the original text on the spot.
Try Baisoku Survey
Baisoku Survey starts with the Free plan (¥0, 10 per month). Write the questions, collect responses, and organize the free-text answers on one screen.
Open the Baisoku Survey site (opens in a new tab)