Back to all guides
Organizing and analyzing opinions

How Do You Organize and Classify Public Comment Submissions with AI?

Key points

Divide the submissions by issue and group opinions with the same intent before writing the government's views.

Use AI to draft classifications and summaries, and have staff check them against the original text.

A large number of submissions does not make an opinion correct, so present the government's views based on the content of the opinions.

Updated

Overview

In public comment (a call for public opinions), the opinions submitted on a draft plan or regulation are organized and published together with the government's views. When there are many submissions, reading them, classifying them, and writing the views takes a lot of time.

With AI, you can produce drafts in a short time for tasks such as dividing submissions by issue and grouping opinions with the same intent. However, the government is responsible for what is published. The premise is that staff check the AI's classification and that the original submissions can be traced at any time.

The purpose of organizing is not to reduce the number of opinions, but to put them in a form where both the government and the people who submitted them can confirm which part of the draft each opinion addresses and for what reasons. Keeping this purpose in mind makes it easier to decide how much to leave to AI and what people should handle.

This page sets out the general steps for organizing public comment submissions with AI and the points to watch.

What to decide before you start

Before you start using AI, deciding the following three points within the responsible department reduces rework later.

  • Unit of classification: whether to divide according to the chapter structure of the draft or by theme
  • Scope passed to AI: items such as names and contact details to mask before passing the data
  • Division of checking: who checks the AI's drafts, and in what order

Steps

  1. Divide the submissions by issue. A single submission often contains several issues. First split the submissions into units of issues, and tag each with the part of the draft (chapter, article, or measure) it addresses. If you give the AI the structure of the draft and then have it split and sort the submissions, the draft is easier to produce.
  2. Group opinions with the same intent. For each issue, group together opinions with the same intent. Combine those that make the same claim even if the wording differs, and keep separate those whose claims differ even slightly. Staff review whether the AI's grouping is too coarse or too fine and adjust it.
  3. Link the original text to the counts. For each grouped opinion, attach the numbers and count of the original submissions it is based on. This is to avoid a situation where only the summary remains and is no longer connected to the original text. It also lets you answer if someone later asks "where did my opinion end up?"
  4. Write the government's views (responses). For each grouped opinion, write whether the draft will be revised or not, and why. Even if you have AI produce a draft, the responsible department decides the policy and reasons and rewrites it.
  5. Publish. Publish the gist of the opinions, the counts, and the government's views. Before publication, check that no statements that could identify individuals remain.
TaskWhat is easy to leave to AI as a draftWhat staff handle
SplittingDividing submissions into units of issues and tagging the relevant part of the draftChecking whether the units of splitting are appropriate
GroupingGrouping opinions with the same intentAdjusting the granularity of grouping
LinkingTallying original submission numbers and countsChecking for omissions and duplicates
ViewsDrafting the textDeciding the policy and reasons, and finalizing the text

Points to watch

A large number does not mean correct

A large number of submissions for an opinion shows that many people are interested in it, but it does not show that the opinion is correct. Even an opinion with few submissions may point out something the draft overlooked. Write the views in response to the content of the opinions, not their count.

Mass submissions of identical text

Sometimes many submissions with identical wording arrive. Treating identical wording together and showing it with the count makes it less likely that other opinions get buried. Even if the wording is the same, if an original opinion has been added at the end, pick up that part separately.

Staff check the AI's classification

AI may misread the intent of opinions that depend on context, such as sarcasm or conditional support. Read the original text for each category and correct any sorting errors. In particular, check one by one the opinions that did not fit any category.

Keep the original text

Even after classifying and summarizing, keep all of the original submissions. A summary is a tool for organizing, not a substitute for the original text. If the correspondence between the original text and the summaries is kept, the organization can be redone or verified later.

What you can consult Plural Reality about

Plural Reality can work with you on designing how to divide the submitted opinions into units and how to organize them while linking them to the original text.

As a useful example, in Ota City, Gunma Prefecture's program review (10 programs in total), Baisoku Survey was used to collect the views of observers, and a report was produced that also broke down the comments on the evaluation sheets into issues and reasons. For each responsible department, it was set up so that the breakdown of verdicts, how the conclusions were divided, and the original text could be checked. The idea of dividing opinions into issues and reasons and organizing them so that you can go back to the original text applies to public comment as well.

Related case studies

Frequently asked questions

Can AI classification results be published as they are?
Staff check them before publication. AI classification may misread the intent of an opinion or mix in opinions on a different issue. Publish the version that has been corrected against the original text.
What should be done when a large number of submissions with identical wording arrive?
One approach is to treat identical wording together and show it with the count. A large number of submissions does not mean the opinion is correct, so write the government's views in response to the content of the opinion.
How much of the original text should be kept?
Even after classifying and summarizing, keep all of the original submissions. Making it possible to trace which summary is based on which submissions makes it easier to check the work and respond to inquiries.
What can I consult Plural Reality about?
You can consult us on designing how to divide the submitted opinions into units and how to organize them while linking them to the original text. In Ota City's program review, we produced a report that broke down the comments on the evaluation sheets into issues and reasons.

Talk to us about your project

From designing a meeting or survey to running it, we help you decide where to start. Please contact us.

Contact us