Open-ended questions: what they do, what they cost, and when to close

The formal definition is about constraint. A closed question offers a set of permitted answers, explicitly ("weekly, monthly or never?") or implicitly ("did that work?"). An open question does not, so the respondent chooses the terms.

That is openness of form, and it is the version every guide teaches. It is easy to check and easy to fake.

Openness of substance is different and matters more. A question is open in substance when its answer could genuinely surprise you and could genuinely be bad news. By that test, "what would make this better?" is closed, because it presupposes that the thing should exist. "Would you pay for this?" is closed, because there is a socially obvious answer. "Tell me about the last time you had this problem" is open in both senses, because you do not know what is coming and the respondent is not being asked to be kind to you.

Most bad interviews are conducted entirely in open-form questions. Keeping both tests in mind is the difference between collecting answers and collecting information.

The experimental work on question form goes back to Howard Schuman and Stanley Presser's Questions and Answers in Attitude Surveys, and its central finding about closed questions is stronger than the usual summary.

The usual summary is that closed questions restrict what people can say. What the experiments show is that they also teach. Respondents treat the options offered as the universe of reasonable answers, so a closed question about causes tells the respondent which causes count, and answers outside the list are suppressed even when they were the respondent's actual view. The list is not a menu, it is a frame.

Which means that a closed question asked too early does not merely gather less. It contaminates everything after it, because you have now told the participant what kind of thing you are interested in, and cooperative people take that hint.

None of this makes closed questions bad. It makes their position in the sequence the thing that matters, which is what the funnel below is for. When you do want them, closed questions are the right instrument in four situations, and there is a section on those further down.

If you only borrow one method from outside your field, borrow this one. The cognitive interview was developed by Ronald Fisher and R. Edward Geiselman in the mid-1980s for police work, and it is built on an uninterrupted open-ended account followed by four specific components.

• Mentally reinstate the context. Ask the person to reconstruct where they were, what was around them, and how they felt, before asking what happened. Recall is context-dependent, and rebuilding the context is the cheapest available intervention.
• Report everything. Explicitly instruct the person to include detail regardless of how peripheral it seems. People edit for relevance by default, using a model of relevance that is yours, not theirs, and they discard exactly the material you did not know to ask for.
• Describe it in several orders. Backwards, or from the most memorable moment outward. Narrative order is a story, and stories smooth over the parts that do not fit.
• Change perspective. Ask what someone else in the situation would have seen. This reliably surfaces detail the person has, and would not have volunteered from their own point of view.

The revised version reported a 45 per cent increase in correct information retrieved over standard interviewing. The finding that makes it credible rather than merely impressive comes from Amina Memon, Christian Meissner and Joanne Fraser's meta-analysis of twenty-five years of the literature: accuracy was essentially unchanged, around 85 per cent against 82 per cent. So the technique produces substantially more information without degrading its quality, which is exactly the claim a practitioner should want tested, and it has been.

The funnel is the sequence: broad open questions first, narrowing to specifics, and closed questions last if at all. Nielsen Norman Group teaches it as the core structure of a user interview, and it is worth knowing that in one field it is not advice but policy.

The UK's PEACE model, developed in the 1990s in direct response to documented forced confessions and wrongful convictions, spells out its five stages as preparation and planning, engage and explain, account, closure, and evaluate. The rule inside the account stage is explicit: before any probing questions are asked, interviewees are encouraged to give their free, uninterrupted account.

That origin is the argument. The funnel was not adopted because it is polite or because it makes for better rapport. It was adopted because the alternative demonstrably produced false information from cooperative people, and a profession with the strongest possible incentive to get quick answers concluded that leading early costs more than it gains.

The practitioner shorthand taught alongside it is TED: tell me, explain to me, describe. Three stems that are hard to make leading and that produce an account rather than an answer. Treat them as craft rather than as a citation; they are widely taught and have no traceable origin.

Rob Fitzpatrick's The Mom Test makes the argument the interviewing literature mostly does not, and it is aimed at exactly the questions that pass the grammatical test and fail everything else. Its rule of thumb is that you should be able to ask your questions of your own mother and get useful answers, which is impossible if the questions are about your idea, because she will be kind.

Three moves do most of the work. Ask about the past rather than the future, because people are poor forecasters of their own behaviour and excellent reporters of it. Ask about their life rather than your idea, because their life is a fact and your idea is a hypothesis they are being asked to be nice about. And look for what they have already done about the problem: money spent, workarounds built, time lost. Behaviour that has already happened is the only kind that cannot be politeness.

Stems are craft rather than science, but the right stem for the situation removes most of the risk of leading.

The probe column matters more than the opening. Anyone can ask a good first question; the interview is decided by whether the second one follows the participant or returns to your list.

Open questions are not universally better, and a page that pretends otherwise is not useful. They cost time, they are harder to code and compare, and they produce worse data from people who lack the vocabulary or the motivation to elaborate. Four situations call for a closed question.

Comparability across participants. If you need to say that six of eight participants did something, you need the same question asked the same way, and an open question will not give you that.

Surveys at scale. Open fields in a survey go unanswered, and the ones that get answered come from an unrepresentative minority with strong feelings. Ask closed questions and use one open field at the end.

Time pressure. Ten minutes with a senior stakeholder is not the place for a free account. Ask the two closed questions you actually need.

Confirming a specific fact. Once the account is given, closing down to check a detail is not contamination, because the frame has already been set by the respondent.

Some useful closed forms, since they get less attention than they deserve. A binary check: "did you do that before or after the export failed?" A scale: "on a scale of one to five, how confident were you?" A forced choice: "if you could only keep one of those, which?" And a list confirmation: "you mentioned three things, was there a fourth?" Each is precise and each would have been destructive as an opening move.

Everything above is a skill, and skills need feedback. The problem is that in a session, the moderator is the person least able to observe the moderating: you are listening, deciding where to go next, and managing the time. The three failures in the previous section are all invisible from inside, which is precisely why they persist.

The practice that works is unglamorous. Read your own questions back afterwards, in order, without the answers. Five checks, and every one of them works on the questions alone.

• Did the funnel actually narrow? Most moderators believe they funnelled and jumped to specifics by the third question. The sequence of forms tells you immediately.
• How many were leading? A leading question is identifiable from its own text. You do not need the answer to see it.
• How many were stacked? Two questions in one turn is the most common defect and the least visible live.
• What did you do with the silence? A question immediately followed by another from your side is the textual signature of filling a pause.
• Open in form, closed in substance? This is the one you cannot catch any other way, because in the moment "would you use this?" sounds like curiosity.

All of that requires the session to still exist in text, which is where Earkeep fits. It records your day continuously on your own device, transcribes it there, and writes plain files you own, so the session you moderated is available afterwards in order with timestamps, without a separate recording step. An agent in the panel can list the questions asked in order, which is a mechanical transformation of the record rather than an assessment of it: no scoring, no coaching, no feedback, just your questions extracted from the conversation so you can read them.

Two honest limits. The record does not label speakers, so this is your reading of your own session; in a two-person interview the questions are almost always identifiable from context, which is enough for all five checks, and is not the same thing as the tool knowing who spoke. And a research session is recorded with the participant's knowledge: what Earkeep does and does not keep is on privacy, and how to say it before you start is on telling people you're recording.

Questions are one of three things that decide an interview. The second is whether the person is willing to tell you anything, which is establishing rapport, and the third is what you do with the answer once it arrives, which is empathic listening. Those three pages are written to be read together. On the professional side, the interviewing craft as it lands in a research practice is in Earkeep for UX researchers and for UX designers, and the discovery half is in for product managers. The review that turns a session into something the team learns from is a debrief.