We recently published a special report titled 25 Common Field Mistakes to Avoid When Conducting Your Qualitative Market Research. In point #12, we suggest that researchers scrap open-ended screening questions that are intended to identify respondents who are outgoing and able to express themselves.
Here is the complete text of point #12:
“If your screener contains open-ended questions that are intended to elicit expressive types of people, drop those questions. Questions like that don’t work, and they needlessly lengthen your screening process. Yes, you do want to exclude respondents who cannot or will not express themselves, but you don’t need an extra question to identify these people. Well-trained recruiters will eliminate them within the first few minutes of screening. If you want further reassurance that your respondents will be outgoing and talkative, over-recruit and include a pre-discussion telephone interview. This would be conducted by the moderator who would then select appropriate respondents.”
Our suggestions elicited a lot of feedback. Some readers agreed with us, others didn’t. One of the most interesting comments we received came from a qualitative fieldwork manager at a major full-service research firm. It went like this: “I disagree with one item in your list about excluding open end questions from screeners. Good recruiters may be able to easily identify articulate respondents, but tired or distracted recruiters may occasionally go on autopilot…I think a little qualitative in a screener is worth the time.”
Do you agree with this reader’s comment? Should you?
What is an articulation question?
Articulation questions measure a respondent’s ability to communicate. Articulation questions also judge respondents’ expected communicativeness in a focus group or interview.
Some synonyms for “communicative” include: outgoing, open, forthcoming, talkative, unrestrained, chatty. So who decides what is communicative? The recruiter? The recruiting supervisor? The client who reads the verbatims on their daily reports? And how much communicativeness is enough? How much is too much?
Even the most experienced recruiters can’t determine how outgoing, open, forthcoming, talkative, unrestrained or chatty a respondent will be at a future point. That’s a judgment call recruiters are not qualified to make. But they can be counted on to spot respondents who have…
- language barriers
- casual attitudes toward the recruiter, the recruiter’s questions or the research
- reservations about their ability to attend the research
- any problems communicating during the screening process
What you must watch for…
Respondents get tired or go on autopilot when screening interviews last too long (10 minutes or longer).
Articulation questions don’t belong at the end of your screener. For some reason, articulation screening is almost always performed at the end of the screening interview. But why is a question that is supposedly so important put at the end of the screener, when the chances for respondents to be tired or distracted are the highest? What are recruiters learning about respondents at this point in the process that they don’t already know?
Articulation questions don’t belong at the front of your screener, either. Well-trained recruiters immediately engage respondents in conversation regarding the details of the research. It is during this prelude to the screening questions that recruiters deal with respondents’ questions and concerns and make an assessment about a respondent’s ability to communicate.
Articulation questions aren’t magic bullets that ensure good focus group participants. These questions simply ask recruiters to use their own biased judgment to decide if a respondent can communicate clearly.
Articulation questions lengthen your screener. Remember this. The longer your screener, the higher your costs.
Respondents become anxious when asked questions out of left field that are unrelated to the screening questions. Being asked, “What is a gazinkle?” or “How many different things can you do with a paperclip?” or “If you were a tree…?” might stump even the most articulate respondent. Off-the-wall questioning from recruiters confuses and frustrates respondents. This line of questioning is the moderator’s territory.
Of course, group dynamics and respondent personalities affect how open and responsive respondents will be. For example, a person may be forthcoming over the phone with the recruiter, but feel intimidated if an aggressive personality dominates the group. Or, a respondent may not be as comfortable with the research topic as they thought they’d be and feel out of place – especially if the subject matter offered during recruiting was vague. How can recruiters know how respondents will act in a variety of conditions? Handling reserved respondents is the moderator’s area of expertise.
In fact, moderators are best qualified to know what can and should be expected of respondents in terms of communicativeness and articulation. So it makes sense that, as we suggest in point #12 of our special report, moderators should pre-interview respondents and select the right personalities for the research.
So what about articulation questions being useful for snapping distracted or unconscious recruiters out of their daze (as our reader suggested)? Assuming that a tired, distracted recruiter missed all of the red flags during screening, will the articulation question suddenly remind the recruiter that the respondent isn’t chatty? What should you do about recruiters on “autopilot?” Simple.
The researcher’s job is not to craft questions that keep recruiters alert and focused. Tired or distracted recruiters are not an asset to your research. They don’t help you get great respondents. And neither do articulation questions. Don’t use either of them.
