Comments regarding the recent Alliance Focus Group Report

Recently, the Alliance, an organization of 15 of the 20 Colorado Community Centered Boards and many provider agencies for services for individuals with intellectual and developmental disabilities, issued a widely distributed report (attached) based upon five focus groups throughout the State of Colorado. According to the report, various Colorado Arcs furthered the process by selecting (using specific criteria, including the two criteria listed below) the participants for the focus groups.

There are significant concerns with the validity and generalizability of the findings of this report. We have had the report reviewed by some experts and practitioners of focus group research. The comments of these individual reviewers (below) are in italics.

In regards to the use of focus groups for this type of “research”
.
Comment: All the limitations on generalizability and utility are pretty well spelled out there [referencing the book below]

Merton, R.K., Fiske, M., & Kendall, P.L. (1956).The Focused Interview: A Manual of Problems and Procedures. New York: Free Press.

.Most research methodology geeks that I know of believe that Merton’s original intents were legitimate, but that the method has since been expanded to absurd dimensions – to try to do things that the method was never designed or intended to do.

I think the report [the Alliance Report] in question is such an instance.

The report then lists the following criteria for choosing participants:

Criteria #1. Representative of the designated region;

Comment: “Item 1 is, to me, utterly unjustified and unjustifiable. Only strict valid sampling theory can guide one to be “representative” of the designation region.”

Comment: “Focus groups are a qualitative method that must never claim to form a representative image of any larger group phenomenon. Yet the stated selection criteria in the report seem to claim that, right off the bat:”

Comment: Instead, a focus group should be seen as a completely non-representative “sample” intended only to expand the range of issues to be explored. It is a tool to help researchers think of things they haven’t yet thought of, things they can investigate more deeply with better methods.

Criteria #4) Unlikely to polarize others (sic) participants

Comment: Item 4 is bizarre, in that it automatically excludes participants who are “outliers.” And one of the primary purposes of focused interviews is to learn more, to uncover things we haven’t thought of. Excluding the extremes, even if they polarize, seems to me to contradict the most fundamental purpose of the entire approach. It suggests a fundamental misunderstanding (on top of the usual error of assuming findings represent some larger population).

Comment: But it’s a lot like jury selection. When screening is overdone, when it goes beyond simply having experience with the topic at hand and being able to refrain from disrupting the discussion, participants may get selected according to the result wanted. There are entire books and journals devoted to jury selection. It ain’t random, and juries sure don’t reflect the characteristics of the population at large. Prosecutors and defenders try to select for desired outcomes, just like this focus group facilitator MIGHT have.

Comment: "This past year I did a series of focus groups around the issue of the kind of training service coordinators (disability related programs) and care mangers (aging related programs). I welcome people who others see as polarizing. While it means I may have to work harder, it's what I get paid to do, and you often hear people who are willing to tell the emperor that he's not wearing clothes. Sounds like your concerns about the data are worth consideration in terms of identifying someone who is polarizing.”

A general comment: on focus group research:

Comment: One of the things I did in my career was to study situations and present solutions. I did this over a wide variety of subjects, industries and profit/non-profit groups. One of the things I learned was that focus groups were almost always used as an advocacy vehicle for a particular viewpoint. Seldom were they useful in arbitrating or resolving issues unless those forming the focus group had almost enough power to get their way and needed just a little more to close the deal.