Double-barreled responses: Congress poll on lowering minimum age of criminal responsibility

A one-question online poll on the website of the Philippine Congress gives me the opportunity to discuss questionnaire design a little bit.

congressciclpoll

 

This primer by Jon Krosnick outlines conventional wisdom on how to design a survey question from decades of research in survey methodology. Among the recommendations are to make response options exhaustive and to avoid leading or loaded word choices that push respondents towards an answer.

The problem with the response options above is that they are what are known as “double-barreled” response options, where the question asks you for one answer but each response options is actually two answers. You have the option of choosing Yes AND agreeing that the youth should be responsible for their actions and words as early as possible, or choosing No AND agreeing that punishing children violates child rights. The response options thus do not exhaust other possible ways of framing the issue. For example, a respondent might think that the youth should be responsible as early as possible but that nine years is too young, or that effective punishment would not necessarily deprive children of the chance to improve their lives.

Clearly, the question laudably tried to present balanced viewpoints on the issue. Certainly, allowing respondents to grapple with some arguments in favor of or against the question would result in more informed responses than if the response options were just straight-up Yes, No or Undecided. In order to avoid the double-barreling discussed above, a better way to ask the question would be to simply move the arguments from the responses into the question, as follows:

Lawmakers have proposed that the minimum age of criminal responsibility be lowered from 15 to 9 years. [RANDOMIZE ORDER OF PARENTHESES] Some argue that (the youth should be responsible for their actions and words as early as possible and that this would serve as a deterrent to the use of youth in the commission of crimes), while others argue that (punishing children in conflict with the law violates child rights and deprives them of the chance to rebuild their lives and improve their character). Do you favor or oppose this proposal, or are you undecided?

  1. Favor
  2. Oppose
  3. Undecided

By doing so, we allow respondents to consider policy arguments in their heads while not railroading them into agreeing with a fixed set of arguments along with their opinion on whether they favor or oppose the proposal.

I would also suggest two changes which are already incorporated into the above question. First is to remove the phrase that mentions “unduly pampering” children with “impunity from criminal responsibility”. These word choices are loaded with negativity and may thus influence respondents’ answers. Furthermore, the argument being made in that phrase is effectively the same as arguing that children should take responsibility at an earlier age, thus making this statement redundant.

Second is to randomize the order in which the arguments are presented to the respondent. The above text would not be what is shown to the viewer; it would be what is shown to instruct whoever is in charge of programming the survey. Upon visiting the website, each respondent would have a 50-50 chance of seeing one of the two following questions:

Lawmakers have proposed that the minimum age of criminal responsibility be lowered from 15 to 9 years. Some argue that the youth should be responsible for their actions and words as early as possible and that this would serve as a deterrent to the use of youth in the commission of crimes, while others argue that punishing children in conflict with the law violates child rights and deprives them of the chance to rebuild their lives and improve their character. Do you favor or oppose this proposal, or are you undecided?

vs.

Lawmakers have proposed that the minimum age of criminal responsibility be lowered from 15 to 9 years. Some argue that punishing children in conflict with the law violates child rights and deprives them of the chance to rebuild their lives and improve their character, while others argue that the youth should be responsible for their actions and words as early as possible and that this would serve as a deterrent to the use of youth in the commission of crimes. Do you favor or oppose this proposal, or are you undecided?

There is a possibility that the argument that respondents see either first or most recently would stick out in their minds more. We do not want this to influence their answer, and switching up the order of the arguments should help prevent this.

There are of course broader issues to consider. The poll itself is an interesting exercise, but without any information on who is answering (i.e. who makes up the sample), the poll cannot provide any evidence on Philippine public opinion. Nearly 18,000 respondents as of 10:00 AM on 13 January 2017 may look more impressive than the 1,200 respondents we see from SWS or Pulse Asia surveys, but they may, for example, largely be highly-educated, English-speaking, middle to upper class Filipinos with Internet access who were interested enough in politics to answer a poll on the Congress website – or, in short, unrepresentative of the Filipino population as a whole. Hopefully no lawmaker or pundit will look to this poll to say that 90% of Filipinos oppose the proposal, because this poll does not provide sufficient evidence for that assertion.

Statistical adjustments of the results of online polls in order to reflect the population is an active field of research, but the bare minimum required to start doing so would be demographic information on the respondents. Without that, the poll is nothing more than a curiosity.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s