Why the Lima-commissioned SWS survey is not inaccurate

Please read the entire post.

Social Weather Stations (SWS) have taken a lot of flak for only partially publishing the results of a survey commissioned by Davao businessman William Lima, in where the “if the elections were held today…” question seems to have been asked in a way that would prime Rodrigo Duterte in respondents’ minds. The question also seems to be the second of two or more questions asking respondents who they would vote among different sets of candidates, and only the results of the question mentioning Duterte were published. However, it was not SWS who published the results, but rather the businessman who paid for the survey and who had the option to hide or publicize the results as they see fit.

lima_sws_duterte

A lot of people have already discussed the issue at length, so I won’t belabor it here. Instead I want to point something important out about the nature of surveys.

Surveys sample a small portion of a population and, through probability sampling or post-stratification, obtain results that are supposed to reflect the opinions of that population. The actual opinions of the entire population are unknown and will always be unknown.

No matter how well done the survey is, these surveys are still estimates that contain various kinds of errors. The “margin of error” is just one kind of error that can occur in a survey; other errors include coverage error, where some segment of people might be unreachable; nonresponse, where some segment of people refuse to answer the survey; timing error, where a survey is conducted but the results published only several weeks afterwards, between which opinions might have changed (especially regarding something as volatile as Philippine politics), and the main error of focus in this controversy, survey wording error, where a question was not asked in the right way to get the respondent to supply the information intended. Any survey will always have a margin of error, because it is a mathematical inevitability. But a well-done survey will have been conducted in a way that reduces if not outright eliminates errors that have to do with how the survey was administered.

The presence of one or more of these errors, however, only raises questions about the validity of the survey instrument, not about the accuracy of the results. If we think, for example, that the way the question in the Lima SWS survey was worded biases the measured results, this does not mean that the results are right or wrong. It just means we can’t consider the survey to be a credible measurement. We cannot ever know the true preferences of the entire population unless we gain the magical ability to instantaneously survey every single person and give them a dose of truth serum to ensure they aren’t lying to us.

The Lima SWS survey is not “wrong” per se, it is simply invalid. Had the question been asked in a less biased manner, it is very possible that Duterte would still come out on top. Thus we cannot take the unreliability of the SWS survey as proof that Duterte is not actually people’s top choice. We cannot ever know whether the results of a survey are accurate; we can only know whether the survey was valid.

The only way we can assess the validity of a survey is to review in detail the exact methodology of the survey. This includes not just the details of how stratification or weighting was conducted, but also details like whether respondents were contacted through phone or through face-to-face interviews (the Lima survey was through face-to-face interviews, as I think all SWS surveys are), whether or not surveyors used a list of registered voters to sample from per stratum, the number of surveys that were not completed, etc., etc. A long-standing institution like SWS may have decades of experience in administering surveys, but they and any other surveyor are still obliged to publish exactly how each survey was conducted, even if it’s the same way every time. Any missing details count against the validity of the survey. But let’s give credit where credit is due: the full question wording was published instead of being covered up, for example.

If a survey is valid enough, then we can trust that its results have a greater probability of being accurate. But that probability is not 100%. All surveys are estimates; some estimates are just better than others.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s