BCIFV home > Newsletter > 1997 Archives > Winter 1997 articles

Research Notes
The Problem of the Self-Report in Survey Research

Whether or not people tell the truth when answering questions as part of a survey is a thread that is woven through past methodological work on survey research. Given that the self-report is the very foundation of survey research, however, it is surprising that this issue has not received, in relative terms, more sustained attention.

The most basic way to determine whether respondents give honest answers to survey questions is to use a validation measure external to the interviewing situation to verify respondents' answers. For example, responses to questions about voting can be compared to voting records; question about owning a library card can be compared to library records; age reported in surveys can be compared to birth certificates; drug use can be tested by comparing survey answers to results of tests designed to measure the presence or absence of drugs in blood, urine or saliva, and so on. Validation or 'record check' studies cannot be completed for questions that measures attitudes, values, or beliefs, however, as there are no external sources available for comparison.

A general conclusion that can be drawn from reviewing validation studies completed over the last four decades is that misreporting is associated with the extent of perceived question threat. Misreporting is negligible for non-threatening questions such as home ownership, low for questions about library card ownership, higher for questions about drinking and driving, and highest for questions about abortion - especially when the woman is from a cultural or religious group where abortion is unambiguously unacceptable behavior. This relationship between increased question threat and decreased reporting is widespread.

Does the problem of the self-report mean that survey research is not a useful tool in understanding the attitudes, values, beliefs and behaviours of individuals and groups in society? No. This is not to say, however, that there is not bad survey research or questions for which reliance on the self-report makes the survey a poor tool for serious efforts to understand the social world. The arguments for the utility of survey research are as follows.

If 15 to 20 percent of respondents' answers to questions about owning a library card are dishonest, then 80 percent answered honestly. The percentage of honest (i.e., validated) answers about voting is estimated at about 70 percent. While there is considerable variation in the amount of denial, as many as two thirds of survey respondents who have been arrested for drinking and driving, which is both a criminal act as well as stigmatized behaviour, have not denied their actions when questioned about it is a survey. That 80 to 90 percent of drug users give consistent answers in longitudinal surveys indicates that a high proportion of survey respondents give honest reports about counter-normative behaviours.

Certainly, it is important to maximize the conditions under which respondents will give honest answers, but there is good reason to believe that dishonesty is not the norm for survey questions about sensitive behaviours. Given that misreporting is positively associated with the degree of question sensitivity or threat, it is reasonable to conclude that misreporting for non-sensitive questions is very limited.

The survey process itself is structured to encourage honesty. Participation itself is structured to encourage honesty. Participation is voluntary and in most surveys one quarter to one third of the people approached to participate decline. While some respondents may feel it is bad manners not to assist in a survey, and others may feel pressure to participate because the government or another powerful group is conducting the survey, most people who complete the survey do so willingly.

The survey puts respondents in a position where they are told that they have something to contribute and their views matter - that is, the survey approach creates the dynamics for truth-telling. Good interviews convey to respondents that all answers are acceptable. Some people will give honest answers as they believe in telling the truth, others because they do not feel the need to give socially acceptable answers, and others will be more likely to be honest because of the way the interview is conducted. For these respondents, voluntary participation, encouragement to voice their views, and assurances that there are no right or wrong answers will help them overcome the need to give the socially acceptable answer.

Survey research simplifies complex processes, but just like the simplification of an experiment adds to our understanding of complex natural phenomena, the survey adds to our understanding of complex human phenomena.

Surveys do not have the controls, nor the artificial world, of the laboratory experiment. The survey is closer to the real world of attitudes and behaviours than the experiment, but it is not as close to the real world as unobtrusive observational studies. (Of course, observational studies and laboratory experiments are too, not without their own measurement problems.) Unlike observational studies, the survey provides us with large samples which shed insights into how different social and demographic factors intersect with the attitude, value, belief, or behaviour of interest.

The problems faced in obtaining accurate information in surveys are the same problems we face in everyday communication. Spouses exhort spouses to tell each other what they really think and feel. Parents, particularly when their child has engaged in unacceptable behaviour, work hard at encouraging children to tell the truth, even if it is painful. Workers promise their bosses the assignment will be completed well, and on time, when they know that it is almost impossible to do both. There is much we can learn in everyday life by reading between the lines; the same is true in survey research possibilities as well as research problems.

This article is abstracted from a larger paper on the problems of the self-report in survey research. Copies of the paper are available from the Institute for Social Research.

David A. Northrup, Manager of Survey Research
Institute for Social Research.
York University, 4700 Keele Street
North York, Ontario M3J 1P3
Ph: (416) 736-5061 Fx: (416) 736-5749
Internet: ISRNEWS@VM1.YORKU.CA