Online Surveys and Self-Selection

For this week, I figured I would go back and take another look at more scientific online surveys as a research method. After the first week, with the personality quiz type of survey which I personally tend to strongly dislike and distrust, I thought it would be interesting to look at more scientific online surveys, ones that gather data rather than attempt to tell people something after a small number of multiple choice questions.

When looking at online surveys as a method of gathering information, they can be extremely useful as their convenience can create larger sample sizes and through this generate better data. There are a number of problems with them as well, however. Knight makes two very important points about this, that questionnaires which are not conducted face to face, like those that rely on a strict script, allow for very little response from the researcher as the questionnaire progresses. They also tend to work best when the questions being asked can be answered quantitatively, and if the information being sought is known ahead of time (Knight, 2002).

Knight also mentions one particular problem with online surveys, that of the response rate. It can often be as low as 30%, and is sometimes worse (Ibid). A particularly interesting study I found a year or two ago, working on an unrelated topic, was used to see who had been answering surveys related to an online game, specifically World of Warcraft. Their conclusions are largely what would be expected, and the people who were most likely to be highly interested in the game were the ones who would actually answer the surveys about the game. Considering that they used the level and gear possessed by the respondents' avatars to determine this, they were unquestionably the people answering were certainly the ones who had sunk the most time or money into the hobby (Khazaal et al., 2014).

The questions I am looking at relate to how to attract new users to cultural institutions, while this research shows that the people who are actually most important to question, the new and non-users, are least likely to want to take the time to answer surveys. When looking at how to potentially question people as to what will work in bringing cultural institutions to their attention, a method which will attract mostly those who are already interested is not going to be quite as useful. This self-selection based on existing interest, while not unexpected, is also not particularly helpful.

Sources:

Khazaal, Y., Singer, M. van, Chatton, A., Achab, S., Zullino, D., Rothen, S., … Thorens, G. (2014). Does Self-Selection Affect Samples’ Representativeness in Online Surveys? An Investigation in Online Video Game Research. Journal of Medical Internet Research, 16(7), e164. http://doi.org/10.2196/jmir.2759 

Knight, P. (2002). Small-scale research: pragmatic inquiry in social science and the caring professions. Thousand Oaks, CA: SAGE. Retrieved from