Title
Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research
Document Type
Conference Proceeding
Publication Date
8-2012
Abstract
Arranging survey items to group measures of the same construct together has several benefits, including ease of administration and enhanced statistical reliability and validity of constructs. Yet some IS researchers claim this practice contributes to common methods bias and camouflages “true” measures of reliability. Our study takes a new approach to this issue by using a range of IS research measures in an online survey context to contrast grouped-item survey administration with a design in which the ordering of item administration is programmatically re-randomized for each individual subject. We find significant differences in construct reliability between grouped-item and individually randomized treatments as well as strong temporal effects and widespread anomalies related to item-ordering in the grouped-item treatment. Our results suggest the purported benefits of grouped-item surveys are outweighed by hazards these create to the integrity of research findings, and we caution IS researchers against their continued use.
Recommended Citation
Wilson, E. Vance and Lankton, Nancy, “Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research” (2012). ICIS 2012 Proceedings. Paper 4.
Comments
Presented at the Thirty Third International Conference on Information Systems, Orlando 2012. Copyright © 2003 Association for Information Systems.