Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research

Authors

E. Vance Wilson
Nancy K. Lankton, Marshall UniversityFollow

Document Type

Conference Proceeding

Publication Date

8-2012

Abstract

Arranging survey items to group measures of the same construct together has several benefits, including ease of administration and enhanced statistical reliability and validity of constructs. Yet some IS researchers claim this practice contributes to common methods bias and camouflages “true” measures of reliability. Our study takes a new approach to this issue by using a range of IS research measures in an online survey context to contrast grouped-item survey administration with a design in which the ordering of item administration is programmatically re-randomized for each individual subject. We find significant differences in construct reliability between grouped-item and individually randomized treatments as well as strong temporal effects and widespread anomalies related to item-ordering in the grouped-item treatment. Our results suggest the purported benefits of grouped-item surveys are outweighed by hazards these create to the integrity of research findings, and we caution IS researchers against their continued use.

Comments

Presented at the Thirty Third International Conference on Information Systems, Orlando 2012. Copyright © 2003 Association for Information Systems.

Recommended Citation

Wilson, E. Vance and Lankton, Nancy, “Some Unfortunate Consequences of Non-Randomized, Grouped-Item Survey Administration in IS Research” (2012). ICIS 2012 Proceedings. Paper 4.