Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports

The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of the methodological decisions faced by researchers is deciding on the question scheduling. The literature defines three distinct schedule types: randomised, interval-based, or event-based (in our case, smartphone unlock). However, little evidence exists regarding the side-effects of these schedules on response rate and recall accuracy, and how they may bias study findings. We evaluate the effect of these three contingency configurations in a 3-week within-subjects study (N=20). Participants answered various objective questions regarding their phone usage, while we simultaneously establish a ground-truth through smartphone instrumentation. We find that scheduling questions on phone unlock yields a higher response rate and accuracy. Our study provides empirical evidence for the effects of notification scheduling on participant responses, and informs researchers who conduct experience sampling studies on smartphones.

Authors:
van Berkel Niels, Goncalves Jorge, Lovén Lauri, Ferreira Denzil, Hosio Simo, Kostakos Vassilis

Publication type:
A1 Journal article – refereed

Place of publication:

Keywords:
accuracy, contingency, data quality, Ecological momentary assessment, EMA, ESM, Experience sampling method, mobile questionnaires, Response rate, sampling, Self-report, smartphone, validation

Published:

Full citation:
Niels van Berkel, Jorge Goncalves, Lauri Lovén, Denzil Ferreira, Simo Hosio, Vassilis Kostakos, Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports, International Journal of Human-Computer Studies, Volume 125, 2019, Pages 118-128, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2018.12.002

DOI:
https://doi.org/10.1016/j.ijhcs.2018.12.002

Read the publication here:
http://urn.fi/urn:nbn:fi-fe2018121951344