Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports

The Experience Sampling Method is widely used to collect human labelled data in the wild. Using this methodology, study participants repeatedly answer a set of questions, constructing a rich overview of the studied phenomena. One of the methodological decisions faced by researchers is deciding on the question scheduling. The literature defines three distinct schedule types: randomised, interval-based, or event-based (in our case, smartphone unlock). However, little evidence exists regarding the side-effects of these schedules on response rate and recall accuracy, and how they may bias study findings. We evaluate the effect of these three contingency configurations in a 3-week within-subjects study (N=20). Participants answered various objective questions regarding their phone usage, while we simultaneously establish a ground-truth through smartphone instrumentation. We find that scheduling questions on phone unlock yields a higher response rate and accuracy. Our study provides empirical evidence for the effects of notification scheduling on participant responses, and informs researchers who conduct experience sampling studies on smartphones.

van Berkel Niels, Goncalves Jorge, Lovén Lauri, Ferreira Denzil, Hosio Simo, Kostakos Vassilis

Publication type:
A1 Journal article – refereed

Place of publication:

accuracy, contingency, data quality, Ecological momentary assessment, EMA, ESM, Experience sampling method, mobile questionnaires, Response rate, sampling, Self-report, smartphone, validation


Full citation:
Niels van Berkel, Jorge Goncalves, Lauri Lovén, Denzil Ferreira, Simo Hosio, Vassilis Kostakos, Effect of experience sampling schedules on response rate and recall accuracy of objective self-reports, International Journal of Human-Computer Studies, Volume 125, 2019, Pages 118-128, ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2018.12.002


Read the publication here: