New Study Reveals the Reliability of Mobile Phone Surveys to Collect Non-Communicable Disease Risk Factor Information

A new study compared the reliability of data collected by mobile phone surveys to gather information on non-communicable disease (NCD) risk factors, such as tobacco and alcohol use, medical conditions, level of physical activity, and diet. The study, led by researchers at the Johns Hopkins Bloomberg School of Public Health, compared the consistency of respondents’ answers when they took the same survey using two different survey methods: the computer assisted telephone interview (CATI) method, which uses a human caller, and the interactive voice response (IVR) method, which uses an automated system. The study was conducted in Bangladesh and Tanzania.

Even with rapidly increasing mobile phone access and use in low- and middle-income countries (LMICs) such as Bangladesh and Tanzania, it is expensive to set up and operate a survey call center staffed by personnel who need training and close supervision, used in the CATI survey method. This motivated the research team to explore the potential for using an automated system, the IVR survey method, already widely used in high-income countries, to see if it is feasible and could collect comparable data in LMICs.

The study found that, overall, answers to most of the survey questions had a moderate to high reliability from one method to the other, but not all questions yielded consistent answers. In both countries, answers to questions on history of alcohol consumption, current tobacco use, and having been checked for diabetes or hypertension showed high consistency between the two survey methods. Answers, however, to questions about levels of physical activity and weekly consumption of fruit and vegetables showed low consistency. The self-reported data was collected from 1,135 respondents in Bangladesh and 1,061 respondents in Tanzania.

Mobile phones are increasingly being used to track health in hard-to-reach populations and often have the benefit of being less costly and easier to implement than traditional face-to-face surveys. While frequently documented in high-income countries, there is little published information comparing the quality and reliability of data collected by different mobile phone survey methods in low- and middle-income countries. The lack of guidance around using mobile phone surveys in LMICs hinders their use in these regions, even though mobile phone access and use rates are high, according to data tracked by the International Telecommunications Union. This study sought to fill the gap in literature by comparing the reliability of survey answers for two different mobile phone survey methods in LMICs.

“This study shows that an automated system can be used to collect public health risk factor information quickly and results can be largely comparable to those obtained by a human caller. As these methods improve, taking advantage of increasing mobile phone access and use to collect public health information has potential to provide decision-makers the information they need to make more timely decisions and increase effectiveness of non-communicable disease prevention programs,” says George Pariyo, PhD ’00, lead author of the study and senior scientist in the Department of International Health at the Bloomberg School.

Researchers suggest that reasons for the consistency in some answers but not in others could be due to the respondent’s familiarity with the concept being portrayed in the question, how clearly certain words can be translated into the local language, length and complexity of the question itself, and potential social desirability bias associated with replying to a human interviewer. Respondents had a higher familiarity with concepts such as tobacco and alcohol use, and may have been able to readily identify response options that fit their experiences. Concepts such as descriptions of physical activity or fruit and vegetable servings may be harder for respondents to comprehend and simultaneously translate into an answer option on their mobile phone keypad in the limited time available. Vigorous or moderate physical activity, fruit or vegetable intake in a typical week, and serving sizes may have resulted in long questions, thus increasing the difficulty for respondents to answer accurately over a mobile phone.

The research was designed by a core team of faculty in the Johns Hopkins Bloomberg School of Public Health, implementing the research and development arm of the non-communicable diseases component of the Bloomberg Philanthropies Data for Health Initiative (D4H). The study team had local partners in each country to help implement the survey: The Institute of Epidemiology Disease Control and Research (IEDCR) in Bangladesh and the Ifakara Health Institute (IHI) in Tanzania.

Going forward, the research team suggests selecting questions to be administered according to the intended survey method. A careful design can help with clarity for respondents who may not have had experience answering an automated survey before. Further research is needed to explore survey reliability between mobile phone surveys and well-designed face-to-face interviews, and to better understand factors that may influence survey cooperation, completion, refusal and attrition rates across populations and contexts.

Does mobile phone survey method matter? Reliability of computer assisted telephone interviews and interactive voice response non-communicable diseases risk factor surveys in low- and middle-income countries was written by George W. Pariyo, Abigail R. Greenleaf, Dustin Gibson, Joseph Ali, Hannah Selig, Alain B. Labrique, Gulam Muhammed Al Kibria, Iqbal Ansary Khan, Honorati Masanja, Meerjady Sabrina Flora, Saifuddin Ahmed and Adnan Hyder.

The study was funded by Bloomberg Philanthropies.