Tag Archives: questionaire

Research ON Research: To Catch a Cheat

High quality data is the backbone of a successful and actionable marketing research project.  Online data collection continues to be a popular way to collect data due to its cost effectiveness and ability to collect large sample sizes in a relatively short period of time.  While online data collection is a highly effective means for collecting data, a concern over data quality has emerged over the years with respondents who cheat while filling out surveys. Thus, it is important for marketing research professionals to stay on top of solutions in order to preserve data integrity.A survey cheater can be identified in many ways.  The most common types of cheating are as follows:

The Pretenders – gives false information in order to qualify for a project for which they would normally not qualify.

The Roadrunners – speeds through surveys without giving proper thought or attention to the questions being asked.

At Forbes Consulting, we have identified three ways to catch and remove survey cheaters:

  1. Speed traps:  This involves taking the average survey length and disqualifying respondents who have completed the survey in half the time.  For example, if we have a 20-minute survey and respondents complete the survey in 10 minutes or less, they would be classified as invalid completes.
  2. Attention traps:  This involves inserting a statement within a battery of attributes/statements that literally asks respondents to check a “3” on a scale.  If respondents fail to follow this directive, they would be classified as invalid completes.
  3. Consistency traps:  Here we ask similar questions at the beginning and at the end of the survey.  For example, in the screener, we ask respondents their age and in the demographic section their year of birth.  If there are discrepancies, they would be classified as invalid completes.

Moving forward, it is important for research firms and panel houses to continue to work together to eliminate cheaters.  Panel houses need to stay innovative and diligent by working together with their clients to identify and purge dishonest respondents from their panel.  Authors of surveys need to be mindful of the impact of survey length and complexity, and work to maximize respondent engagement by creating a positive survey experience.  With a team approach, we can continue to capitalize on the positive benefits of online research.

When Questionnaire Length Threatens Data Quality

When Questionnaire Length Threatens Data Quality

BACKGROUND: When surveys go on too long, respondents can get tired of answering them. They may quit part-way through or just become lazy and speed through without thinking about what is being asked. We discard data from both “drop-outs” and “streamliners,” respectively, but this adds time and money to data collection because many more quality respondents will be needed to make up the difference. The longer the questionnaire, the more drop-outs and streamliners (a.k.a. “straightliners”) result.

RESEARCH METHOD: We analyzed data collected from three questionnaires to measure the incidence of dropping-out and streamlining. Drop-out rate was defined as the percentage of respondents who didn’t complete the questionnaire. Streamline rate was defined as the percentage of respondents who engaged in any of the following: