Catching catfish

Online surveys have become a popular research tool because they are cost-effective, accessible, and capable of reaching diverse participants. However, as online data collection becomes more common, researchers must also consider potential risks to data integrity.

A recent study by Dr. Wanda Snow at The Centre for the Advancement of Teaching and Learning, published in the Journal of Computer Information Systems, examined nearly 2,000 responses to the Learner Shock survey, a community-recruited study exploring barriers to job-related learning and training. The findings revealed that 56.1 per cent of all submissions were flagged as suspected fraudulent responses.

Notably, these responses were collected through local community partner organizations, a recruitment approach often considered less vulnerable to deceptive participation than methods such as social media recruitment.

Fraudulent responses can compromise data quality, distort findings, and ultimately affect the decisions, recommendations, and policies informed by research. The study highlights the importance of incorporating data screening measures into survey design and analysis to help ensure research findings remain accurate and reliable. 

The study identified several indicators that researchers should watch for when reviewing survey data:

• Inconsistent name entries 
  Mismatched names across survey components were the strongest predictor of suspected fraud, increasing the likelihood of a response being flagged by 45 times.

   

• IP address and location discrepancies 
  Duplicate IP addresses and responses originating outside the study's eligible geographic area were common indicators of suspicious activity.

   

• Unusual email addresses 
  Email accounts containing random strings of letters and numbers frequently appeared among responses that were ultimately classified as fraudulent. 

The findings underscore the value of making data screening a standard part of the research process. By proactively identifying potential threats to data integrity, researchers can strengthen the quality of their data and increase confidence in their findings.

Read the full study here.

Preparedness for Learning (Learner Shock) was funded by the Government of Canada’s Future Skills Centre - Préparation à l’apprentissage (Le choc d’apprentissage) est financé par le Centre des Compétences futures du gouvernement du Canada.