CHAMPAIGN, Ill. – Self-administered surveys are a vital tool for researchers who gather sensitive information about adolescents. But young people who provide untruthful answers on questionnaires as pranks have the potential to throw researchers’ findings way off track, particularly studies that involve minority groups.
Educational psychologist Joseph P. Robinson-Cimpian has developed a four-step sensitivity analysis that helps researchers identify and screen out potential mischievous responders, a term he coined for pranksters who intentionally provide untruthful answers on questionnaires.
Robinson-Cimpian explained his process – and compared its efficacy to several other statistical methods – in a paper published in the May issue of the journal Educational Researcher.
Robinson-Cimpian first encountered questions about potential mischievous responders while collaborating with bullying/school violence expert Dorothy Espelage on a 2011 study. Both researchers are faculty members in the College of Education at the University of Illinois.
When a study reviewer asked the researchers how they could be certain that all of the teens who were claiming to be gay on the survey actually were gay, Robinson-Cimpian came up with the idea of using select questions on the survey as screeners.
From the data collected, Robinson-Cimpian selected several low-frequency characteristics that were conceptually unlikely to be related to respondents’ sexual orientation, but that teen pranksters might find amusing to answer untruthfully, such as reporting that they were exceptionally tall or that they had parented two or more children.
Robinson-Cimpian theorized that respondents who provided greater numbers of these low-frequency answers might be more likely to lie about their sexual orientation as well. And the final analyses supported his hypothesis: 41 percent of the respondents who reported being transgender also reported that they were unusually tall or exceptionally short.
In his latest paper, Robinson-Cimpian applied the same methodology to re-examining estimated disparities in suicidal ideation, use of cocaine/crack and school belongingness among lesbian/gay/bisexual/transgender/questioning teens, heterosexual teens and young people with physical disabilities. Recent studies have suggested that LGBTQ teens and youth with disabilities are at higher risk of suicide, substance abuse and other problems than their peers.
Robinson-Cimpian analyzed data collected from more than 11,800 students at 22 Wisconsin high schools through a Web-administered survey called the 2012 Dane County Youth Assessment.
Based upon 10 screener items from the assessment, more than 95 percent of the respondents provided fewer than two low-frequency responses, such as reporting that they were blind or exceptionally tall. Another 2 percent provided three or more of these types of responses.
But when Robinson-Cimpian examined response patterns by sexual orientation, gender identity and physical disability, he found striking differences: 11.7 percent of respondents who claimed to be LGBQ provided three or more low-frequency responses versus 1.5 percent of participants who said that they were heterosexual.
Among teens with the most extreme response patterns, 80 percent claimed to be LGBQ, 76 percent said they were transgendered and 75 percent reported that they had physical disabilities.
However, when Robinson-Cimpian looked at the teens with more moderate responses, just 1 percent said they were LGBQ and only 2 percent said they were disabled – percentages that were more in line with population estimates.
“These patterns of extreme responses are consistent with there being more potential mischievous responders among the participants who reported that they were sexual minorities and had physical disabilities,” Robinson-Cimpian said. “It doesn’t take many mischievous responders from the nonminority group to claim that they are members of a minority group to lead researchers to reach wildly incorrect estimates of between-group disparities.”
Recent studies have suggested that sexual-minority teens are at higher risk of substance abuse, suicide and other poor outcomes. And data from the full sample of the 2012 Youth Assessment suggested that more than 25 percent of the transgender teens frequently considered suicide, compared to 1.2 percent of their peers.
However, when Robinson-Cimpian screened out respondents who provided three or more low-frequency responses (less than the top 2 percent of participants), the number of transgender teens reporting frequent suicidal thoughts dramatically decreased to less than 1 percent – about the same number as their heterosexual peers.
Removing the top 2 percent of extreme responders from the data pool reduced the LGBQ-heterosexual disparity in frequent suicidal ideation by more than two-thirds and decreased the disparity between disabled-nondisabled teens by more than 80 percent.
Because some prior studies of adolescents have suggested that up to 12 percent of young people provide untruthful answers on surveys, removing the top 2 percent – or as much as the top 5 percent – of extreme responders does not seem overly conservative to ensure that data aren’t being skewed by mischievous responders, Robinson-Cimpian wrote.
“Researchers must strive for accuracy in their data, their estimates and their conclusions in order to present the public and policymakers with the most accurate and credible findings,” Robinson-Cimpian said. “By identifying likely mischievous responders and removing them from analyses, we can make progress toward these goals.”
A video of Robinson-Cimpian discussing his work is available on YouTube. The American Educational Research Association produced the video.
To contact Joseph Robinson-Cimpian, call 217-333-8527; email jpr@illinois.edu.
