Fighting Scientific Bias Through Crowdsourcing

“Humans desire certainty, and science infrequently provides it.”

You’ve probably seen (and even posted) these sorts of questions on social media—queries like “Does anyone near me know whether they finished the construction work at the post office yet?” or “Help me win an argument: What are the first words that come to mind when you hear the name ‘Ferris Bueller’?”

That’s crowdsourcing, of course, and it can be a great way to seek advice or take an informal poll. But can it also be used to make science better?

Raphael Silberzahn, an assistant professor at IESE Business School in Barcelona, and Eric L. Uhlmann, an associate professor of organizational behavior at the Singapore campus of INSEAD, another international business school, say yes.

Their recent article recommending that scientific research be crowdsourced was one of several approaches published by the journal Nature after an August article in Science laid bare the results of an international project that revealed a rather alarming reality: Out of 100 studies published in three respected psychology journals in 2008, scientists were able to replicate or mostly replicate the results of fewer than half. That effort was part of the Reproducibility Project: Psychology, led by University of Virginia psychologist Brian Nosek.

“The present results suggest that there is room to improve reproducibility in psychology,” the study finds. “A large portion of replications produced weaker evidence for the original findings”—despite using the same materials and methodology as the original authors.

So what’s the next step? Recommendations include the seemingly straightforward move of insisting on rigorous research standards, which is easier said than done. Researchers could more frequently leverage blind analysis, in which the scientists themselves don’t know what data the values represent until the analysis is done and the blind is lifted. But these days, a uniquely modern solution has presented itself: Crowdsourcing.

Silberzahn and Uhlmann aren’t exactly recommending that we start crowdsourcing science on Facebook or Twitter. Instead, they’re advocating that scientists and researchers crowdsource with their peers. Right now, most researchers attempt to serve as their own devil’s advocates—a single team comes up with their own findings and also tries to poke holes in them. But with human beings thrown in the mix, such a task is at best a challenging one. At worst, it veers towards the unethical.

So how would crowdsourcing as a bias-check work in practice, and what would it mean for science?

Well, let’s look at the crowdsourced experiment Silberzahn and Uhlmann conducted last year. They asked 29 teams of researchers to use the same data set to figure out whether soccer referees are more likely to give red cards to dark-skinned players than to light-skinned ones. Each team came up with its own method of analysis and had a chance to revise its analytical technique based on feedback from the other researchers.

The findings of all the research teams taken together were decidedly more tentative than the results of any one study would have been. Though there was general agreement that darker skin did result in more red cards, the extent varied widely, with findings ranging from a strong trend of dark-skinned players being more heavily penalized to a slight tendency—notably, not statistically significant—for referees to give more red cards to light-skinned players.

Basically, crowdsourcing can be expected to lead to results that might be a little less sexy, but are a lot more reliable. And, especially because this method is resource-intensive, it may perhaps best be reserved for occasions when research will likely serve as the basis of real-life policy decisions. “The transparency resulting from a crowdsourced approach should be particularly beneficial when important policy issues are at stake,” Silberzahn and Uhlmann write. “The uncertainty of scientific conclusions about, for example, the effects of the minimum wage on unemployment, and the consequences of economic austerity policies, should be investigated by crowds of researchers rather than left to single teams of analysts.”

The issue extends beyond the choice of analytical model and into actual bias, even if unintended. In the soccer example, let’s say that before embarking on the study, some of the scientists involved were expecting that referees would be harsher with darker-skinned players, while others expected race to have no bearing on what happens on the field. Couldn’t this affect how they interpreted the results, even assuming they had no conscious intention of skewing their findings?

That’s more or less what a September study on gender bias found. The study, led by Montana State University psychologist Ian M. Handley and published in Proceedings of the National Academy of Sciences, asked members of the general public as well as academics (both STEM and non-STEM) to evaluate the quality of research on gender bias by having them read either an abstract of a PNAS study from 2012 that found bias against women in the sciences—or an altered abstract that purported to find no bias. Handley’s examination of how people react to a study indicating gender bias found what appears to be, well, gender bias: Men view the findings less favorably than women, and, of greatest concern, this difference is especially prominent among male STEM faculty members.

But it’s not just racial and gender biases that can motivate scientists to distort the inferences they draw from the data. The authors note that other prejudicial factors can include the desire, whether conscious or unconscious, to support your own theory or refute someone else’s—or to be the first to report what seems to be a new phenomenon.

Maybe the core problem is that people often have unrealistic expectations of science. “Scientific progress is a cumulative process of uncertainty reduction that can only succeed if science itself remains the greatest skeptic of its explanatory claims,” write the authors of the Science article that kicked off all this important talk about increasing reproducibility and reducing bias. “Humans desire certainty, and science infrequently provides it.”

via Chela Horsdal / Twitter

Amazon's "The Man in the High Castle" debuted the first episode of its final season last week.

The show is loosely based on an alternative history novel by Philip K. Dick that postulates what would happen if Nazi Germany and the Empire of Japan controlled the United States after being victorious in World War II.

Keep Reading Show less
via Mike Mozart / Flickr

Chick-fil-A is the third-largest fast food chain in America, behind McDonald's and Starbucks, raking in over $10 billion a year.

But for years, the company has faced boycotts for supporting anti-LGBT charities, including the Salvation Army, the Fellowship of Christian Athletes, and the Paul Anderson Youth Home.

The Salvation Army faced criticism after a leader in the organization implied that gay people "deserve to die" and the company also came under fire after refusing to offer same-sex couples health insurance. But the organization swears it's evolving on such issues.

via Thomas Hawk / Flickr

The Fellowship of Christian Athletes explicitly announced it was anti gay marriage in a recent "Statement of Faith."

God instituted marriage between one man and one woman as the foundation of the family and the basic structure of human society. For this reason, we believe that marriage is exclusively the union of one man and one woman.

The Paul Anderson Youth Home teaches boys that homosexuality is wrong and that same-sex marriage is "rage against Jesus Christ and His values."

RELATED: The 1975's singer bravely kissed a man at a Dubai concert to protest anti-LGBT oppression

In 2012, Chick-fil-A's CEO, Dan Cathy, made anti same-sex marriage comments on a radio broadcast:

I think we are inviting God's judgment on our nation when we shake our fist at Him and say, "We know better than you as to what constitutes a marriage". I pray God's mercy on our generation that has such a prideful, arrogant attitude to think that we have the audacity to define what marriage is about.

But the chicken giant has now decided to change it's says its charitable donation strategy because it's bad for business...Not because being homophobic is wrong.

The company recently lost several bids to provide concessions in U.S. airports. A pop-up shop in England was told it would not be renewed after eight days following LGBTQ protests.

Chick-fil-A also has plans to expand to Boston, Massachusetts where its mayor, Thomas Menino, pledged to ban the restaurant from the city.

via Wikimedia Commons

"There's no question we know that, as we go into new markets, we need to be clear about who we are," Chick-fil-A President and Chief Operating Officer Tim Tassopoulos told Bisnow. "There are lots of articles and newscasts about Chick-fil-A, and we thought we needed to be clear about our message."

RELATED: Alan Turing will appear on the 50-pound note nearly 70 years after being persecuted for his sexuality

Instead, the Chick-fil-A Foundation plans to give $9 million to organizations that support education and fight homelessness. Which is commendable regardless of the company's troubled past.

"If Chick-Fil-A is serious about their pledge to stop holding hands with divisive anti-LGBTQ activists, then further transparency is needed regarding their deep ties to organizations like Focus on the Family, which exist purely to harm LGBTQ people and families," Drew Anderson, GLAAD's director of campaigns and rapid response, said in a statement.

Chick-fil-A's decision to back down from contributing to anti-LGBT charities shows the power that people have to fight back against companies by hitting them where it really hurts — the pocket book.

The question remains: If you previously avoided Chick-fil-A because it supported anti-LGBT organizations, is it now OK to eat there? Especially when Popeye's chicken sandwich is so good people will kill for it?


Oh, irony. You are having quite a day.

The Italian region of Veneto, which includes the city of Venice, is currently experiencing historic flooding. Venice Mayor Luigi Brugnaro has stated that the flooding is a direct result of climate change, with the tide measuring the highest level in 50 years. The city (which is actually a collection of 100 islands in a lagoon—hence its famous canal streets), is no stranger to regular flooding, but is currently on the brink of declaring a state of emergency as waters refuse to recede.

Keep Reading Show less
The Planet
via Gage Skidmore / Flickr and nrkbeta / flickr

The Southern Poverty Law Center (SPLC) dropped a bombshell on Tuesday, announcing it had over 900 emails that White House aide Stephen Miller sent to former Breitbart writer and editor Katie McHugh.

According to the SPLC, in the emails, Miller aggressively "promoted white nationalist literature, pushed racist immigration stories and obsessed over the loss of Confederate symbols after Dylann Roof's murderous rampage."

Keep Reading Show less
via Twitter / Bye,Bye Harley Davidson

The NRA likes to diminish the role that guns play in fatal shootings by saying, "Guns don't kill people, people kill people."

Which is the same logic as, "Hammers don't build roofs, people build roofs." No duh. But it'd be nearly impossible to build a roof without a hammer.

So, shouldn't the people who manufacture guns share some responsibility when they are used for the purpose they're made: killing people? Especially when the manufacturers market the weapon for that exact purpose?

Keep Reading Show less