2,364 crowdsourcing and crowdfunding sites
Back in 2010, economics and psychology Professor Joseph Henrich confirmed what many of us have suspected for a while: that students who took part in psychological studies were predominantly WEIRD.
While it is quite possible that Professor Henrich was shouting when he told people about his findings, this is not the reason we capitalize it here. WEIRD is actually an ironic acronym, standing for western, educated, industrialized, rich and democratic.
From a random sampling of psychological papers, Heinrich discovered that the average American undergrad is 4,000 times more likely to be the subject of a psychological study than anyone else. This means that any assumptions made about the whole of the human race based on the results of one of these studies may be flawed.
Just how different could American undergrads be from the rest of the population? As discussed in a recent Economist article, the answer to this question became apparent when psychologists began to use crowdsourcing to carry out their studies.
Crowdsourcing platforms such as Mechanical Turk have become increasingly popular amongst psychologists for sampling because they are cheaper and more convenient than traditional methods. It is also increasingly evident they offer a more diverse sample.
When David Rand of Harvard repeated a classic ethics experiment using Mechanical Turk, the difference became clear. It was a straightforward questionnaire that asked participants whether they would sacrifice one person (by pushing him in front of a trolley) to save many, or let fate take its course and allow the many to die.
Mechanical Turk users are not without their own biases. For one, they all use Mechanical Turk — they obviously wouldn’t be much use in a study trying to figure out why people don’t use Mechanical Turk. They are also predominantly young and have at least an undergraduate degree. They are religiously diverse, however, unlike university students, who are more likely to be atheist. This might be the reason why the Mechanical Turk users were much more likely to sacrifice one person to save many than participants in earlier studies.
Religious beliefs are not the only area where crowdworkers offer greater diversity than university students. For example, because many are located in developing nations, they also offer broader representation in economic terms. This means studies using crowdworkers are likely to generate more accurate results. As the number of crowdworkers increases, the pool will become even more diverse and useful for researchers.
In terms of reducing the time taken to carry out a study, crowdsourcing also has a lot of potential. For instance, instead of profiling volunteers within each study, crowdworkers could fill out one comprehensive profile that would apply for any study. This way researchers would be able to select an appropriately diverse population in advance.
Many enthusiastic proponents of crowdsourced research, like the people at Small Social Systems, are already hard at work making ideas like this reality. Lots of studies are also looking directly at the benefits and limitations of crowdsourcing research, such as Rand’s own paper, "The Promise of Mechanical Turk." It seems that, for behavioral studies, the future is looking a little less WEIRD and a lot more crowdsourced.
- Ville "Wili" Miettinen is the founder and CEO of Microtask. He is a serial entrepreneur and investor, with 15 years of professional experience in software engineering and computer graphics. Wili was one of the founders of Hybrid Graphics (later NVIDIA Finland), a pioneer in real-time 3D graphics technologies. He has also been involved with the evolution of various open standards. Over the last few years Wili has been active in the Finnish early-stage technology investment scene, and holds board positions in a number of companies in the industry.