Editor's Note: This is a summary of a South by Southwest talk “Getting a Crowd to Work for You: For Pay or Play?” featuring CrowdFlower founder Lukas Biewald and Kaggle founder Anthony Goldsbloom. They discussed how crowdsource work is transforming traditional business and research, provided background on their companies, and offered insights on the motivations of people involved with crowdsourcing. This post originally appeared on the CrowdFlower blog and is re-posted here with permission.

The Rise of Crowdsourcing

CrowdFlower’s company name was inspired by a 2006 article by Wired author Jeff Howe, who coined the term “crowdsourcing”. Since then, the concept has rapidly gained mainstream recognition.

These graphs below show the growth of Google searches of the term, and the rising number of academic journal articles which also reference crowdsourcing:

The crowdsourcing industry has grown in size and diversity, and now involves hundreds of companies in at least eight sub-sectors, as this chart by Crowdsourcing.org shows. CrowdFlower and Kaggle fit within the Cloud Labor segment, providing on-demand services from a virtual pool of workers. In line with broader industry trends, both CrowdFlower and Kaggle have enjoyed very strong growth in recent years: The number of judgments performed for enterprise and self-service clients by CrowdFlower’s workers is now over 300 million, and the growth of registered members on Kaggle now stands at about 30,000.

The two companies have very different approaches to cloud labor:

CrowdFlower deploys massive groups of workers to complete complex but simple jobs, while Kaggle leverages smaller (and competing) groups of specialists to solve difficult technical challenges.

More on this below:

How Kaggle Crowdsourcing Works

Kaggle’s basic operation resembles the famed Netflix competition, in which thousands of people vied to improve the video service’s search algorithm. Kaggle posts challenges to highly technical problems, and contestants around the world compete to solve them. One notable example is the search for a better algorithm to map “dark matter” in the universe. (NASA, the European Space Agency, and the Royal Astronomical Society posted this challenge on Kaggle.) To everyone’s surprise, the entry which outperformed the others came not from an astrophysicist, but Martin O’Leary, a PhD student in glaciology. (O’Leary created an algorithim similar to one used for mapping glaciers.) Thanks to Kaggle, someone from a totally different field of science was able to come forward and provide this leap of insight. Sometimes, the winning entry doesn’t even come from a scientist — in another Kaggle competition involving HIV research, the winner was an English professor.

As noted, Kaggle’s challenges are competitive. So in the weeks after O’Leary’s entry was submitted, other challengers presented their own dark matter mapping algorithms. Not to be outdone, O’Leary improved his own original entry. As the chart above shows, competition pushed subsequent entries to grow ever more accurate.

How CrowdFlower Crowdsourcing Works

CrowdFlower’s system leverages large groups of workers to solve massive but technically simple tasks. One of these services is sentiment analysis, the study of public opinion on a given subject. Other sentiment analysis programs use computational linguistics to perform this task, but computers cannot always discern nuances of tone, such as sarcasm or irony. CrowdFlower’s version is unique in that actual humans perform the sentiment analysis.

This service can be applied to a variety of jobs. For example, an academic used CrowdFlower’s self-service site to perform a sentiment analysis of Twitter messages about the weather. (See chart above.) By tracking thousands of Tweets from people expressing their feelings about the weather (positive or negative) across the country, the researcher was able to create an accurate map of climate conditions through the entire continental US.

To insure accuracy of the crowd’s work on a given job, CrowdFlower’s system also leverages the crowd. By comparing individual results with the aggregate, final results are 95% accurate. At SXSW, Biewald said that CrowdFlower is so confident of the company’s quality control, they have an open API for their crowdsourcing service.

While some CrowdFlower workers perform tasks to earn a side income, many are motivated by other goals. Notably, 50% of them come to CrowdFlower through online games, where they can earn game money and other rewards by performing crowdsource jobs.

CrowdFlower and Kaggle Crowdsource Worker Demographics

Much of CrowdFlower’s crowd workers are based in the US and India, while Kaggle has a strong presence in the US, and Australia as well (owing to its origin in that nation.) Both companies’ workforces are based in over half the countries of the world.

By gender, CrowdFlower’s work force skews heavily female. (According to Biewald, this may be because CrowdFlower jobs can be performed successfully without the worry of gender bias.) In contrast, Kaggle would make a poor dating site, since the vast majority of its competitors are male. This could be due to the over-representation of men in the sciences, Goldsbloom suggested. Some have also argued that women are generally less motivated by the kind of competition that Kaggle leverages.

By occupation and education, about 80% of Kaggle’s registered users have a technical background, such as computer science, economics, and the hard sciences.

Crowdsourcing Work Motivations on CrowdFlower and Kaggle

In two surveys, CrowdFlower workers cited multiple motivations for performing tasks on the system. While a large percent cite “money”, about as many also list “fun”. Interestingly, about 40% cite a “sense of purpose.” For CrowdFlower’s part, Biewald said he believes it’s important that crowdsourced jobs are paid for, since it indicates they are valuable, and assures the workers that the company cares enough about the task to pay for it.

In individual interviews, Kaggle competitors and CrowdFlower workers cite a number of motives: “I like it is that you are solely awarded on your quality of work. Unlike a normal work situation where your social skills and appearance greatly affect the rewards that you receive,” one CrowdFlower worker said, while a Kaggle competition winner cited his motive as: “I was just bored one Saturday afternoon.” Others mention the desire to exercise their brain, learn new things, and in the case of CrowdFlower, to earn virtual goods and currency.

Unsurprisingly, competition is also a major driver for many Kaggle competitors: “Initially it was a 'Can I do it? motivation,” as one put it. “Almost immediately, that was replaced by 'Can I do it better then others?” The prospect of winning a competition cash prizes is also a motivator. Indeed, Kaggle hopes that teams will eventually make a full-time income through their competitons.

Slides courtesy CrowdFlower and Kaggle