Web's Largest listing of crowdsourcing and crowdfunding events

Events

Advertise

Web's Largest Directory of Sites

2,412 crowdsourcing and crowdfunding sites

Tournaments for Ideas

document Distributed Knowledge

Tournaments for Ideas By John Morgan Yahoo Research and Haas School of Business, UC Berkeley And Richard Wang Haas School of Business, UC Berkeley

Keywords: Tournaments, Strategy, Innovation, Information Technology

John Morgan is a Research Scientist at Yahoo and the Gary & Sherron Kalbach Professor of Business Administration at the Haas School of Business at UC Berkeley. Richard Wang is a Ph.D. Candidate with the Business & Public Policy Group at the Haas School of Business at UC Berkeley.

Mailing Address:

University of California Haas School of Business #1900 Berkeley, CA 94720-1900

The authors wish to thank the editor and three anonymous referees for their comments. Financial support to Richard Wang was provided by the California Management Review.

1

Abstract

Governments and foundations have successfully harnessed tournaments to spur innovation. Yet this tool is not widely used by firms. We offer a framework for managers seeking to organize tournaments for ideas. We present the theoretical underpinnings of tournaments. We then connect the theory with three recent innovations—the power of the network, the wisdom of crowds, and the power of love—that boost the effectiveness of tournaments. Short cases and academic studies are used to illustrate our framework.

2

TOURNAMENTS FOR IDEAS

During the Age of Discovery in Europe, innovations in navigation technology were of great importance in conquering the seas. In particular, a method for accurately determining the longitude of a ship’s location was needed. Sea-faring empires created Longitude Prizes to attract inventors. In 1714 the British Parliament held a tournament and offered a grand prize of £20,000 (roughly £6 million in today’s term) to the inventor who arrived at the best solution.

Over the next decades, two competing concepts emerged as most promising. In one camp, intellectual giants such as Isaac Newton and Gottfried Leibniz supported the lunar distance method; in the other, inventors like Larcum Kendall and John Harrison chased after marine chronometers. The lunar distance method required only simple measurement tools but involved complex calculations. The marine chronometer, while easy to read, was initially expensive with the cost equaling one third of a ship’s. Subsequent design breakthroughs made the marine chronometer more affordable and accurate, thus winning the hearts of ship captains. The Longitude Prize not just led to the invention of a piece of sophisticated equipment, it essentially gave the British Empire a competitive advantage in dominating the seas.1

The Longitude Prize is a classic example of a tournament for ideas—a contest designed to produce important innovations. Typically, these tournaments are proposed either by governments or non-profit foundations, such as the X Prize Foundation, and for grand innovations like space travel. But innovation need not be grand to be important and contests need

3

not be sponsored by governments or foundations to be successful. In this paper, we argue tournaments for ideas offer a powerful vehicle for firms looking to spur innovation.

USING TOURNAMENTS TO UNLOCK THE POWER OF IDEAS

The idea of using a contest to create incentives is a familiar one in business. Firms regularly organize tournaments based on sales performance. Even CEO compensation is sometimes tied to firm performance against industry benchmarks—effectively a tournament against rivals in the same industry. Tournaments work well in cases where measuring relative performance is easier than measuring absolutes. Indeed, economic theory tells us that, in these circumstances, tournaments are often the incentive strategy.2 Tournaments systems are also used in setting career milestones. For instance, GE’s famous 70-20-10 employee performance evaluation system is recognizably a tournament. Likewise partnerships with “up or out” systems such as law firms, consulting firms, and academia may be thought of as tournaments.

Ideas are similarly difficult to measure on an absolute scale. While one can perhaps distinguish between better and worse ideas, delineating a bright line standard for a “good” idea is often difficult. Idea generation is at least as critical to long-term firm success as sales or promoting the right people. Yet despite this, it is still uncommon for firms to use tournaments when seeking to innovate.

This article explains how (and why) firms can use tournaments to power innovation. We start with the theory of tournaments, including their strengths and weaknesses. We then connect

4

the theory with three recent innovations—the power of the network, the wisdom of the crowd, and the power of love—the boost the effectiveness of tournaments. Throughout, we use short cases and academic studies to illustrate our framework. We close with a how-to guide for managers seeking to apply the ideas in the article.

Tournaments as Incentive Mechanisms Tournaments are reward structures where compensation is based on relative rank as opposed to absolute levels of output.3 Economists have developed theoretical models to examine the performance of tournaments compared to other compensation schemes. These models show that an appropriately structured tournament does at least as well as traditional contracts based on piece-rate outputs.4 Tournaments are especially effective in situations where (1) efforts are difficult to monitor; (2) performances are observable but difficult to judge in absolute terms; and (3) performances are subjected to common underlying and unobservable shocks.5

To see why, it is useful to visit Staten Island on a cold November morning for the start of the New York City Marathon. While a runner’s performance (i.e. the time it takes to complete 26.22 miles) is easily measured, the ingredients leading to this performance—the strain and effort undertaken in the race, the countless hours of training leading up to the event—are not. But it is these latter ingredients that make the race such a compelling event. Moreover, even when we have perfect measurement of the finishing time we might still wonder if the runner’s performance is the best it could be. For example, a runner finishing in two hours and eight minutes claimed the first place finish in the 2008 New York City Marathon. However, the same duration of two hours and eight minutes is only good enough for a second place finish at the

5

Boston Marathon in the same year.6 Furthermore, all the runners in a race are subjected to the same factors, such as the route and weather conditions, that could affect performance. The tournament filters out the common underlying uncertainties to reward the best athlete. At the same time, contestants are still motivated to put in their best efforts regardless of the elements.

But what does this have to do with generating ideas? The three features where tournaments have the advantage in theory are all strongly present in incentivizing ideas and innovation. To start with, the “effort” input is extremely difficult to monitor—it is difficult to distinguish someone staring out the window while thinking about the next great idea from someone staring out the window while daydreaming. Judging on the basis of idea output is likewise difficult or impractical. Unlike the finishing time in a marathon, the ultimate value of an idea, like the marine chronometer, is often not known until years later. Similarly, common shocks apply to ideas generation as well as marathon running. For instance, advances in computer technology have transformed “impossible” to solve mathematical problems, like the four-color theorem, into easy problems. These features all point to the appeal of tournaments as a means of incentivizing innovation.

Designing Tournaments for Ideas Tournaments come in many formats with variations in prize structures, number of rounds, handicapping systems and entry restrictions, to name a few. These features help us to fine tune the incentive structure in order to accommodate for different scenarios. In the following we focus on a few common situations and discuss how we could make use of these tournament design features.

6

Adapting to the Nature of Ideas Depending on the nature of the problems to be solved, we can customize tournaments to better incent efforts from the contestants. There are times when we look for revolutionary ideas, say we want to design a green car that achieves a hundred miles per gallon, or evolutionary ideas, such as raising the fuel efficiency of an existing car model by ten percent. To arrive at a revolutionary idea, it may require the contestants to formulate new paradigms which entail intensive R&D and integration efforts. The probability of arriving at a revolutionary idea therefore hinges on the peak level of efforts by the contestants. Evolutionary ideas, such as enhancing fuel efficiency through improving aerodynamics and reducing vehicle weight, are more often diversified, modular and involve moderate level of efforts by each contestant. Successful evolutionary ideas are therefore more dependent on aggregating a large pool of contestant idea contributions and efforts.

Economic theory tells us that, when seeking revolutionary ideas, we should employ high powered incentives in order to motivate intensive efforts from our agents. Besides offering greater prize money, there are other ways to sharpen incentives. Theory suggests that limiting competition is often an effective strategy. By restricting entry, each contestant perceives a higher chance of winning the contest and raises effort accordingly. While the total “bandwidth” devoted to the problem may be lower, peak bandwidth is higher and hence truly revolutionary ideas are more likely to be generated.

7

For generating evolutionary innovations, maximizing total bandwidth devoted to the problem is crucial. Here, theory suggests the opposite approach, making the tournament open to all comers is the appropriate. 7

The prize structure is another key lever. The two most common prize structures are winner-take-all and multiple prizes. The right prize structure depends on the type of innovation and the variance in the ability of the contestants. In winner-take-all tournaments, only contestants who believe they have the absolute best idea will participate. This deters entry by potential contestants who are not completely confident about their ideas. Awarding multiple prizes invites participation from contestants who believe they have workable ideas even though they see their ideas might not be the absolute best but at the cost of reducing incentives for the very best ideas. The latter approach could help increase the quantity of ideas received which is more suitable for generating evolutionary ideas.

Motivating Contestants with Different Level of Abilities Incentive problems arise when contestants of various abilities compete in a tournament. Weak contestants are unlikely to contribute much to the tournament outcomes but they dilute the probability of talented contestants winning the tournament. This in turn lowers peak efforts from the strong contestants. At the same time, when weaker contestants find out that they are competing with a stronger contestant, they would perceive their chance of winning the prize vastly diminished and therefore become discouraged to put in efforts. For example, it has been found that professional golfers score nearly one stroke higher (worse) than they normally do when competing in tournaments which Tiger Woods (a superstar in the game of golf) is present.8

8

There are several ways to mitigate this type of incentive problem. Besides restricting entry, the host can organize multiple-rounds tournaments with each subsequent round offering a larger prize. Multiple-rounds tournaments eliminate weaker contestants in earlier rounds and save larger prizes for the final rounds to keep talented contestants motivated. Tournament organizers sometimes institute handicapping systems or group their contestants by talent levels (e.g. junior category, amateur category, professional category, etc.) to level out the playing field. Under these arrangements, greater total effort could be achieved but they are unlikely to raise peak efforts.9

Understanding When Not to Use Tournaments To be sure, there are situations in which tournaments do not work well. In some cases, tournaments undercut workers’ incentives to cooperate. For example, a survey of over 800 employees in Australian firms reveals that they are less likely to work cooperatively – such as sharing of production tools – under tournament incentive systems.10 In other cases, tournaments could tempt contestants to collude, such as in competitive bidding process. Generally speaking, tournaments do not work well in situations where the participants’ performances are interdependent.

9

INTEGRATING NEW CONCEPTS TO TOURNAMENTS FOR IDEAS

In the following we present how organizations are integrating tournaments for ideas with three emerging business concepts – hosting tournaments for ideas on platforms, leveraging social acceptance as a non-pecuniary motivator and democratizing the idea generation process.

1. Platforms as Ecosystems for Ideas When you have a question and know who has the answer, you would probably go straight to the person and ask. Similarly, traditional tournaments organizers have some ideas on whom and where the experts are and therefore would promote the tournaments in the corresponding channels. In the past when search and promotion costs are high, this unidirectional questionerseeking-expert approach might have been an efficient way to organize tournaments. Information technology has markedly reduced these costs and with it we want to revisit, or perhaps even improve, the way tournament organizers interact with target experts. In this section we focus on how tournaments for ideas, when coupled with platforms and network effects, have created a market ecosystem that fosters the generation and exchange of ideas.

Platform and Network Concepts In markets for ideas there are two parties: the questioner and the answerer. While in many markets the two parties can trade directly, Economics Nobel Laureate Kenneth Arrow has pointed out that this may not be the case in market for ideas. The reason, now commonly referred to as the information paradox, is that the questioner does not know the true value of the idea ex

10

ante unless answerer reveals the idea. But once the idea is revealed, the questioner could behave opportunistically and pay little, if any at all, to the answerer.11 An intermediary agent such as a platform can alleviate this type of market failure caused by the information paradox.

Platforms exist to serve multi-sided markets with two or more distinct groups of customers who value each other’s participation. Examples of platforms include credit cards, advertiser-support media, video game consoles, shopping malls, and e-business portals such as Amazon.com. While the primary purpose of a platform is to connect customers from both sides, in market for ideas, a platform not only connects questioners and answerers but also take on the responsibility of protecting the ownership as well as verifying the validity of the idea. Yet for any platform to become truly effective, the key to success hinges on what economists refer to as network effects.

Raised to prominence by UC Berkeley professors Michael Katz and Karl Shapiro, the theory of network effects describes how a product, such as a telephone, has little value when existing in isolation but the value grows exponentially as it becomes one of many connected in a network.12 Similarly, a platform is worthless when few customers join in. However, as more customers join in, the value for the next potential customer to join the platform increases. Next we will look at how Innocentive, by hosting a tournament for ideas on a platform and leveraging on network effects, enhanced the probability of matching questions with answers.

11

Case: Innocentive Challenges Originally founded as an e-business venture of Eli Lilly in 2001, Innocentive is now operating independently as a neutral online marketplace for ideas. Organizations with difficult to tackle questions can register as seekers with Innocentive which posts the questions online as “challenges”. The challenges fall into four categories – Ideation, Theoretical, Reduction to Practice (RTP) and Request for Proposals (eRFP). In Ideation, seekers invite solvers to brainstorm for ideas and submit them in writing that are typically under two pages in length. Winners of Ideations grant seekers non-exclusive license to use the ideas. Theoretical challenges require solvers to submit thorough solutions and, if chosen by the seekers, solvers will formally transfer the intellectual property rights. RTP takes the requirement one step further than Theoretical challenges by requesting the solvers to present, in addition to detail description of the ideas, physical evidences demonstrating their ideas are the best among all submissions. The fourth type of challenge, eRFP, permits greater interactions between seekers and solvers. In eRFP, solvers are not asked to reveal confidential details about their ideas in their initial submissions. Instead, seekers can selectively get in touch with solvers to negotiate specific contracts. Some recent challenges are presented in Table 1.

[Insert Table 1 about here.]

The tournament format is ideal for Innocentive. Tournament reward schemes eliminate the need for seekers to monitor efforts of solvers. Moreover, seekers can evaluate the ideas by simply benchmarking them against one another without needing to worry about environmental

12

uncertainties. The winner-take-all prize structure offers solvers high powered incentives to take on difficult challenges. The amounts of prize money, ranging from five thousand to a million dollars, usually commensurate with expected effort levels – prizes offered for RTP are typically greatest since these types of challenges requires most efforts, while prizes for Ideation challenges are usually smallest as seekers requests for early stage ideas (see Table 1). Nevertheless, Ideation usually generates large participations despite of small prizes because the effort needed to produce a two-page report is comparatively little. Ideation therefore offers seekers an economical way to collect vast numbers of ideas. Finally, while challenges in the Ideation, Theoretical and RTP categories are open to all solvers, the eRTP format provides seekers with flexibilities in screening solver participations.

Answers from Out of Left Field With 160,000 solvers registered with Innocentive, seekers frequently receive solutions from half way around the world. For example, in a typical challenge you can find a Russian scientist, a retired head of Hoechst R&D, a Chief Research Officer from Northern Ireland, and a head of an Indian research institute competing for a $25,000 prize in finding a synthetic strategy for a chemical compound.13

More interestingly, it is quite common to have winning solutions generated by solvers whose expertise lies outside of the seekers’ domain. The New York Times reported a story on John Davis, an Innocentive solver and chemist from Illinois who had some experience pouring concrete. Davis won $20,000 from the Oil Spill Recovery Institute (OSRI) of Cordova in Alaska by offering a solution to keep oil from freezing. The idea was simple and widely known within

13

the cement industry – concrete will not set if it is kept vibrated. Scott Pegau, the research manager at OSRI, was impressed by Davis’ solution. “The oil-flow problem was solved by an outsider,” said Pegau. “If it could easily have been solved by people within the industry, it would have been,” he added.14 The story of John Davis underscores a fundamental problem with the top-down, unidirectional expert search associated with traditional tournament for ideas – the organizers don’t know who has the best answer to their questions. By reaching only to experts whom they think might possess answers, tournament organizers are limiting their chance of finding the best solutions.

Researchers at Harvard have investigated into the expert search problem. After examining all cases from Innocentive between 2001 and 2004, Professor Karim Lakhani finds that many problems that were unable to be solved by experienced corporate researchers were cracked by outsiders. In particular, Lakhani concludes that a more diverse problem-solving population will lead to greater likeliness that the problem will be solved.15 Here, the platform technology frees up tournament organizers and enables them to economically tap into idea sources from beyond their traditional domains. Outside experts could then arbitrage their specialized knowledge developed for other purposes to solve the problems. But to be sure, reducing information dissemination cost for tournament organizers is only half the story. From the outside experts’ point of view, those who are confident that their ideas are valuable and want to find questions to match them with can now do so on the Internet.

14

Look Who’s Searching – Experts Hunting for Questions Ed Melcarek from Ontario is a case in point. Melcarek has a Ph.D. in Engineering Science and years of design experience ranging from conventional heating vents for buildings to spray-paint robots to working in a world-class particle accelerator laboratory. But Melcarek had a hard time landing on a corporate R&D job. “Too diverse a set of skills and experience” was the most common rejection answer he got at interviews. Then one day Melcarek came across Innocentive and saw a problem posted by Colgate-Palmolive. The question was about injecting fluoride powder into a toothpaste tube without dispersing into surrounding air. “It was really a very simple solution,” commented Melcarek. He used his particle accelerator experience and submitted an answer. The $25,000 prize money he won, as Melcarek recalled, not only saved him from the welfare office but also “re-affirmed his confidence in himself”. Now a six-time Innocentive challenge winner, Melcarek continues to look for challenges where he can turn his ideas into paychecks.16

Leveraging on Network Effects With the emergence of companies such as Innocentive as well as a new breed of experts like Ed Melcarek, the Internet has created platforms for ideas where both questioners and experts actively search for one another.17 By transforming the unidirectional search mode in traditional tournaments to a two-way matching process, the platform for ideas model has improved both the problem solving rates as well as the caliber of the solutions generated. This is consistent with network economic theory predictions – as more experts succeed in solving problems posted on the platform, seekers become more inclined to post their problems, which in turn attracts greater

15

number of experts and hence raise the probability of successful problem-solution matching. Network effects are fueling the growth of the platform.

Professor Lakhani and Jill Panetta, a co-founder and former chief science officer at Innocentive, offer an assessment on how the platform model is living up to the reputation: On average, a problem posted on the Innocentive platforms receives detail attention from more than 200 potential solvers, from which about ten will submit solutions. These statistics constitute to Innocentive’s remarkable achievement of thirty percent problem solving rate.18

Concerns over Secrecy Platforms like Innocentive are not the right forum for all types of innovations. In particular, the information paradox is a problem not just vertically—between questioner and answerer—but horizontally as well. In some industries, it may be enough to know the question posed by a rival, even if the answer is kept from view, to gain a competitive advantage. For instance, a pharmaceutical company posting a question detailing an intravenous delivery method of a pediatric protein-based drug for FDA clinical trial could potentially disclose strategic information to its closest competitors. While there are ways to mitigate this information leakage, this possibility suggests the need for some care on the part of managers in determining whether the platform approach offers the right fit. For larger organizations, one alternative is to reinvent the platform in-house—away from the prying eyes of the competition.

16

The Platform Ecosystem While traditional tournaments for ideas use a top-down approach—a question or challenge is posed upstream seeking answers downstream—the network economics of platforms are capable of transforming the process to a bi-directional, questioner and answerer side-by-side process. With this transformation, synergistic use of existing ideas becomes possible and hidden expertise can be unlocked.

2. Optimal Prize Structure: The Importance of Social Motivators Economics Nobel Laureate John Harsanyi once said “People’s behavior can largely be explained in terms of two dominant interests: economic gain and social acceptance.”19 Simply put, most of our actions are motivated by love and money. While tournaments such as the Longitude Prize and the ones taking place at Innocentive are driven by pecuniary motives, tournaments powered by love can be an attractive alternative. In this section, we will first look at a study on how a form of social acceptance – reciprocity – can unlock hidden value. Next we turn to Flickr, the largest photo image hosting site on the Internet, to see how they leverage social forces to power a large-scale tournament.

Measuring the Returns to “Love” Reciprocity is one of the most common forms of expressing social acceptance. Reciprocity happens all around us: We tip more generously in restaurants when served by smiling waiters. Free samples often lead to purchases in supermarkets. In short, we repay

17

kindness with kindness.20 But how powerful is the force of reciprocity in economic life? To try to quantify the returns to kindness, we turn to controlled laboratory experiments.

To examine this question, some Swiss researchers asked the simple question: do employers benefit by offering above-market wages? In other words, do employees pay back the “gifts” of their employers with additional effort? Is this enough to make such gifts profitable in the first place?21 In the experiment, “employers” were asked to offer salaries and state their desired level of efforts to “employees.” The suggested effort is non-binding—employees are free to work as much or as little as they like while still receiving the contracted wage. The profits to “employers” depends on the (costly) effort of their “employees.” If employees were solely motivated by pecuniary considerations, all wage offers would lead to the outcome—employees would choose the lowest possible effort level. Anticipating this, employers would optimally offer the lowest possible wage and the returns to kindness would be negative. If reciprocity is an important consideration, however, smart employers could benefit by offering generous wage packets knowing that these would be repaid with employee effort.

[Insert Figures 2 and 3 about here]

Even in the stark and impersonal laboratory setting, the researchers found strong evidence of reciprocity. The results are presented in Figures 1 to 3. Figure 1 illustrates that “employees” reciprocate to “employers” generous salary offers. Employees receiving only the minimum salary gave zero effort back to their employers. Increasing the wage offer produced

18

systematically more effort. Figure 2 illustrates the profitability of this strategy. A 33% increase in wages at the lowest salary brackets produced 20% higher profits. ROI becomes even more favorable at higher brackets. Between the top second and third salary brackets, an increase of 16% in salary boosted “employer” profits by 22%. Nevertheless, the researchers noted that there are limits to the power of love. Figure 4 recasts the “employer” profits in terms of percentage increase. While percentage increases in profits are positive for most wage brackets, the returns to offering the highest wage bracket turn negative. In other words, the law of diminishing returns applies also to “love” as the additional income gained by “employee” efforts is insufficient to compensate for the higher wage outlays.

[Insert Figure 4 about here]

Economists and businesses are increasingly recognizing the power of non-pecuiary incentives in motivating employees. This is especially true when it comes to generating ideas. For instance, Wikipedia owes its success almost entirely to the power of these incentives. But what happens when we harness non-pecuniary incentives in a tournament-like structure. To examine this, we turn to the case of Flickr.

Case: Powering Tournaments with “Love” at Flickr Consistently ranked as a top ten Web 2.0 site, Flickr receives over 20 million unique visitors each month. By November 2008, over 3 billion photos were posted to the site.22 While Flickr is useful for organizing personal photos, so are many of its rivals. Instead, Flickr owes its success to the social nature of its site. Many users post there to share their “best shots” with the

19

broader online community. In fact, over 80 percent of the photos are available for public viewing.23 The high quality of many Flickr photos has led firms to look there, rather than at traditional stock photo repositories, for images to be used in promotional material and elsewhere.

Flickr started as a digital photo storage and sharing company in 2004 and was acquired by Yahoo for an estimated $35 million a year later. When asked why they became interested in Flickr, then Yahoo executive Bradley Horowitz explained:

“With less than 10 people on the payroll, they had millions of users generating content, millions of users organizing that content for them, tens of thousands of users distributing that across the Internet, and thousands of people not on the payroll actually building the thing. That's a neat trick. If we could do that same thing with Yahoo, and take our halfbillion user base and achieve the same kind of effect, we knew we were on to something.”24

Flickr is, in effect, a platform for ideas, only here the ideas are the photos. Searchers visit Flickr to see high quality photos while photographers post their works on Flickr to gain appreciation, recognition, and, in some cases, money. As with Innocentive, network effects play an important role—the platform becomes more valuable to both parties as it scales up. But rival sites have the potential for the same network effects. What makes Flickr stand out?

Flickr provides no direct monetary rewards to photographers posting there. Indeed, the opposite is true, those who want to post more than a few images must pay Flickr for the privilege.

20

Likewise, Flickr does not charge browsers to visit the site. In many cases, browsers do not even have to pay to download photos posted by Flickr users. Instead, Flickr owes its success to its tournament for ideas fueled by social acceptance.

The tournament, in this case, is Flickr’s Explore page. Multiple times per day, Flickr combs through the billions of photographs on its and selects the 500 it considers to be the best. These are displayed prominently and receive disproportionately many views from browsers. Essentially, Explore is continuously running photo contest. For photographers, having a photo featured in Explore is an important recognition of excellence—one that generates wide recognition and occasionally commercial opportunities.

But how does Flickr select the best photos? While the exact algorithm is proprietary, Flickr acknowledges that the social recognition of other users plays a key role. Specifically, photos posted to Flickr can receive comments from other users. The more positive comments a photo receives the higher the photo will be ranked. Often, these comments come from Flickr groups, which are powered by reciprocity—those posting to a group are expected to offer comments to other worthy photos.

The 11,000-member Flickr group, “All You Need is Love”, is a case in point. 25 The group has voluntarily established a reciprocity rule that requests members to express “love” to three fellow members’ photos whenever they post a photo of their own. In the words of the group administrator, the reason behind this social institution is because “[t]here's an awful lot of negativity among Flickrites. It's time we build one another up instead of tear one another down.”

21

This type of “pay it forward” reciprocity, analogous to “employers” offering generous salaries to “employees” in the reciprocity experiment, generates abundant circulation of “love” within Flickr. “Love” is quite probably what makes Flickr’s world go ’round.

Reciprocity works locally as well. Flickr users can connect to each other through a contact system. Photos of user’s contacts are displayed prominently on the page and contacts often comment on each other’s photos—again the power of reciprocity.

With an ever expanding portfolio, the task of organizing and evaluating photos has become enormous for Flickr. On one of Flickr’s lead webpages they state, “Flickr labs have been hard at work creating a way to show you some of the most awesome content on Flickr.”26 Indeed, Flickr is pioneering a way to discern photo qualities with the help of user participation – a tournament for ideas powered by social acceptance.

Harnessing the Power of Social Acceptance While Flickr demonstrates the possibility of using social incentives to power tournaments, the practical lesson for managers is not to neglect non-pecuniary motives in contest design. Creating avenues for “peer review” of ideas in the form of social acceptance, creating opportunities for “answerers” to gain recognition beyond simply cash payments, and making the pecuniary component “generous” (but not necessarily unprofitable) are all ways to build more powerful and cost-effective tournaments for ideas.

22

3. The Democratization of Ideas Both Innocentive and Flickr benefit from participation by millions of non-experts. Flickr posters are, for the most part, enthusiastic amateur photographers rather than seasoned professionals. Answerers on Innocentive come from a wide array of fields and often have no particular connection or expertise with the industry of the seeking firm. This inspires to our next question: When you look for a solution to a problem, would you rather seek help from an expert, or from a large group of ordinary people?

An ancient Chinese proverb says: “Three smelly leather shoemakers put together can be counted as a Zhuge Liang.” A statesman and military strategist from the Three Kingdoms period of China (AD 220-280), Zhuge Liang is widely recognized by the Chinese as a mastermind with the wisdom exceeds that of the wisest person. In recent years, this old wisdom of Chinese proverb is being embraced by business strategists. For example, perhaps learning the “neat trick” from Flickr, Yahoo decided to tap into the crowds when they set up the Yahoo Answers platform in 2005.

Case: Yahoo Answers An online platform where askers post their questions, Yahoo Answers is organized as a tournament where volunteer answerers compete to offer solutions to be voted upon by users. The best answers chosen are displayed prominently and the winning answerers are awarded recognition points. A vast number of questions have been answered by the online community using this tournament system. Within a half year, Yahoo Answers became the second most

23

popular Internet reference site after Wikipedia. And by their first birthday, Yahoo Answers surpassed 60 million users and 160 million answers.27

Launched three years before Yahoo Answers, Google Answers had a substantial head start in the online knowledge market. The conventional wisdom suggests that this head-start, combined with Google’s proven business acumen, should have proved an insurmountable barrier for Yahoo. While both Answers platforms share similarities, such as taking advantage of the Internet to lower communication and administration costs, Google decided to pursue the expert route – at one time, over 500 experts were selected by Google to answer questions. With hundreds of selected experts available, one would expect better quality answers from Google than Yahoo. Surprisingly, Google Answers succumbed to the competition from Yahoo. In December 2006, Google shutdown the service, citing the Answers community's limited size as a reason for pulling the plug.

Democratization of Ideas Google Answers was built on the conventional wisdom that expert answers are the only valid ones. Google was not alone about their belief in experts. Governments in the past had doubts in giving democracy to ordinary citizens – they were concerned that decisions by commoners might be inferior to those made by elite aristocrats.

For example, at the turn of the 20th century, British scientist Sir Francis Galton was skeptical about the wisdom of ordinary citizens. To prove his point, Galton visited a county fair in 1906 where a weight-judging competition was underway. In hope of winning the prize money,

24

participants paid an entry fee of six pennies to submit guesses on the weight of an ox. Galton compared the competition to democratic voting since many of the fairgoers lacked expertise with livestock and meat processing just as most voters lacked knowledge on government policies. Eager to show that the “average voter” was incapable of making good decisions, Galton borrowed the entire lot of entries from the organizer after the competition. Of the 787 valid guesses, Galton found none has gotten the exact correct answer of 1,198 pounds. But to his surprise, the median value of all guesses came out to be 1,207 pound, which was off by less than 0.8 percent of the true weight! “This result is, I think, more creditable to the trustworthiness of a democratic judgment than might have been expected,” concluded Galton.28

Galton’s experiment has been credited as the first mathematical demonstration of the “wisdom of crowds.”29 Popularized by James Suroweicki’s 2004 bestseller of the same title, the wisdom of crowds proposes that ideas gathered from many ordinary people are at least as good as or even better than those proposed by the experts.30 Yahoo Answers is a modern day demonstration of the concept. By scaling up the collection of wits from three cobblers to include millions of minds from around the world, Yahoo Answers operates on the likelihood that someone somewhere possesses the exact solution to the question, or by accumulating bits of ideas from a vast pool of people, the final answer will approach the best.

Harnessing the Wisdom of Crowds: Is it Compatible with Tournaments for Ideas? While the wisdom of crowds potentially adds an appealing element to traditional tournament of ideas, we need to be aware of the compatibility between the two concepts. Note that the key to achieving the wisdom of crowds is by statistically processing information from

25

many people, each offering a bit of knowledge, albeit incomplete, of the solution. Though it can seem magical, the wisdom of crowds can only extract information when there is information from the crowds to be extracted. For example, in Galton’s weight-judging competition, the participants were motivated to contribute in their honest best guess because they each paid an entry fee and aimed to win the tournament prize. As the number of participants grew, on one hand, the probability of learning the true answer improved, but on the other, the chance of any one contestant winning the tournament prize decreased. Recognizing that their chance of winning was diminishing, contestants might casually throw in their guesses instead of diligently making good effort answers. Random guesses from the crowd, each containing virtually no useful information, could cripple the wisdom of crowds.

To avoid stalling the wisdom of crowds, Yahoo Answers turns to another Flickr “neat trick” – using non-pecuniary rewards to motivate contestants. Yahoo leverages on what cognitive psychologists call intrinsic motivation of activities – that every activity has a motivation of its own. This is evident in the way Yahoo brands the platform as “a whole new kind of volunteerism” and persuading the users with phrases such as “You make someone's day each time you reply” and “You share your intelligence for a good cause” to boost the intrinsic motivation of the tournament.31 As intrinsic motivations are independent of pecuniary rewards, the incentive level of the tournament is preserved even when the number of participants expands. The advantage of using intrinsic motivation goes beyond economizing the prize money. Most importantly, Yahoo has the answer to uniting the wisdom of crowds and tournaments for ideas.

26

Fusing the Wisdom of Crowds and Tournaments for Ideas Since the publication of Surowiecki’s book, the idea of “crowd-sourcing” has gained much currency among managers. However, indiscriminate application of this idea is unlikely to produce useful results. The lessons from Yahoo Answers and the tournament theory suggest three key lessons. First, crowdsourcing works when what matters is the collection of answers rather than the single best answer; that is, when small bits of information can be combined together in a useful way. For evolutionary innovations, this is often the case. For those seeking revolutionary innovations, it rarely is. Second, crowd-sourcing works best when there is a competition. Market surveys, which may be thought of as an early version of crowd-sourcing, are capable or producing great results or disastrous ones (e.g., New Coke). Pairing the crowdsourcing model in a tournament setting is much more likely to produce incentives for useful answers. Finally, users are drawn to crowd-sourcing models under a variety of motives. Some are motivated by money; some by recognition; some simply want to “make a difference.” Tournaments should be designed to offer rewards for these varied incentives.

A FRAMEWORK FOR ORGANIZING TOURNAMENTS FOR IDEAS

We close by offering a framework for managers seeking to incorporate the concepts presented here into their own tournaments for ideas. Figure 5 presents a flowchart illustrating the key decision milestones.

27

Several key features are worth noting. First, it is critical to understand what type of innovation is to be sought. Typically many inputs can be combined in producing evolutionary innovations. The tournament structure should reflect this. It should be open, take advantage of scale economies of the network, and harness non-pecuniary as well as pecuniary incentives. Revolutionary innovations, however, rely on what statisticians refer to as the “order statistic”— the quality of the best idea. Here, the appropriate framework may be closed and high-powered.

Ability differences among participants are another important dimension to consider. In a closed tournament, these ability differences can undermine the incentives to produce great ideas. The correct solution is to “level the playing field.” This can be done by combining weaker individuals into teams or separating individuals into different tiers by ability level.

Secrecy and the information paradox play an important role. Creating safeguards so that sensitive information is not disclosed in the questions themselves, and so that answerers do not feel at risk of being expropriated are obviously critical to the success of the tournament.

Last, it is important to recognize the many motives of individuals to contribute their ideas. Creativity is a fundamental human attribute. While money is one motive to induce the creation of ideas, there are many others including fame, self-satisfaction, love of competition, curiosityseeking, recognition from peers, and so on. Tapping these varied motives is crucial in designing a successful tournament for ideas.

28

Table 1. Examples of Innocentive Challenges

Challenge Type Title Improving Banking Processes in the Developing World Ideation

Number of Participation 641

Prize $7,500

Extending Shelf Life of a Microbiological Product

Theoretical

427

$20,000

Plastic with Property of Glass

RTP

81

$50,000

Corrosion Resistant Nylon

eRTP

54

Varies

Examples of Seeker Questions and Prizes on Innocentive Marketplace32

29

Figure 1. A Decision Tree for Designing Tournaments for Ideas
Tournaments - Offer High Powered Incentives - Charge Entry Fee - Establish Entry Criteria - Organize Multiple Rounds

Yes

Yes

Do potential contestants’ ability vary by much?

No

Tournaments - Offer High Powered Incentives - Winner-take-all Prize

No

Does the problem require a revolutionary idea? Tournaments - Offer Flattened Incentive Structure - Set up Handicapping Systems - Group contestants by categories

Yes Are the contestants’ performances interdependent?

No Do potential contestants’ ability vary by much?

No Tournaments - Offer Flattened Incentive Structure - Open Entry

Yes Tournaments not appropriate

30

Figure 2. Effects of Wage Offer on Employee Effort Level
33

0.6 0.5

0.4

Effort

0.3 0.2 0.1

20

30

40

50

60

70

80

90

Wage

31

Figure 3. Effects of Wage Offer on Employer Profit34
25

20

15

Profit
10 5 0 20 t o 25 26 t o 35 36 t o 45 46 t o 55 56 t o 65 66 t o 75 > 76

Wage Interval

Figure 4. Percentage Increase in Profits with Wage Offer35
40

30

20
Percentage Increase in Profit

10

0 Low -10 Medium-Low Medium Medium-High High Very High

-20

-30

-40 Wage Offer Interval

32

Figure 5. A Decision Tree for Designing Tournaments for Ideas in the Internet Era
Tournaments - Organize own tournament - Offer high powered incentives - Entry by invitation only Tournaments - Use a tournament platform - Offer high powered incentives - Charge entry fee - Establish entry criteria - Organize multiple rounds Tournaments - Organize own tournament - Offer high powered incentives - Winner-take-all prize

Yes Is secrecy of the problem important? No

Yes

Yes

Do potential contestants’ ability vary by much? Yes No Is secrecy of the problem important? No

No

Does the problem require a revolutionary idea? Yes Is secrecy of the problem important? No Do potential contestants’ ability vary by much?

Tournaments - Use a tournament platform - Offer high powered incentives - Winner-take-all prize Tournaments - Organize own tournament - Offer flattened incentive structure - Entry by invitation only - Leverage intrinsic motivators Tournaments - Use a tournament platform - Offer flattened incentive structure - Organize multiple rounds - Leverage social motivators Tournaments - Organize own tournament - Offer flattened incentive structure - Leverage intrinsic motivators

Yes Are the contestants’ performances interdependent? No

Yes No Is secrecy of the problem important? Yes Tournaments not appropriate No

33

Tournaments - Use a tournament platform - Offer flattened incentive structure - Leverage social motivators

Notes
1

For background of the British Longitude Prize, see D. Sobel, Longitude: the true story of a lone genius who solved the greatest scientific problem of his time (New York: Penguin Group, 1996) and N. Kollerstrom, Newton’s Forgotten Lunar Theory, His Contribution to the Quest for Longitude (Santa Fe, New Mexico: Green Lion Press, 2000). 2 See, for example, E. Lazear and S. Rosen (1981) “Rank-Order Tournaments as Optimum Labor Contracts” Journal of Political Economy, 89(5): pp. 841-864. 3 This definition is also used by J. Green and N. Stokey (1983) “A comparison of tournaments and contracts” Journal of Political Economy, 91(3): pp. 349-364. 4 Lazear and Rosen (1981), op. cit. 5 Green and Stokey (1983), op. cit. 6 Marilson Gomes Dos Santos of Brazil was the winner for the 2008 New York Marathon, completing the race in 2:08:43. Robert K. Cheruiyot of Kenya won the 2008 Boston Marathon with a time of 2:07:46 while Abderrahime Bouramdane of Morocco held the second place finish with a time of 2:09:04. 7 K. Konrad (2009) Strategyand Dynamics in Contests Oxford University Press: Oxford 8 J. Brown (2009) “Quitters Never Win: The (Adverse) Incentive Effect of Competing with Superstars” Northwestern University Working Paper. 9 For the theoretical details, refer to M. O’Keefe, W.K. Viscusi and R. Zeckhauser (1984) “Economic Contests: Comparative Reward Schemes” Journal of Labor Economics, 2(1): pp. 27-56. 10 Robert Drago and Gerald T. Garvey, “Incentives for Helping on the Job: Theory and Evidence,” Journal of Labor Economics, Vol. 16, No. 1., (1998): pp. 1-25. 11 Kenneth Arrow (1959) Economic Welfare and the Allocation of Resources for Invention, Rand Corporation P1856-RC. “[T]here is a fundamental paradox in the determination of demand for information; its value for the purchaser is not known until he has the information, but then he has in effect acquired it without cost. Of course, if the seller can retain property rights in the use of the information, this would be no problem, but given incomplete appropriability, the potential buyer will base his decision to purchase information on less than optimal criteria. He may act, for example, on the average value of information in that class as revealed by past experience. If any particular item of information has differing values for different economic agents, this procedure will lead both to a nonoptimal purchase of information at any given price and also to a nonoptimal allocation of the information purchased.” 12 M. Katz and C. Shapiro (1994) “Systems Competition and Network Effects” Journal of Economic Perspectives, 8(2): pp.93-115. 13 Solver statistics from Innocentive company website < http://www.innocentive.com/crowd-sourcing-news/faq/ >, accessed December 12, 2008. Innocentive Case 3109, presented on March 3, 2005 at MIT by Innocentive Chairman of the Board and Founding CEO Darren J. Carroll, posted at , accessed December 12, 2008. 14 C. Dean, “If you have a problem, Ask Everyone,” New York Times, July 22, 2008, posted at < http://www.nytimes.com/2008/07/22/science/22inno.html?_r=2&oref=slogin&oref=slogin>, accessed December 12, 2008. 15 Karim Lakhani and Lars Bo Jeppesen, Getting Unusual Suspects to Solve R&D Puzzles, Harvard Business Review, May 2007, pp. 30-32. 16 Ed Melcarek’s Bio from Innocentive website, posted at < http://blog.innocentive.com/2008/08/20/ed-melcarek/ >, accessed December 12, 2008 and J. Howe, “The Rise of Crowdsourcing,” Wired Magazine, Issue 14.06, June 2006, posted at , accessed December 12, 2008. 17 Besides Innocentive, yet2.com, Nine Sigma, and YourEncore are examples of online markets for ideas, which are also referred to as “ideagoras”. See D. Tapscott and A.D. Williams, “ Ideagora, a Marketplace for Minds,” BusinessWeek, Feb 15 2007, posted at < http://www.businessweek.com/innovate/content/feb2007/id20070215_251519.htm>, accessed December 12, 2008.

34

18

Lakhani and Jeppesen Op cit.; Karim Lakhani and Jill Panetta, The Principles of Distributed Innovation, Innovations: Technology, Governance, Globalization 2(3) summer 2007, pp. 97-112. 19 J.C. Harsanyi, “Rational choice models of behavior versus functionalist and conformist theories,” World Politics, 21/4 (1969): 513-538. The quote was taken from page 524. 20 Examples taken from K.L. Tidd and J.S. Lochard, “Monetary significance of the affirmative smile: a case for reciprocal altruism,” Bull. Psychonom. Soc. (1978) 11, 344-346, and R. Cialdini, Influence – The Psychology of Persuasion, New York: Quill William Morrow, 1993. 21 For details of the experiment, see E. Fehr and A. Falk, “Wage Rigidity in a Competitive Incomplete Market,” Journal of Political Economy, 107/1 (1999): 106-134, and E. Fehr and A. Falk, “Psychological Foundations of Incentives,” European Economic Review, 46/4-5 (2002): 687-724. 22 http://blog.flickr.net/en/2008/11/03/3-billion/ 23 For traffic statistics on Flickr, see , , , , , accessed in October 2008. “Plain Text: Photo for the Masses,” Newsweek Web Exclusive, March 11, 2005, posted at , accessed December 12, 2008. 24 S. Levy and B. Stone, “The New Wisdom of the Web,” Newsweek, April 3, 2006, posted at , accessed December 12, 2008. 25 Other examples of Flickr social groups with some form of reciprocity rules include Flickr Hearts, Hearts Award, and Flickr My Winners. 26 From Flickr website, , accessed December 12, 2008. 27 See Yahoo press release, December 13, 2006, posted at , accessed December 12, 2008. S. Hamner, “The Secret to Yahoo Answers Success,” Business 2.0 Magazine, October 2, 2006, posted at , accessed December 12, 2008. 28 F. Galton, “Vox Populi,” Nature, 75 (1907):450-451 and F. Galton, Memories of my life, (London: Methuen & Co., 1908). 29 “Wisdom of the Crowds,” NOVA scienceNow, aired June 25 2008, posted at , accessed December 12, 2008. 30 J. Surowiecki, The Wisdom of Crowds: Why the many are smarter than the few, (New York: Doubleday, 2004). 31 Retrieved from Yahoo Answers website, . accessed December 12, 2008. 32 These examples are drawn in April 2009. The Innocentive project numbers for the projects are: 8135922, 8110538, 8219824 and 8197987, respectively. 33 Graph regenerated based on Fehr and Falk (1999), op. cit. Figure 5. 34 Graph regenerated based on Fehr and Falk (1999), op. cit. Figure 6. 35 Graph generated based on data from Fehr and Falk (1999), op. cit. Figures 5 and 6.

35

Summary A Yahoo Research and Haas School of Business, UC Berkeley study regarding the crowdsourcing of innovation.

Description Abstract: Governments and foundations have successfully harnessed tournaments to spur innovation. Yet this tool is not widely used by firms. We offer a framework for managers seeking to organize tournaments for ideas. We present the theoretical underpinnings of tournaments. We then connect the theory with three recent innovations—the power of the network, the wisdom of crowds, and the power of love—that boost the effectiveness of tournaments. Short cases and academic studies are used to illustrate our framework.

Tags: Haas School of Business, Innovation, UC Berkley, Yahoo Research

Posted by Cristina Caine

Jan 07, 2011 06:16 am GMT Viewed 619 times

Author: John Morgan and Richard Wang Source

Contact us if you are owner of this content and want to make changes.

Download Flag This

Events

Tournaments for Ideas

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®

Events

Related Content

Tournaments for Ideas

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®