Web's Largest listing of crowdsourcing and crowdfunding events

Events

Advertise

Web's Largest Directory of Sites

2,358 crowdsourcing and crowdfunding sites

Design of Experiments for Crowdsourcing Search Evaluation: Challenges and Opportunities

document Distributed Knowledge, Open Innovation

Design of Experiments for Crowdsourcing Search Evaluation: challenges and opportunities
Omar Alonso
Microsoft

23 July 2010

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Disclaimer

The views and opinions expressed in this talk are mine and do not necessarily reflect the official policy or position of Microsoft.

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Introduction
• Mechanical Turk works
– Evidence from a wide range of projects – Several papers published – SIGIR workshop

• Can I crowdsource my experiment?
– How do I start? – What do I need?

• Challenges and opportunities in relevance evaluation
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Workflow
• • • • • • Define and design what to test Sample data Design the experiment Run experiment Collect data and analyze results Quality control

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

A methodology
• • • • Data preparation UX design Quality control Scheduling

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Questionnaire design
• Instructions are key • Ask the right questions • Workers are not IR experts so don’t assume the same understanding in terms of terminology • Show examples • Hire a technical writer • Prepare to iterate
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

UX design
• Time to apply all those usability concepts • Need to grab attention • Generic tips
– – – – – Experiment should be self-contained. Keep it short and simple. Brief and concise. Be very clear with the task. Engage with the worker. Avoid boring stuff. Always ask for feedback (open-ended question) in an input box.

• Localization
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Example - I
• Asking too much, task not clear, “do NOT/reject” • Worker has to do a lot of stuff

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Example - II
• Lot of work for a few cents • Go here, go there, copy, enter, count …

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Example - III
• Go somewhere else and issue a query • Report, click, …

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

A better example
• All information is available
– What to do – Search result – Question to answer

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

TREC assessment example
• Form with a close question (binary relevance) and open-ended question (user feedback) • Clear title, useful keywords • Workers need to find your task

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Payments
• How much is a HIT? • Delicate balance
– Too little, no interest – Too much, attract spammers

• Heuristics
– Start with something and wait to see if there is interest or feedback (“I’ll do this for X amount”) – Payment based on user effort. Example: $0.04 (2 cents to answer a yes/no question, 2 cents if you provide feedback that is not mandatory)

• Bonus • The anchor effect
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Development
• Similar to a UX design and implementation • Build a mock up and test it with your team • Incorporate feedback and run a test on MTurk with a very small data set
– Time the experiment – Do people understand the task?

• Analyze results
– Look for spammers – Check completion times

• Iterate and modify accordingly
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Development – II
• Introduce qualification test • Adjust passing grade and worker approval rate • Run experiment with new settings and same data set • Scale on data first • Scale on workers

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Experiment in production
• • • • • Ad-hoc experimentation vs. ongoing metrics Lots of tasks on MTurk at any moment Need to grab attention Importance of experiment metadata When to schedule
– Split a large task into batches and have 1 single batch in the system – Always review feedback from batch n before uploading n+1
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Quality control
• Extremely important part of the experiment • Approach it as “overall” quality – not just for workers • Bi-directional channel
– You may think the worker is doing a bad job. – The same worker may think you are a lousy requester.

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Quality control - II
• Approval rate • Qualification test
– Problems: slows down the experiment, difficult to “test” relevance – Solution: create questions on topics so user gets familiar before starting the assessment

• Still not a guarantee of good outcome • Interject gold answers in the experiment • Identify workers that always disagree with the majority
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Methods for measuring agreement
• What to look for
– Agreement, reliability, validity

• Inter-agreement level
– Agreement between judges – Agreement between judges and the gold set

• Some statistics
– Cohen’s kappa (2 raters) – Fleiss’ kappa (any number of raters) – Krippendorff’s alpha

• Gray areas
– 2 workers say “relevant” and 3 say “not relevant” – 2-tier system
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

And if it doesn’t work ..

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Time to re-visit things …
• Crowdsourcing offers flexibility to design and experiment • Need to be creative • Test different things • Let’s dissect items that look trivial

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

The standard template
• Assuming a lab setting
– Show a document – Question: “Is this document relevant to the query”?

• Relevance is hard to evaluate • Barry & Schamber
– – – – Depth/scope/specifity Accuracy/validity Clarity Recency
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Content quality
• People like to work on things that they like • TREC ad-hoc vs. INEX
– TREC experiments took twice to complete – INEX (Wikipedia), TREC (LA Times, FBIS)

• Topics
– INEX: Olympic games, movies, salad recipes, etc. – TREC: cosmic events, Schengen agreement, etc.

• Content and judgments according to modern times
– Airport security docs are pre 9/11 – Antarctic exploration (global warming )
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Content quality - II
• Document length • Randomize content • Avoid worker fatigue
– Judging 100 documents on the same subject can be tiring

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Presentation
• People scan documents for relevance cues • Document design

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Presentation - II

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Scales and labels
• Binary
– Yes, No

• Likert
– Strongly disagree, disagree, neither agree nor disagree, agree, strongly agree

• DCG paper
– Irrelevant, marginally, fairly, highly

• Other examples
– – – – Perfect, excellent, good, fair, bad Highly relevant, relevant, related, not relevant 0..10 (0 == irrelevant, 10 == relevant) Not at all, to some extent, very much so, don’t know (David Brent)

• Usability factors
– Provide clear, concise labels that use plain language – Terminology has to be familiar to assessors
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Difficulty of the task
• Some topics may be more difficult • Ask workers • TREC example

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Relevance justification
• Why settle for a label? • Let workers justify answers • INEX
– 22% of assignments with comments

• Must be optional • Let’s see how people justify

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

“Relevant” answers
[Salad Recipes]
Doesn't mention the word 'salad', but the recipe is one that could be considered a salad, or a salad topping, or a sandwich spread. Egg salad recipe Egg salad recipe is discussed. History of salad cream is discussed. Includes salad recipe It has information about salad recipes. Potato Salad Potato salad recipes are listed. Recipe for a salad dressing. Salad Recipes are discussed. Salad cream is discussed. Salad info and recipe The article contains a salad recipe. The article discusses methods of making potato salad. The recipe is for a dressing for a salad, so the information is somewhat narrow for the topic but is still potentially relevant for a researcher. This article describes a specific salad. Although it does not list a specific recipe, it does contain information relevant to the search topic. gives a recipe for tuna salad relevant for tuna salad recipes relevant to salad recipes this is on-topic for salad recipes

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

“Not relevant” answers
[Salad Recipes]
About gaming not salad recipes. Article is about Norway. Article is about Region Codes. Article is about forests. Article is about geography. Document is about forest and trees. Has nothing to do with salad or recipes. Not a salad recipe Not about recipes Not about salad recipes There is no recipe, just a comment on how salads fit into meal formats. There is nothing mentioned about salads. While dressings should be mentioned with salads, this is an article on one specific type of dressing, no recipe for salads. article about a swiss tv show completely off-topic for salad recipes not a salad recipe not about salad recipes totally off base

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Data analysis tools
• • • • Excel R Databases Pivot

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Platform alternatives
• Do I have to use MTurk? • How to build your own crowdsourcing platform
– Back-end – Template language for creating experiments – Scheduler – Payments?

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

MapReduce with human computation
• MapReduce meets crowdsourcing • Commonalities
– Large task divided into smaller sub-problems – Work distributed among worker nodes (turkers) – Collect all answers and combine them

• Variabilities
– Human response time varies – Some tasks are not suitable
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Platform perspective
• Problems
– – – – Very rudimentary No tools for data analysis No integration with databases Very limited search and browse features

• Opportunities
– What is the database model for crowdsourcing? – MapReduce with crowdsourcing – Can you integrate human-computation into a language?
• crowdsource(task,5)> 0.80
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Research questions
• What are the tasks suitable for crowdsourcing? • What is the best way to perform crowdsourcing?

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Conclusions
• Crowdsourcing for relevance evaluation works • Fast turnaround, easy to experiment, few dollars to test • But you have to design the experiments carefully • Usability considerations • Worker quality • User feedback extremely useful
SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Conclusions - II
• • • • • • • Crowdsourcing is here to stay Need to question all aspects of relevance evaluation Be creative Lots of opportunities to improve current platforms Integration with current systems MTurk is a popular platform and others are emerging Open research problems

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Announcement
Omar Alonso, Gabriella Kazai, and Stefano Mizzaro. Crowdsourcing for Search Engine Evaluation: Why and How. To be published by Springer, 2011.

SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation

Summary Omar Alonso discussed about an effective design of experiments for crowdsourcing search evaluation. Alonso presented a detailed workflow of employing crowdsourcing in evaluation starting with defining the evaluation exercise and collecting the data to be used and testing their quality. Designing the experiment was the next crucial step described which
requires a careful user experience (UX) design, careful selection of workers, policy and process for awarding payment and determining its amount, and testing and analyzing the design itself.

Description Alonso concluded his talk laying a number of challenges and opportunities for improving crowdsourcing. He pointed out, for instance, the strengths and limitations of the current crowdsourcing platforms, the need for data analysis tools, browsing and searching features and integration with
the databases technology. The talk offered the attendees a variety of practical advice for increasing the quality of crowdsourcing based on Alonso’s extensive experience.

Tags: Crowdsourcing, Crowdsourcing for Search Evaluation Workshop, CSE 2010, Omar Alonso, search evaluation, workshop

Posted by Tim Satterthwaite

Mar 27, 2011 07:37 am GMT Viewed 915 times

Author: Omar Alonso Source

Contact us if you are owner of this content and want to make changes.

Download Flag This

Events

Design of Experiments for Crowdsourcing Search Evaluation: Challenges and Opportunities

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®

Events

Related Content

Design of Experiments for Crowdsourcing Search Evaluation: Challenges and Opportunities

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®