TurKit: A Toolkit for Human Computation Algorithms
Rob Miller & Greg Little
User Interface Design Group MIT CSAIL
Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein
June 10, 2010 1
Outline
Iterative human computation
± TurKit: a toolkit for human computation algorithms
Systems with human computation inside
± VizWiz: vision for blind users with camera phones ± Soylent: putting a crowd inside Microsoft Word
June 10, 2010
2
What is TurKit?
What is TurKit?
2 Ways
Way 1
Way 1
ADD 1, 2
Way 1
ADD 1, 2 JMP somewhere
Way 1
ADD 1, 2 JMP somewhere «
Way 1
ADD 1, 2 JMP somewhere « TURK ³How do you feel?´
Way 1
ADD 1, 2 JMP somewhere « TURK ³How do you feel?´
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Way 2
Demo
Demo
TurKit
What can you do with TurKit?
x50
31
19
Improve
Improve
Vote
Improve
Vote
Improve
Progression of Other Offer Description: $10,000 more better package far more competitive offer more competitive offer competing offer
x
Iterative
Parallel
Iterative
Parallel
Iterative
Parallel
Iterative
Which is better?
Which is better?
30 images
Which is better?
30 images 6 iterations
Which is better?
30 images 6 iterations
Which is better?
Which is better?
Which is better?
Outline
Iterative human computation
± TurKit: a toolkit for human computation algorithms
Systems with human computation inside
± VizWiz: vision for blind users with camera phones ± Soylent: putting a crowd inside Microsoft Word
June 10, 2010
58
VizWiz: Helping the Blind See
joint work with Jeff Bigham (University of Rochester)
Which door is the women¶s restroom?
June 10, 2010
59
Helping the Blind See
Mechanical Turk
1. the left one 2. LEFT 3. on the left
June 10, 2010
60
Field Study
Field deployment with 11 blind iPhone users
± Answers received within a minute for ~5 cents a question ± Latency can be reduced to less than 30 seconds by keeping workers warmed up (at $4 per hour)
June 10, 2010
61
All Mobile Users Are Situationally Disabled
joint work with Rajeev Najak
Mechanical Turk
When is Barack Obama speaking at MIT?
June 10, 2010
62
Soylent: Putting a Crowd inside MS Word
joint work with Michael Bernstein
June 10, 2010
63
Crowd-Driven Proofreading
June 10, 2010
64
Crowd-Driven Shortening
June 10, 2010
65
Bigger Idea: REAL Wizard of Oz
Wizard of Oz is a tried-and-true prototyping technique in AI and HCI
± Putting a human behind the curtain until we figure out how to put software there
Crowd computing enables Wizard of Oz systems that are useful and deployable
± So we can start collecting data about how the system is really used in practice ± Adding the AI backend becomes a performance or cost optimization
June 10, 2010
66
Outline
Iterative human computation
± TurKit: a toolkit for human computation algorithms
Systems with human computation inside
± VizWiz: vision for blind users with camera phones ± Soylent: putting a crowd inside Microsoft Word
Funded in part by
June 10, 2010
67
Summary
The Crowdsourcing and Translation workshop was held at the University of Maryland's Inn and Conference Center on June 10-11, 2010. The workshop was conducted over a two-day period, which included:
Analysis of Existing Research Areas
Brainstorming Activities on Research Areas
Workshop Outcomes & Next Steps
This presentation was prepared by Robert Miller and Greg Little, who were two of the speakers during the said workshop.
Description
Rob Miller, associate professor and leader of the User Interface Design Group in MIT's Computer Science and Artificial Intelligence Lab, whose research lies at the intersection of user interfaces and programming, with a recent focus on toolkits and systems that incorporate human computation and crowd computing; he will be joined by Greg Little, a PhD student in the same lab and the creator of Turkit.