Web's Largest listing of crowdsourcing and crowdfunding events

Events

Advertise

Web's Largest Directory of Sites

2,355 crowdsourcing and crowdfunding sites

Best Practices for Sentiment Analysis Webinar [Presentation slides]

document Distributed Knowledge, Open Innovation

Best Practices

for

Sentiment Analysis
Presented by:

John Hoskins

Amazon Mechanical Turk

Max Yankelevich

Freedom OSS CrowdControl

2011 Amazon.com, Inc. or its Affiliates.

Introductions

2011 Amazon.com, Inc. or its Affiliates.

Welcome
 Logistics
§

How to ask a question:.
Ø Ask a question with the question panel. Ø We will moderate questions.



 Agenda
§ § § § §
 

Introductions Sentiment Analysis Applying Mechanical Turk CrowdControl – optimized for Sentiment Q&A

2011 Amazon.com, Inc. or its Affiliates.

What is Sentiment Analysis?
 Keeping

your finger on the pulse of your market – in near real time
With the explosion in use of Facebook, Twitter, and blogging – it is essential to hear the true Voice of the Customer. Yet keeping up with the round-the-clock information torrent is impossible with old, manual methods. As businesses look to automate the process of filtering out the noise, understanding the conversations, identifying the relevant content and actioning it appropriately, many are now looking to the field of sentiment analysis



2011 Amazon.com, Inc. or its Affiliates.

Why Sentiment Analysis?
The

ability to lead the competition

in customer satisfaction and support, brand and reputation management product design and marketing


With the proliferation of reviews, ratings, recommendations and other forms of online expression, online opinion has turned into a kind of virtual currency for businesses looking to market their products, identify new opportunities and manage their reputations.


2011 Amazon.com, Inc. or its Affiliates.

How to properly analyze Sentiment.

2011 Amazon.com, Inc. or its Affiliates.

Context & Cognitive Recognition are the keys to an
accurate analysis
Human analysis provides more accurate assessment.


    

Complex emotions such as sarcasm are presented in tweets and blog posts. Wit, sarcasm and complex emotions are difficult to analyze with technology alone Processing the unnatural language of text messaging: lol, omg Positive and negative emotions are difficult to accurately assess Avoid misleading classification of SPAM

2011 Amazon.com, Inc. or its Affiliates.

Is human judgment affordable and scalable?


Yes.

2011 Amazon.com, Inc. or its Affiliates.

Introducing Mechanical Turk


An affordable solution for human judgment.

2011 Amazon.com, Inc. or its Affiliates.

Mechanical Turk is a marketplace for work.
Mechanical Turk gives businesses and developers access to an on-demand, scalable workforce.


§ § § §

Flexibility: Scale your workforce up and down quickly Accuracy: Get high quality, efficient and cost effective results. Price: Pay only when you are satisfied with the results. Speed: Start receiving results in minutes

2011 Amazon.com, Inc. or its Affiliates.

Workforce
500,000
  Who
§

Workers 190+ Countries
are the Workers? Your Workforce
Workers are global: 24X7 Follow the Sun

 Managing
§ § § §

Can narrow to US based (i.e. when you need western culture competence) Leverage qualifications to find your best Workers

2011 Amazon.com, Inc. or its Affiliates.



How it Works

2011 Amazon.com, Inc. or its Affiliates.

Popular Use Cases
D a ta M a n a g e m e n t D a ta V e ri ca ti n fi o D a ta E n try & C o l e cti n l o D a ta D e -d u p l ca ti n i o A l o ri m Tra i i g g th n n

Categorization
Classification Tagging Sentiment Analysis

C o n te n t & M e d ia

Moderate Photos & Content Content Creation & Editing Transcription

Business Services Search Relevancy Product Usability Testing Research

2011 Amazon.com, Inc. or its Affiliates.

Case Study – Sentiment Analysis


Problem: A large consumer brand reporting customer need 10’s of thousands of human coded samples to establish a baseline – for each project Solution: Code them through Mechanical Turk. Details:
§ §







Projects are completed in hours providing faster to market production of broad analysis. By integrating Mechanical Turk into their process, they freed up analysts time to do value add work and can start projects in a day instead of weeks.

2011 Amazon.com, Inc. or its Affiliates.

Strategies for Analyzing Results
 Plurality

Asking multiple Workers to do assignments (if their answers agree then the result is validated)  Qualifying & Training Workers Assess competence on your coding instructions before allowing them to work for you  Known Data Sets Include work with known answers to quickly assess worker accuracy.


2011 Amazon.com, Inc. or its Affiliates.

Implementing a Sentiment Analysis Workflow

2011 Amazon.com, Inc. or its Affiliates.

Freedom at a Glance


Long term AWS Partner § Enterprise Cloud Business § CrowdControl – fastest growing division
◦



Corporate Office: Newtown PA § Regional Offices: Newark, NJ. Reston, VA, Seattle, WA § Offshore Engineering Centers § Global operations centers
◦



Privately held § Established in 2008 § 300% annual growth

2011 Amazon.com, Inc. or its Affiliates.

Workflow, Adjudication, Worker Management.


Programmatic connection to crowdsourcing.
   

How do I know I have the best answer? How do I break my workflow into discrete tasks Who are my best workers? How do I put it all together?

2011 Amazon.com, Inc. or its Affiliates.

CrowdControl for Sentiment Analysis
  


Provides High Sentiment Quality Data at Lower Cost Ability to Process Large Number of UGC Combines best of breed Artificial Intelligence to handle most challenging nuances of “Crowdsourcing” for efficient Sentiment Analysis § Worker Management § Adjudication strategies § Workflows § Can easily retrieve and send data to and from any data source (e.g. Database, File ,etc.)





2011 Amazon.com, Inc. or its Affiliates.

What Can CrowdControl™ Do for Me?
Customer IT Systems Customer IT Systems Quality Information

Complete Automation

Manual Steps

Mechanical Turk Portal Turkers

Mechanical Turk Portal Turkers

2011 Amazon.com, Inc. or its Affiliates.

Sentiment Analysis Process Setup

2011 Amazon.com, Inc. or its Affiliates.

Brand Sentiment Coding Template

2011 Amazon.com, Inc. or its Affiliates.

2011 Amazon.com, Inc. or its Affiliates.

John Hoskins, Amazon Web Services:

hoskins@amazon.comS S : M a x Ya n ke l vi , Fre e d o m O e ch

m ya n ke l vi @ fre e d o m o ss. e ch co m
@ a m a zo n m tu rk Fa ce b o o k . co m / a m a zo n m tu rk M e ch a n i l rk . typ e p a d . co m ca tu

2011 Amazon.com, Inc. or its Affiliates.

Q&A


Your Questions. Answered.

2011 Amazon.com, Inc. or its Affiliates.

Summary An informative session on Amazon Mechanical Turk where you will learn how your company can leverage the human crowd for human sentiment analysis of content such as tweets, articles, RSS feeds and blog posts. This session digs into the details of getting started and provides information on how to be successful so you get accurate results. Additionally, FreedomOSS will share their experiences designing and managing sentiment tasks and demo's their CrowdControl crowdsourcing platform that is built on top of Mechanical Turk.

Description Sentiment Analysis is a technique used to monitor, measure and derive information from the comments customers make about companies and their products on blog posts, tweets and other mediums. It is also a growing use case for Mechanical Turk.

In this Webinar we reviewed:

The value of sentiment monitoring, and the need to incorporate human judgment into the review process
How to use Mechanical Turk for sentiment analysis

Freedom OSS also provided an overview of how their CrowdControl platform, built on Mechanical Turk, can be used to programmatically analyze sentiment.

Tags: Amazon Mechanical Turk, AMT, Best Practices for Sentiment Analysis, crowd, Crowdsourcing, Freedom OSS, sentiment analysis, webinar

Events

Best Practices for Sentiment Analysis Webinar [Presentation slides]

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®

Events

Related Content

Best Practices for Sentiment Analysis Webinar [Presentation slides]

Comments

Crowdsourcing.org

Categories

Content

Crowdsourcing.org, an initiative by massolution®