The Industry Website

Web's Largest listing of crowdsourcing and crowdfunding events


Web's Largest Directory of Sites

2,968 crowdsourcing and crowdfunding sites

What to Keep in Mind When Crowdsource Testing
© Image: Dusit /

What to Keep in Mind When Crowdsource Testing

Editor's Note: This is the fourth and final in a series of posts from our friends at passbrains, who will be walking us through a primer on crowdsourced testing. So far we've defined crowdsource testing, and looked at exactly how it works, as well as why it works. In this part, we cover some other things to keep in mind.

What to look out for?

Crowdsourced testing is best when the product under development is consumer-centric rather than enterprise-centric, such as gaming, mobile or web-driven consumer applications. A global user base to test the product should exist and the product should be relevant to the community at large. This is also a test for the application’s potential success in the marketplace.

Paul Herzlich, a software-testing analyst who oversees crowdsourcing services at Ovum, an institution that provides its clients with independent and objective analyses stated, “If you are testing software that all kinds of strangers are going to use, then why not use a bunch of strangers to test it. Also, it depends on what kind of testing you need to do. For testing user interfaces, sure – it makes sense.”

There should also be an earnest interest from the community to proffer critical feedback for the product under consideration such as a monetary reward. This also brings forth another interesting challenge. The product company is not obliged to follow through on the community’s recommendations and may dispense with the feedback for various internal reasons. In this case, the community may feel unheard and this mandates a fine balancing act of the entire ecosystem.

The product company should be committed to working with a large group of people and understand that it involves some degree of overhead in such a decentralized test effort. It also requires certain subject matter experts to mentor and monitor various testing efforts as well as offer support and relevant guidance to the testing teams. If the product team does not have the resources to take on full-fledged testing in-house, but has a good understanding of the testing requirements, it can realize its overall strategy from a globally sourced team.

With normal employment contracts, employees receive a salary for their contribution and the firm owns any intellectual property developed by the employee during their tenure with the organization. In a crowdsourcing constellation, people are participating voluntarily. Therefore it is an essential necessity to explicitly state the position on Intellectual Property (IP), i.e. a condition of the right to participate is the acceptance of Intellectual Property transfers to the client, to avoid potential risks of IP infringement by the contributors.

A crowd-sourced project requires skills and mastery in designing the compensation structure, both in monetary and non-monetary terms. The testers are usually paid a certain amount of money for the deliverables provided and for approved bug/issue reports. In some cases, the testers would prefer non-monetary aspects like recognition and personal satisfaction rather than monetary compensation. Thus, it is vital to understand the motivators prior to mission critical deployments.

In cases where participants are compensated on a per task basis, an incentive for participants to choose speed over accuracy exists. This is especially the case with micro tasks, which are susceptible to mistakes and could result in erroneous overall outcomes. Therefore, robust governance mechanisms need to be instilled, continually monitored and policies regularly updated to reflect the changing trends.

Checks & Balances

Last in our tutorial, but certainly not least, security is a crucial element to crowdsourced testing. More often than not, confidential customer information is exposed to testers during application testing. Any breach of this data can lead to serious damage, both to the brand and the business. Test data management ensures the availability and security of test data by obfuscating sensitive information for large-scale testing engagements. Masking such information or creating ‘test-only’ data helps maintain privacy and security while using crowdsourced testing services.

In almost all cases, the testers are required to sign a Non-Disclosure Agreement (NDA) when they join the community. The NDA forbids them from talking about customers, their products or specific defects, both offline and online on LinkedIn, Facebook, Twitter, personal blogs or anywhere outside the confines of the private testing platform. Beyond that, the customers can upload a customized NDA, which testers must sign before viewing the customer’s project. For projects that require a high level of security, a pre-screened list of white hat engineers, that have a long professional relationship with the platform company are selected.

By combining an internal, permanent team of testers with a crowd of experienced software testers working from around the globe, superior quality in testing is delivered. By constantly filtering the network of testers to accept only experienced software testing professionals, applicants without formal training and significant professional experience are eliminated. This ensures the quality and the validity of the bugs reported. Finally, tests are dispatched to individual testers based on their experience, available material, and languages mastered. The testers and test project exposure are continually monitored to ensure both quality and integrity, not only of the test results, but also of the associated environment.

What's Next?

Crowdsourced testing is a relatively new methodology in software quality engineering. Thoroughly planned, organized and applied, it is a great tool for software producers to ensure compatibility of their applications on most major platforms and configurations and it helps to discover and eliminate harmful defects which survived traditional software QA and testing efforts.

The further evolution of crowdsourced testing will lead to new collaboration forms between service buyers and vendors, between organizations and crowd testers, and maybe even between crowd testers themselves. The emerging global crowd testing community will continuously specialize into different industry domains, software product categories and customer relationships, most probably materializing in the form of various quasi-vertical categories of on-demand, self-service software testing workbenches.

- The above was originally written by Mithun Sridharan, and adapted by Dieter Speidel of Disclosure: is a client of / massolution.

Flag This



 Join or Login