2,650 crowdsourcing and crowdfunding sites
Editor's Note: Last month, in the first in a series of posts, our friends at passbrains gave us a brief introduction to crowdsource testing, its advantages and disadvantages. In this second part, we learn exactly how it works.
Most vendors of crowdsourced testing (also called “crowd testing”) services provide the platform for managing their crowd testers and the testing projects. Clients specify the type of tests that they wish to have performed and the types of devices and configurations that the software product must be tested on.
Testers complete a profile, indicating the skills and domain knowledge they have, the devices to which they have access to, and the countries where they reside. Once a tester has completed his profile, he/she can check the project dashboard for a listing of projects and releases that are available for testing. The dashboard may also include sample test scenarios, additional tools and scripts, instructions for testers about what is expected from them, etc. Usually, the testers are provided with or required to submit their inputs for a test plan, which outlines both high level test cases and detailed test scenarios. The plan may also include whether or not the test can be automated and expected results.
A qualified Project Manager, who is typically a proven community leader or a person from the client/the platform company, designs or reviews such plans, and approves or amends such plans to cater to the client’s specific testing requirements.
Each project includes an explanation and access to a forum where bugs and issues are discussed and additional questions can be asked. Testers submit documented bug reports and are rated based on the quality of their reports. The amount the testers earn increases as their rating increases.
The community combines aspects of collaboration and competition, as members work to finding solutions to the stated problem. Forums facilitate networking and discussion of bugs or relevant issues; rating systems allow for recognition of a job well done, which helps participants gain credibility and improved career.
The crowdsourced testing team ideally works in addition to the organization's testing team, and not as a replacement.
At the heart of crowdsourced testing is the community that tests a given software product. The community encompasses people from diverse backgrounds, cultures, geographies, languages, all with a diverse approach to software usage. The community, represented by a diverse and extended user space, tests any given software by putting it to use under realistic scenarios, which a tester in the core test team may not be able to envision, given a tester’s constraints, such as limited bounds of operation, knowledge, scenarios. Thus, it is easy to observe the broad set of usage patterns that put the software under intense scrutiny. Crowdsourced software testing draws its benefits from delegating the task of testing a mobile, web or software project, while in development, on to a large number of Internet users, working in parallel for comparably short test cycle durations, to ensure that the software contains no defects.
The method of crowdsourced testing is particularly useful when the software is user-centric, when software’s success and adoption is determined by its user feedback. It is frequently implemented with gaming or mobile applications, when experts who may be difficult to find in one place are required for specific testing, or when the company lacks the resources or time to carry out internal testing.
The spectrum of issues that such test efforts could uncover within a short lead-time is particularly noteworthy. Such testing efforts yield productive results with reasonable costs. Testers usually get paid for deliverables, such as test cases and for valid reported bugs. Additional small contribution rewards often help to achieve higher project participation rates. Crowdsourced testing providers offer their services either based on flat rates for certain test capacities and durations, or they charge fixed rates per reported bug, or a combination of both.
In general, crowdsourced testing provides a much higher Return on Investment (ROI) compared to the traditional means of software testing.
- The above was originally written by Mithun Sridharan, and adapted by Dieter Speidel of passbrains.com. Disclosure: passbrains.com is a client of crowdsourcing.org / massolution.