Editor's Note: This is the first in a series of posts from our friends at passbrains, who will be walking us through a primer on crowdsourced testing. In this first part, we cover what exactly crowdsourced testing is, its advantages, and disadvantages.

Developing a new software product is always exciting, especially to watch ideas take form and truly become a reality. Sometimes a fresh perspective or an innovative use case is all it takes to turn a product from good to great. However, when it comes to testing, we often find ourselves in uncharted waters wondering if the product will actually work in diverse customer landscapes. It is virtually impossible to test the vast number of devices and configurations of software that web-based software can run on today. Truly robust testing is time consuming, and ensuring that every possible permutation and combination of features, localizations, and platforms works, as intended is nearly impossible.

Often times, comprehensive testing is a challenge and buggy code is delivered to the customer. For example, if a Software-as-a-Service (SaaS) application does not render in a particular browser or a critical software tool fails to deliver its intended functionality, a bug fix or a patch is promised and the vicious cycle starts all over again. Either way, the customer withstands the worst of inadequate testing, especially when faced with the escalating costs of software maintenance, performance, etc. For the software development company, ramifications include distress around brand image, perceived quality, relationship and potential future projects, trust, etc.

Welcome to the new world of crowdsourced testing, an emerging trend in software engineering that exploits the benefits, effectiveness, and efficiency of crowdsourcing and the cloud platform towards software quality assurance and control. With this new form of software testing, the product is put to test under diverse platforms, which makes it more representative, reliable, cost-effective, fast, and above all, bug-free.

Crowdsourced testing, conceived around a Testing-as-a-Service (TaaS) framework, helps companies reach out to a community to solve problems and remain innovative. When it comes to testing software applications, crowdsourcing helps companies reduce expenses, reduce time to market and increase resources for testing, manage a wide range of testing projects, test competence needs and exigency to resolve higher defect rates.

It differs from traditional testing methods in that the testing is carried out by a large number of different testers from across the globe. In other words, crowdsourced testing is a form of outsourced software testing, generally a resource-limited and therefore time-consuming activity, to testers around the world, thus enabling small startups to use ad-hoc quality-assurance teams, even though they themselves could not afford traditional quality assurance testing teams.

Advantages of crowdsourced testing:

1. Representative scenarios from the real user base, not hypothetical test cases
2. Tight feed-back loop with rapid feedback processing and agility
3. Comprehensiveness in use cases, platforms, tools, browsers, testers, etc. that is typically impossible to replicate by any single product company
4. Cost efficiency, given by the service models offered by the vendors
5. Diversity among the pool of testers lends to comprehensive testing, especially with regard to applications, which are localization based
6. Reduced time to test, time to market and total cost of ownership as most defects can be identified prior to deployment, which leads to significant reductions in maintenance costs
7. Better productivity and improved product development focus

Disadvantages of crowdsourced testing:

1. Governance efforts around security, exposure and confidentiality when offering a community project to wide user base for testing
2. Project management challenges that stem from the testers’ diverse backgrounds, languages and experience levels
3. Quality assurance efforts to verify and improve bug reports, identify and eliminate bug duplicates and false alarms
4. Equity and equality constraints in the reward mechanism with remuneration as a function of the quality of contributions that meets a prescribed minimum standard

Next in the series we'll learn a little more about how crowdsourced testing actually works.

 - The above was originally written by Mithun Sridharan, and adapted by Dieter Speidel of passbrains.com. Disclosure: passbrains.com is a client of crowdsourcing.org / massolution.