The Industry Website

Web's Largest listing of crowdsourcing and crowdfunding events


Web's Largest Directory of Sites

2,968 crowdsourcing and crowdfunding sites

Two Approaches to Crowdsourced Software Testing

Two Approaches to Crowdsourced Software Testing

Editor's Note: The following comes to us from Markus Steinhauser, co-founder and head of communications at Testbirds, a crowdsourced software testing company. Steinhauser discusses the differences between managed and self-service crowdsourced testing solutions. You can find out the latest news about the company on Twitter @Testbirds.

The spectrum of crowdsourcing projects and providers has seen a tremendous increase in recent years. You can now fund your product, build and test the website to sell that product, and then translate the site to conquer other markets – all using the power of the crowd.

Depending on the task, some platforms offer access to their crowd through a self-service model, a managed service by the provider, or both. But which is better? Well, it depends!

Different types of crowdsourcing use different business models for their service and even within the same field, these might vary. The following article therefore concentrates on this question using the case of crowdtesting, which utilizes actual consumers as well as experts to test software.

Let’s assume that a company is in the process of developing a new app and decides to use crowdtesting as part of their quality assurance. The product manager of this company wants to know two things:

  1. What does my target group think about the app?
  2. Does it work on the multitude of devices out there, especially iOS and Android?

First, the test needs to be set up. Part of this includes defining the scope of the project, selecting a specific target group and defining questions that need to be answered.

In order for a crowdtesting project to be successful, detailed project management during the test is crucial. Testers can work remotely from their homes, the train, or even their local coffee shop. They are asked to document their work through written documentation, screenshots, or screencasts. This is important mainly for two reasons:

  1. Since the testers are not in-house, documentation is the only way to ensure that there is sufficient test coverage. Was everything that was part of the initial test really tested?
  2. If issues are found, they have to be traceable and reproducible. How did the user get to this point and exactly what is the issue?

Once all the participating testers have done their work, the results need to be aggregated and analyzed. Depending on the focus of the project, the resources required for each step may vary. This is usually time-consuming even for smaller tests.

The product manager needs to decide whether to use a managed crowdtesting service or to do the management within his own team. While a managed service will save considerable time and manpower, it is more expensive than using a self-service. With the latter, the setup, project management and evaluation all needs to be done internally.

It is impossible to say which option is better without taking all specific details of the project into consideration. There are, however, some general guidelines that apply. Single setups require the most resources, because all the parameters need to be defined first. In contrast, recurring projects require less time to set up each time. This is especially relevant if the release cycles of a software are happening at frequent intervals, for example several times each month. In addition, managing a crowd of people who are working remotely requires a lot of experience in order to achieve high-quality results.

Consequently, the following aspects can be part of the decision process in between managed and self-service:

  1. Is documentation necessary?
  2. Do we have experience in our team setting up projects?
  3. Are we able to manage them?
  4. Do we have the resources to do all of this internally?
  5. Does the budget allow for a managed test?

Taking all these aspects into consideration, it usually makes sense to start out with at least one managed crowdtesting project. It is then possible to compare the effort and cost with a test done in self-service. A typical scenario would be that a company tries out crowdtesting with a managed service at the beginning. If it turns out to be a long-term option, the service has to be integrated into internal processes, which can also be a challenge. From the experience gained through working together in a managed framework, the company can think about switching to a self-service model to cover recurring testing needs internally. In certain cases, they can also use a flat rate access to the crowd though the provider’s platform instead of paying per project.

Before making a decision, all aspects and goals of a project need to be defined. Through their expertise, crowdtesting providers can then offer valuable advice on the pros and cons of a self-service model versus a managed service.

Flag This



 Join or Login