Uncharted Waters

Jul 22 2014   3:56PM GMT

Crowdsourcing for software testers

Justin Rohrman Justin Rohrman Profile: Justin Rohrman

The world of crowdsourcing for for software testing is pretty small. There are two big players, maybe 3 if we’re being generous. The business model of crowdsourcing is something like a freelance body shop on crack. Software companies, usually small-ish or startups, approach crowd source providers to get testing for a flat or hourly rate. The crowd source company spins up a  bunch of testers to work on the project. The client gets a bout of quick software (and bug reporting) testing, the testers get paid a little bit for each accepted bug, and the crowd source company makes the difference between what the client pays and what they pay out to testers.

There is a new crowd source company called Testlio changing the rules for how crowd sourcing is done.

testlioTestlio was founded by Kristel Viidik and Marko Kruustuk after winning the AngelHack hackathon which led them to getting a round of seed funding from TechStars. The pair took this seed money and went all in with their plan of creating a new spin on crowd sourcing for software testing.


Testlio has a fresh take on crowd sourcing. They are currently differentiating themselvs from the competetion in a couple important ways. First, rather than relying on a swarm of testers with unknown skill levels, Teslio has a smaller community of testers that are selected based on reputation. A testers reputation can come from positive interactions in the testing community, or testing challenges on the testlio site, or beginning testlio work with smaller unpaid projects. The reputation based system results in a team of skilled testers being placed on your project. All this means that the client can count on fast and good testing.

Speaking of pay, this is the second main differentiator that separates Testlio from the other crowd source providers. The most common payout structure for crowd sourcers is paying a tester $X per bug that is accepted by the client. This means a couple things that aren’t great for anyone. One, testers tend to focus on shallow testing in order to document more, but usually less important to the business, issues. Another side effect of this is competition within the test team to get an issue documented in the tracking system first. The result is usually tons of low quality, difficult to understand test reports. Testlio pays their testers by the hour. This pay structure encourages teams to work together, much in the way they would at a normal day job.

My experience

Last weekend, I had the pleasure of participating in a brief round of testing through Testlio. After joining, I received a friendly message from Kristel through the Testlio messaging system, and of course my first question was how to get on a real live testing project. She found an approiate project and gave me an initial two hour time box to do my testing over the course of a few days. After joining the project, I was directed a page with a helpful project description, a list of the areas of the software I could explore, the bug tracking system, and a group chat where I could talk with other testers working on the project. Everything was in one convenient place. I really enjoyed having the group chat to get the opinions from other testers on things I noticed in the test product.

When my two hours was up, that was it.

Long term vision

The longer term vision for Testlio is to create test teams with a little more longevity on client projects. So, rather than spending a couple of hours over a weekend testing for a client, you could work with a team for the duration of that project. Maybe for a week, or a month, or longer.  Or at least for as long as the client is happy. The majority of the work Testlio is doing right now is mobile focused. They are also planning to diversify and offer testing services for a wider variety of software platforms.

Testlio is doing something interesting, I look forward to seeing how they develop.

4  Comments on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • kristelviidik
    Thank you Justin! :)

    Let me know if there's any way we could make your experience even better!
    20 pointsBadges:
  • ajay184f
    I had similar experience with them. The best part about them is that they are very open to honest and critical feedback.
    20 pointsBadges:
  • kristelviidik
    Thanks Ajay, We are always here to listen Testlions ;)
    20 pointsBadges:
  • Genderhayes
    Carried out by a number of  different testers from different places under diverse realistic platforms
    10,720 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: