Uncharted Waters

Jan 16 2017   10:02AM GMT

Considering Crowdsourcing: An Interview

Matt Heusser Matt Heusser Profile: Matt Heusser


Crowdsourcing at TestioTest IO is a Crowdsourced testing company.  Based in Berlin, Germany – the company also has offices in San Francisco. It was as VP of engineering for Lithium Technologies, working to make CRM Software Social, that Phil Soffer saw the opportunity for Crowdsourcing. One of his visceral memories at Lithium was working on a challenging project, getting it to be successful, then losing the idea of that success when a handful of unhappy people called the senior executive team to complain about a bug.

If only those teams had quick access to testers, on-demand, a faucet they could turn on and off just prior to release — they could have found those problems.

When Phil was asked to head a Crowdsource testing startup-up, his answer was yes.

Crowdsourcing - Phil SofferI spoke with Soffer about his vision for Crowdsourcing, the state of his company — and how he was working to avoid the commodity trap.
ITKE: Tell me about your model for crowdsourced testing.

We use exploratory test as a shorthand for “not following predefined test cases.” Under that there are different ways to specify what you are trying to test – broad coverage, a specific feature, or only parts of the app with specific platforms. We match that to people – bot what devices they have, but also their aptitude as a tester, what do they test well.

ITKE: How do you know what they test well? What are some categories?

Soffer:  Many of the people we have had testing with us for years, and we get to know them. Another is how often the bugs they file are rejected by the customer. Then we have people who are very good at boundary testing; they will find the bugs that will crash your product. Then we have the people with a very high ratio of high severity to low severity bugs. Then there are people who specialize in usability and what I call “market empathy” – where the tester might know what the customer is trying to do, but is concerned the end-users might not appreciate, or benefit, from that interaction.

ITKE: One of the concerns about crowdsourcing is the “race to the bottom” – that globalization means we lower rates for workers until they approach zero. Is that a credible problem?

Soffer: If we only paid our testers by test case execution, then that is a kind of race to the bottom mentality. What we are trying to do is to find people who can find human insight into testing, and finding things that are interesting, either higher quality bugs or understanding how the software feels. That’s not a race to the bottom or at least not the same. We find we have two kinds of testers; those who work in the tech industry and those that do not. For those that do not, they are building a portfolio they can use to enter the industry. Plus, they get much better. We track performance over time and people get better at testing – so we give people an opportunity to earn while they learn and try a different kind of career.

ITKE: How much can people earn working for Testio?

Soffer: It depends on how much they work and how good they are.  I can tell you a few of our best people are making thousands of dollars part-time per month. We don’t actually know how many hours people are working; there are some people who can dive into an application for twenty minutes, find three critical bugs, and jump out. On an hourly basis, that’s really good. We also know there are so many anti-patterns that are so prevalent that someone with broad experience can translate that across different companies. I even wrote an article on this called Why Crowdsource Testing is like Sex, in the sense of cross-pollinating of DNA. If I had to tell people starting out their career, with some technical aptitude, but without any specific plans of how to break into software, I would tell them to try crowdsource testing.

Crowdsourcing Testers
ITKE: Any thoughts on the crowdsourcing in the broader sense – including services like uber and taskrabbit?

Soffer: I wrote another article, for TechCrunch, on the race to the bottom. I feel strongly that we all need to avoid it, in one way or another. On a policy level, that means understanding that “gig” work is the new normal for a lot of people. They need to have some control under conditions off work. What I do worry about a little bit, from the point of view of workers rights, is the idea that every worker is treated entirely equally in a crowdsourcing context – because we found a dramatic inequality in the level of people. For our system to work, we need to let anyone try it, then let them decide for themselves if it makes sense to them economically. If, for example, we suddenly had to make sure everyone was paid a living wage, we’d need to close our “open door to anyone” and instead heavily vet people before starting them. As-is, the barrier entry is low. You can sign up and test almost immediately and learn on the job. That’s been transformative for some of our testers – but if we pay per bug, they might start out at a very low hourly rate. I take pride in the

ITKE: Anything else we should be talking about?

Soffer: I’m trying to get my mind around how human-powered testing can fit into this DevOps world. I worry a bit that there is an idealization of a human-free software release process, that is not really what most companies are actually doing, and I’m not sure what to do about that. I wonder if your readers have any ideas?

ITKE: Hopefully they’ll leave a comment! Thank you for your time today, Phil.

Soffer: You’re very welcome.

 Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: