STARTUPS

Moving beyond guerrilla research

Many startups are hooked on guerrilla research methods which aren’t healthy for them in the medium-long term.

8 min readFeb 3, 2020

Guerrilla usability tests are the staple diet of the tech startup. They grab people wherever they can and stick their product in front of those people to get feedback.

Startups should often move quickly on from this and do a bit more deliberate research with more specific types of people, rather than those who hang out in the local coffee shop. Some startups figure this out quite quickly and move on, but many plug away for years with this ill-fitting approach.

There are 3 key issues with it…

Coffee shop dwellers are probably not your target users.
It’s an approach with limited value
You need to do research beyond usability testing

Why guerrilla testing works

Guerrilla testing works because at first pass, some of the issues with a design can often be found with just about anyone outside of your team. When constraints like budget and time exist, then doing this is better than nothing.

If you are following up with more deliberate usability testing, it can also be a good way to stop the more obvious issues early. However it’s often not as quick as you think it will be. If time is short, it can be better to go straight into testing with your actual target users.

It’s more often lack of budget or lack of practice in other approaches, which leads to usability testing with randoms in a coffee shop being the approach taken.

Testing is about CAN, not WILL

Usability testing is a technique for discovering the extent to which people can perform tasks with your product, with a focus on uncovering usability issues. It doesn’t necessarily tell you what’s actually going on when they do. It doesn’t tell you what they will do with your product or what they need from it. It’s quite common to mistake the behaviour you observe in usability tests for the behaviour which will happen in real life.

While usability testing itself is relatively easy to run. Knowing what to take note of and what to discard is a lot harder when the thing you’re testing is a new concept. It’s usually not an experienced person running these tests for a startup, so they are less likely to be able sift out the bad bits and leave just the good.

The 3 types of user research

I often find myself quoting Will Myddleton’s explanation of the types of user research quite a lot. I believe this explanation is easier to understand than many others. They are…

Testing the things the team has built
Working out what the team should build next
Understanding potential users and their lives

Usability testing falls into the 1st category, when you are testing with actual target users then you learn little bits about them which fall into 2 and 3 as a by-product. But when you are doing guerrilla testing with randoms, you get only the very prominent findings for category 1. It’s dangerous to use your findings to inform anything else.

They aren’t your users.

Why bother with the other two types?

It’s perhaps more important for startups to focus on types 2 and 3 over 1. I’ll try to explain why with a made-up example.

Cook4U

This imaginary startup you and I both work for is called Cook4U. It’s an app which allows consumers to buy a week’s worth of wholesome home-made cooking direct from other people who love to cook.

We go into our local coffee shop to test the app, the people who agree to be participants play along nicely and we find some things we could improve about the user experience of the app to make it more usable.

We fix the issues we deem to be most impactful and get some ideas for new features. The same approach is repeated a few more times until we have some hefty pieces of work in the backlog. We try to make decisions based on the things we think will have the biggest impact on our users.

But they aren’t our users

The internal conversations we have refer back to things we saw in coffee shop testing. But nobody is accounting for the fact that these people aren’t our users, they are customers of the coffee shop, not our app.

Our actual users value our app for a bunch of reasons we think we understand. We’ve been guessing a lot and layering on stuff we heard when we spoke to people in coffee shops, who aren’t our users.

Nobody accounts for this in the discussions, so the team goes to work on bigger pieces of work which don’t impact our success metrics. We are wasting time and energy on work which doesn’t have an impact. We scratch our heads and wonder what’s going on.

We don’t really understand the world of our target users and how our product fits into that world. That’s what research types 2 and 3 are for. Not only should we have been testing with target users, but we should have moved on from solely doing usability testing at an earlier stage.

We needed to find out more about our users and less about how people in coffee shops respond to an app they’ve been asked to look at.

Who are you actually targeting?

Startups are usually challenging the status quo, so they are often trying to shift people toward a new behaviour by adopting a new solution. Your eventual market for this new approach is approaching the problem your product solves in a different way. You think their lives will be better if they use your product instead.

Your early users are often very different to your later ones and they make up quite a small percentage of what your overall user base is going to be in the longer term. In short, most of your target customers are often not ready for you.

Even if the entire world is your target audience, if you grab someone off the street for your research just now, then they are most likely to be a later stage user (who you can’t yet satisfy at all) or someone who is simply never going to be one.

Cook4U’s initial target customers are perhaps buying microwave meals just now. Perhaps they are busy people who need the convenience of not cooking, but are aware their diet and health are maybe suffering as a result. They are the most motivated and/or able to give it a go.

During our coffee shop testing, we fixed some usability issues which affected people who drink in coffee shops. We didn’t learn about the problems encountered by the people whose lives our app improves. These people didn’t have the same motivations as our users.

Let’s say one of the most important issues with the app was that people forgot to place their orders in good time, because they were busy doing other stuff. They end up going to the supermarket or ordering a fast food delivery instead. None of them sent us a support ticket explaining this. Instead they just went to the supermarket, ate their microwave dinner and got on with their lives.

We could never have found this out from people in coffee shops, because they never actually used our service for real and weren’t living the same lives as our real users.

Learning about your real users can take time

Learning about your users and their world is a longer, continuous process than sporadic usability tests. But the payback can make the difference between life and death for a new product.

A founder I’m working with recently said something to me about a crucial discovery they had made. This discovery had changed their roadmap entirely…

“I feel so stupid, like we should have discovered this 6 months ago.”

They are being unfair on themselves to think like this, because the path to the discovery had several steps. There was little chance they could have found what they needed to know quickly and without talking to specifically the right people for this stage of the product’s adoption.

They didn’t have to do lots of research to get there, but they needed to speak to the right people. Discovering who those people were took research in itself and then finding them in order to speak to them, took time also. When they finally spoke to those people, it became clear they approached the problem in a very different way to how the team had imagined.

If they had relied on guerrilla usability tests, they would still be wasting time and effort working on the wrong things, going in the wrong direction.

How UX research is imagined

The general mental picture of how UX research works involves someone talking to a bunch of users, analysing the findings and then coming back with a load of things for the team to work on.

Some of the most important things there are to find, can not be found this quickly or easily. Instead you sometimes need to go through a few phases of research in order to make important breakthroughs in your understanding. It’s often a matter of getting things less wrong over time, rather than getting things right.

This fact doesn’t sit well with a sector obsessed with speed of delivery. Instead it defaults to the fastest techniques above all else at times.

Lack of priority

Companies which rely solely on guerrilla testing are ones which don’t prioritise user research highly enough to spend time and money on a more thorough approach. When time and budget are both tight, I admit that it’s debatable whether that is a good call or not. Many successful startups got away with it, others didn’t.

When you see research as a nice thing to have, something to provide a slightly nicer experience, then that’s an understandable view.

Startups (and other organisations) should see it as a means to help decide what to work on and what form your product should take. When you look at it this way, there is a stronger argument for giving decent research more priority

About the author

I’m David Hamill. I help organisations take better decisions with lean but meaningful UX research. If you liked this post, you can read some more below.

If you would like my help to improve your product decisions, then get in touch.