USER RESEARCH

Hey researchers! Stop shirking usability testing.

Observing behaviour is how you build the instinct to understand it.

6 min readJul 27, 2020

There’s a school of thought in UX research where usability testing is a practice which should be mostly left to product teams to do for themselves, freeing UX researchers up to concentrate on ‘more important’ work.

For some researchers this is so they can more in-context observation instead, which is understandable. But others are forgoing all usability testing when it’s their only opportunity to regularly observe user behaviour.

I don’t believe this is good for them professionally.

I’m a hypocrite

I should start by admitting that I had the discussion only this week about getting another person to run usability tests in order to free my time up for discovery research. It isn’t the first time I’ve had such a discussion.

I’ve also enabled scores of product teams to run their own usability testing. My argument here is a little more nuanced than it might initially sound.

It’s great that many designers and product managers are capable of running usability tests. But it’s not great that so many user researchers spend so little time observing user behaviour.

Why does it happen?

Easy to learn/teach

At a basic level, usability testing is quite easy to learn how to do. In my experience though, without some involvement from an experienced practitioner, the standard of this testing will drop quite drastically.

That said, a busy researcher with more work on their plate than they can feasibly do, will often suggest a self-serve approach to usability testing. This means the product teams do it themselves, allowing the researcher to drop it from their ever-increasing list of work to do.

I tend to estimate the impact and the risk of the work the product team is doing before suggesting product teams do their own testing. I also like to keep an eye on it, to ensure it’s still going to be useful.

Less-strategic

Usability testing is also seen as ‘less strategic’ because the company has some plans over and above the value it currently delivers to its users. This view assumes they’ve pretty much nailed the existing experience and aren’t really threatened by the competition (big mistake).

If a researcher wants to get those promotions and pay-rises, they are motivated to do the work which is more visible to company leadership. This means dropping the more ‘tactical’ research for the more ‘strategic’ stuff.

This research is sometimes not more strategic at all, instead it’s just a label given to the research that company leadership cares more about it.

Not good at it

Some researchers dodge usability testing simply because they aren’t very good at it. Rather than keep doing it in order to get better, it can be easier to ditch it and get product teams to do it for themselves instead.

The value of observing behaviour

Observed behaviour is at the core of user research. The most reliable way of understanding behaviour is to see it for yourself. Many other user research techniques are a substitute for not having been able to observe. It is often unavoidable or even advisable to use a non-observed method, such as a diary study. But there are also limitations with relying on reported behaviour.

Over time, the experience gained from repeatedly observing behaviour throws a light on non-observed data that other colleagues either don’t see or invent unlikely explanations for. Observation helps to build a better intuition for what’s actually going on out there.

Your opportunities to observe

At a simplified level you have two opportunities which allow you to observe user behaviour — contextual observation (going to the user’s environment and observing them there) and usability testing.

Observing users in their own context is the purest observation, it allows you to see and understand things you might otherwise not have thought of. It also removes a lot of the falseness of a simulated scenario.

But it’s unfortunately rare. While some researchers are lucky enough to have the opportunity to frequently observe users in their own context, for many the opportunity is less frequent or even non-existent.

I’ve spent several hundreds of hours observing user behaviour and would estimate that over 95% of it has been outside of users’ own contexts. I have either been remote from the participants or have invited them into a different setting for the session.

How this has benefited me

If I hadn’t observed so much user behaviour I wouldn’t be anywhere near as good at what I do.

I get a greater understanding of what participants say in user interviews because I have seen what they are often talking about, first hand. I also ask better questions and have a greater appreciation for the consequences of findings.
I can understand more easily what is and isn’t important about some usage analytics.
I can spot issues with copy, that the writer might not see.
I can help designers to refine their designs before testing them
I can help product managers to refine their experiments before releasing them.
I often ask questions and make observations about things that others tend to have thought about.

There are probably more, but these are the benefits which immediately come to mind.

Are you getting your hours in?

Observing user behaviour is like flying a plane. The benefits come from getting your hours in and keeping them topped up.

Prototype hours?

Prototype testing is very useful, but is often not the same as watching someone using live software. If you’re only ever watching people using prototypes then you get a more constrained and artificial view of how people behave.

When you need to pick and choose between which tests you do and don’t get to observe, then choose the testing of live software as much as you can.

Watch recordings

You may have had to suggest the product team carry out their own testing, but that doesn’t mean you can’t watch the recordings or observe some sessions live.

Your intuition will be built up by observing the behaviour not by moderating the sessions.

Natural tasks

Some usability testing studies are, by design, more natural and realistic than others. When watching sessions to keep your ‘hours topped up’ then opt for the more natural sessions where participants are in the mindset of real users and are completing an entire task.

Say for example you work on a website which helps people find a lawyer and you have the option of observing two studies.

The team has recruited participants who have a current need for a lawyer in order to see what improvements could be made to the site
The team has developed a filter to refine lawyers by specialism. They have recruited participants who have used a lawyer in the past and want to see if the filter works the way they would expect.

If you can;t watch both then choose the first study rather than the second one. You can still take things from the second study, but they will be more interaction-based rather than based on needs and wider behaviour.

Realistic not real

Despite everything I just said, you should also still bear in mind that observed behaviour in a research session isn’t identical to real behaviour. Instead it’s just a piece in the puzzle of finding out what’s really going on.

If you want to see an illustration of this, then go watch some recordings of your usability tests with the sound muted. Now go and watch some session recordings of the same screens (if you have HotJar, FullStory or other session recording software). Notice the difference?

About the author

I’m David Hamill. I help organisations take better decisions through lean but meaningful UX research. If you liked this post, you can read some more below.

If you would like my help to improve your product decisions, then get in touch.