How to carry out a UX shopping study
A UX research technique to uncover how people choose between products online.
Inner city financial advice firms in the UK should put pictures of their offices on their website.
This is because some customers will use these pictures as a measure of how upmarket and established the firm is. Some people want an established wealthy looking firm with nice offices. While others don’t want to be paying for the upkeep of lovely city centre offices in their fees. Instead they want to see decent offices, that don’t look extremely expensive to run.
One of the reasons I can give this bit of advice, is that I’ve run what I call a shopping study on the subject.
In this post I explain how to run one yourself. I use this type of study to observe people doing internet research, on a decision they intend to take. I call it a shopping study because I’m often using it to watch people shopping online.
When I do this, I usually notice a big difference between how people say they use/used the internet and apps to make a decision and how they actually do it in practice.
What is a shopping study?
A shopping study is the name I give to watching people do online research for a specific decision they need to take in real life. If that sounds like contextual observation that’s because it is, to a certain extent. It’s just a little more forced than traditional approaches.
The basic format:
- Find people who are in the market for your product or have the job to do that your product helps with
- Ask them to do their research while you watch. But don’t give them a specific site or app to use. Instead let them choose what to use and where to go.
- Learn about their workflow and needs, based on observing their behaviour.
Why do a shopping study?
There are many situations where a shopping study might help you, but here are the two main ones I use them for:
1. Learning what to provide on a website
What you observe in usability testing sessions will mostly tell you what participants can do with your website and less about what they will do. It mostly helps you spot usability issues to fix.
It also forces participants to solely use your website and this is not how people use websites in real life. In real life, people use the internet rather than a website. In reality, they jump from one website to another and back, sometimes spending only a few seconds on some of them. The websites they encounter are part of a wider experience that few companies spend much effort learning about. If you learn about this wider experience, you can learn:
- What content people look for on your website when they are actually in the market for your product.
- What content/websites work better than others and why.
- What turns them away from your website and/or others
It basically teaches you how to make your website work better for the people you’re trying to sell to. It also gives you a more realistic view of how small your website is in the bigger picture.
2. Learning about the wider experience outside of your product
When the product you work on, helps with a wider job or task, then it’s helpful to learn about that how that wider job is done. You can ask participants about how they do it, but it’s usually more insightful to watch them do it instead.
At Skyscanner I learned a lot about how people plan travel and how the design of the product could be improved, by watching people plan trips using other websites.
I didn’t really call this a shopping study at the time, but this is when I first started using this approach.
It’s still not real
It’s worth pointing out that while your sessions will often be more authentic than a usability test, they are still going to be forced to some extent. Like every technique, it comes with its limitations and biases.
I like to think of all UX research techniques as giving you clues about what’s really happening, rather than giving you the absolute truth about it.
Step 1 — Find people in the market for your product
Let’s say you work on the website for accounting software, targeted at small businesses. You want to know how to answer the questions that potential customers, have when they visit your website. You also want to know how to show your product in the best light.
For this, I would find people who are in the mindset of changing their accountancy software or signing up for the first time.
But how do you do that? Here are a few ideas….
A. Recruit people from SEO landing pages
You can use a product like ethn.io or HotJar to recruit and screen people from your SEO landing page or your website. Most of these people have proven with their behaviour that they are likely to be in the market for your product.
B. Placing ads to find people
You can place ads on Facebook/LinkedIn for the product category you’re interested in then recruit from the page the ad leads to. Again you can use ethn.io or HotJar for this.
C. Social media and web forums
You might also be able to find people in the market for your product, on certain websites, social media and forums. Just be mindful that it’s best to approach people who have shown themselves to be in the market, rather than just asking everyone if they are. For example, people who post up asking for recommendations on accounting software, have shown themselves to be in the market for this product. They would make good people to approach.
D. Catch them when you can
Instead of running a time-boxed study, you can have your shopping study as a more passive background work. Whenever you come across someone who is in the market for your product, try to line up a shopping study session with them. Do not discount friends, family or colleagues from this. The benefits of capturing people who are actively in the market/need for your service, will usually outweigh the biases/limitations of your aquaintance with them.
E. Finding people from panels/recruiters
For more ubiquitous product decisions, you might be able to screen people from a panel such as UserTesting, UserInterviews or your research recruiting agency if you use one. This way you can initially mask the name of your company. I’ll talk about the benefits of that later.
But most of these participants will be less engaged in this decision. This is because they’ve essentially stated they’re in the market when asked, while other approaches target people who’ve proven this with their behaviour.
For many jobs/purchases, these panels and recruiters will give you nothing but ringers, who are claiming to be making the decision, just to get on the session. It’s best to only use this recruitment approach for very common situations (like booking a trip).
How to screen participants
When screening participants, avoid questions with yes/no answers. Instead give a range of possible answers, so the qualifying response isn’t easy to guess for people who just want the incentive. Even if you’ve targeted your invites to specific webpages, some people will be there for random, unrelated reasons.
Hide the company name from the recruitment if you can
As much as possible, you want to watch how people research their decision, without them being biased by your presence. But it’s also good practice to tell participants which company is carrying out the research. Doing so upfront can bias things. Instead you can explain to participants that the name of your company will be revealed later in the session. This means trying to hide mentions of your company until you’re ready to tell them.
If you’ve recruited from your own pages, they’re going to know who the researching company is. It’s best to just accept this and try to mitigate against this bias in your moderation of the sessions.
Step 2 — Schedule the sessions
When you find the people you need, schedule them into one-on-one sessions, as you would for a usability test. You’ll probably need to make those sessions at least 60 minutes.
The processes you already use for scheduling usability tests apply here. If you’ve never run a usability study before, then David Travis’s usability testing course course on Udemy is a great place to learn.
Remember it’s a good idea, if possible, to withhold the name of your company until a point in the session where it no longer introduces a bias. But you should always tell them at some point. Let them know that you’re going to withhold this information before they agree to participate, so they can agree to take part in an informed manner.
Step 3— Running the sessions
In the sessions, start with some warm up questions, just as you would in a usability test. Then ask them about the the job/purchase they need to do:
“You said you were looking for some accounting software for your business. Can you tell me a little more about that?”
You want to establish a few things with your follow up questions. Some of these things will be obvious or irrelevant depending on what you’re researching.
What is their role in the decision taking?
They might be the only person involved in this decision, but there might be other people involved. If there are others involved, you want to get a sense of where they fit.
- Some people are researching, but not taking the decision.
- Some groups break different decisions for different people
- Some groups split up then come together to make the decision
- Some people claim to be involved but when you dig deeper, you find they leave it all to someone else.
What stage are they at?
You’re going to be asking them to continue their research from where it left off. To do this, you need to establish where it currently is. Many participants will have the entire things as a ‘back-burner’ task that they’ve never gotten round to starting. That’s fine too, but you just need to know that’s where they are.
Establish your neutral position
You need them to know that you’re not specifically interested in which option they choose or which websites they go to. Instead you want to watch how they go about researching the decision. If they know which company you work for, you need to back these words up with your actions.
What’s their current thoughts on what option to choose?
Which options they’ve considered already? What have they found out and which option, if any, is the current front-runner?
Despite this, many of them will still try to be kind and favour your option if they know who you’re working for.
“OK, now pick up where you left off”
Tell the participants that for the rest of the session, you want them to just pick up this task where they left off. They should use the time to make progress in their research. They should make the time work for them.
Interventions will usually be necessary
Just now I’m making this all sound easier than it might turn out to be. Your intervention will often be necessary in order to make the most of the time.
Remind them they can move on
When participants get bogged down in a site, it’s useful to give them the option to move on. I usually ask if they want to keep on looking at this website or look at another. This reminds them they are in charge and can trigger them to move on. But it can also sound like you want them to move on, so introduce this idea with the impression that you don’t care either way.
Have a shortlist of ideas
Some participants might have been putting off this job so long that they haven’t even gotten started. You can choose to fast forward things a little by giving them some ideas. These should be the popular products. You can include your own in here. I always try to make it sound like I’m trying to think of the names and I’m saying them as they come to mind. This is to try to stop them thinking that I care about any of the websites specifically.
In our accounting example I might say “Why don’t we start with… let’s say.. Xero and see how we get on”.
Pressing fast forward
Some participants might get involved with research which is isn’t going to be useful to you at all. To avoid spending the hour watching them do this, you can hit fast forward and say “Let’s say you’ve done a bit of research and come up with some contenders”. You’ll then need to give them the contenders.
Like I said earlier, you are always trying to balance authenticity with usefulness and sometimes you need to sacrifice authenticity a bit.
The benefits
A shopping study can show you what you need to tell people when they are thinking of buying your product.
Hypothetical needs ≠ real needs
It can dispel your assumptions about what people actually care about. In a usability test for example, your participants can be trying to find information that they hypothetically believe they would find important. But when you watch people who really have the need, those same things can be entirely overlooked as unimportant.
What happens before users arrive
It can also enlighten you about what’s going on before people even arrive at your product. At Skyscanner, I was able to categorise different types of trip planners, by the workflows they went through. We could then be aware of those different types when making design decisions.
The importance of aesthetic
Many UX research techniques fail to highlight the importance of the aesthetics of a design. But shopping studies often show the impact very well. This is because you notice people, picking up signals, rather than necessarily searching for facts. Cluttered and ugly designs tend to fair badly in shopping studies, compared to more contemporary, fresh designs.
I have a term I use for software websites that I call the “Monday effect”. I discovered this in shopping studies. This is where people looking for new software, tend to favour products that looks fresh and well-designed, even when they don’t really know how well it works. They form impressions based on screenshots of the software. It often doesn’t need to be obvious what’s happening on the screenshot. It just needs to look nice and look easy to use.
Generating usability test tasks
When you see for yourself what information people are trying to find in a shopping study, you can use these things as usability test tasks in further testing. You’ve found out what people want to do, then you find out how easy that is to do on your website/product.
Shouldn’t I just do a diary study instead?
A diary study is a more established way of understanding these experiences and has some benefits over a shopping study. However it also has limitations. Here’s a few of each that spring to mind
Benefits of diary study
- You aren’t forcing participants as much to do the effort within as tight a time frame. However, some forcing might still be required
- You can see how the research/job plays out over time. This is what people call a longitudinal study. But again, there’s still a limited time frame.
- You can pick up on offline interventions and activities that occur, like conversations with trusted people and the impact of offline adverts.
Limitations of a diary study
- They take a lot longer to run and there’s loads more data to sift through, so it’s a lot more labour intensive.
- Drop out rates are a lot higher.
- It’s mostly self reported behaviour, rather than observed behaviour.
- It can be a lot harder to reactively direct the conversation.
- They are a lot more expensive to run.
Remember to triangulate
Any technique you use will have biases and limitations.
You should always try to look at the subject from different angles and use different data sources. For example, you can also speak to people who became customers and hear about their experiences. From doing so, you’ll notice some of the blind spots created by both of those pieces of research. Use this to create a clearer picture of what’s going on in reality.
Give it a try
If you think you might get some value out of a shopping study, then give it a try. If you do, feel free to connect with me on LinkedIn and let me know how you get on.
About the author
I’m David Hamill. I help organisations take better decisions through lean but meaningful UX research. If you liked this post, you can read some more below. I also post my thoughts about UX on LinkedIn.
Design
- Is that really a user problem?
- Design guidelines for mobile date-pickers
- Getting real about delightful design
