Testing against competitors and keeping touch with reality

I recently tweeted an online retailer to tell them how they just lost my business to their big rival, simply because the latter had made the checkout process more simple for me.

I’d had the same product in my basket (cart) on both sites. The winning retailer had a checkout with Paypal button as well as the standard checkout button. While when I clicked the standard checkout button on the losing site I was prompted for a password. So I just checked out with PayPal on the other site instead.

But we did user testing

I later found out that quite a bit of competitive user testing had been carried out to define the losing (in this case) experience. So how can this happen? Surely if you design one experience in comparison to another you should be able to make it so it beats your biggest competitor? Especially when you test both approaches with users.

Yes but did it simulate reality?

My hunch is that the manner in which the usability testing took place didn’t account for how cut-throat and fickle internet users can be when it comes to parting with their time and money. I doubt for example the testing involved giving users the choice which checkout process they were going to use.

Instead participants might have dutifully completed both processes and then told the facilitator which of the two they felt provided the best experience. Participants may well have been timed on task also. I could be wrong, this might not be what happened at all, but this kind of thing does happen.

This type of testing isn’t really tuned to reality. People don’t buy a product twice and then cancel the order for the site which provided the worse experience. Instead they favour the site which looks like it’s going to provide the best price and (in circumstances like the one I was in) experience. Only when the process gets tedious or a hidden charge jumps in might they jump ship to a competitor.

Realisation of this crept into my research process a while back. I used to ask people to loyally use our site in user tests and I would look out for difficulties they had. But in reality, someone who wants to book a flight has the internet at their fingertips.

If we dick our users around, they can drop us in an instant. So I often give participants the remit to do this also. Since doing so, I’m a lot more aware of the UX fight than can take place when trying to convert users.

If your competition is just a click away then perhaps you should do the same. Start your users with a blank screen and give them permission to use as many or as few sites and apps as they like. Then see what happens.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store