User testing best practices 1 / 2

Do you know this typical story of a product?

Somebody has a great new idea, writes down initial product plan & requirements, draws high fidelity designs and then quickly move forward to develop an MVP (which unfortunately often takes few months) ... and as time flies by, they are stuck in a vicious “builder’s cycle” where they have to quickly design new features because developers are waiting for them and on the other side you have to develop new features because they were planned in the initial product plan.

Errrr … typical highway to hell 🔥

Important piece of a puzzle missing in this story is - user testing! Any kind of user testing.

• Before defining product requirements we should map the typical user journey at least with few potential users

• Before finalising the designs we should confirm our proposed product flows through wireframes testing

• And definitely before starting any development we should receive feedback from real people on our clickable prototypes

So what are the best practices regarding user testing and how to avoid common pitfalls? From my experience as a product manager, there were usually following 3 typical concerns connected to user testing in tech companies:

1. What kind of user testing method should we use?

2. Where should we recruit them from?

3. How many users do we need to recruit?

Since it is a pretty broad topic I will split this article into two parts. First two questions will be tackled in this blog post and the 3rd will follow in User testing best practices (2/2).

What kind of user testing should we do?

There are many testing methods, based on the phase of your product lifecycle and the goal of your testing. But for me the most valuable were always live moderated testing sessions (either onsite or online). During a live session you can directly discuss with users their behaviour and their intentions, go into necessary details and thus get a broader context of their actions.

It is usually also the most costly and time consuming approach, but I would always prefer to have a 30 minutes discussion with 2 users than receiving 20 survey responses or seeing 20 screen recordings.

The value of user testing is not so much in the hard facts or particular answers, but rather in understanding user's empathy with your product, motivation to use it and those “soft factors” that can be gathered only via direct discussions with real people.

You don’t need to have a full fledged UX lab with a mirror wall to conduct live user testing. It is totally enough to schedule a video call with each tester where you can ask them to share their screen in order to observe and record how they interact with your product, prototype or wireframe. And you keep it short! 30 minutes should be enough for a skilled product manager / UX researcher to test needed product parts and avoid so called “survey fatigue” when testers become bored or tired.

Moderated video calls also allow you to have unlimited internal observers with you in the room or in the call, so that they can see the user interaction with their own eyes (importance of which I will mention later - as my most crucial lesson learned).

But take into consideration that any data collection requires user consent, so ask your testers clearly at the beginning if they agree with being recorded, introduce all other internal participants in the call and to comply with the motto of GDPR - data minimisation - try to collect as little personal data as possible. The best approach here is not to connect your testing notes with any particular user. Just take it all as general insights and it doesn’t matter whether they came from user A or user B.

There is also an option called “coffee-shop testing” which means to go anywhere among people (like a coffee house, coworking space, conference, etc.) and ask them to do a 10 minutes product review with you. This approach is pretty cheap and easy, however I will generally not recommend it since your tester should be as close as possible to the potential final users of your product. So some random coffee-place people can skew your test results, since they don’t feel those user pains that your product is trying to solve.

And this leads us to the topic of “how to lead a user testing session?"

Our goal is to understand as much as possible the perspective of our users without navigating them or influencing them in any way. Actually the best approach is talk as little as possible and let them talk at least 80% of the whole session. This implies that testing moderators are good listeners but this holds true for product managers in general :)

The worst thing are of course self-confirming questions like “Look at this hidden dropdown menu here! This can be pretty useful for you, right?” It seems like an extreme case but I’ve honestly heard such a question so many times. At least try to ask open ended questions like “What is your opinion on this dropdown menu below” or ideally do not navigate users at all and wait until they find and comment those features by themselves. If they haven’t found that feature or didn’t say anything about it, probably it was not such an impactful feature as you assumed.

The only way I like to navigate user behaviour is to stop them before some action (like clicking on the “Continue” button in an app flow) and ask them “What do you expect to happen after you click on this button”. This usually provide great insight into users' expectations and their understanding of the flow.

To sum it up, keep it simple and take the whole testing session as a natural discussion between two people - which really doesn’t require any extraordinary conditions other than a zoom call.

However, spend a proper amount of time on selection of your user base. They should be highly relevant for your product and the main criterion is whether they struggle with the same issues as your product is targeting. Ideally define your user segmentation before the testing so that you have a clear profile of testers to approach.

How to recruit your testers?

We are lucky to live in this age, where recruiting of relevant testers is really not that hard. You have plenty of paid tools & services that can help you with the recruitment process based on your specific user segments and for a reasonable price.

But before you jump into these external means, try to check your internal sources. If you already possess a database of at least a few clients or leads, go on and approach primarily those. You would be surprised how many of them will be totally okay to have a 30 minutes video call with you - mainly if your product brings some value to them. And there are also multiple options how to motivate them - from a freebie access to your paid product, up to a basic Amazon gift voucher, which btw. works really well.

In case you are based in the EU, be aware that you have to deal with GDPR rules before approaching your users. But if you have their consent to send them your marketing communication, you are fine. It is always good to include into your email collection consent & privacy policy also a clause about research purposes, but I doubt how many startups do think about this early in advance. Although general marketing communication consent is enough for testers recruiting campaigns, if there is a way how to opt out and you do not over spam them.

Another ways how to recruit testers from your current user base are for example

• Using HotJar pop-up polls or other live in-app forms of targeting users directly in your product with a question whether they are willing to join your research (ideally after some value adding moment like successfully finishing your flow in the app)

• Or using customer service communication to recruit your users after successfully solving their service case (e.g. at the end of a pleasant chat conversation, after you successfully upgraded an user to a higher subscription tier)

Well, in case that you don’t have any usable user base yet or you cannot approach them you can still count on one of the research recruiting tools like respondent.io, userinterviews.com or usertesting.com . You can nicely target there exactly what type of users you need to recruit - based on industry, demographics, or any other criteria that you would specify. From my experiences it costs on average $300-$400 to hire & successfully execute 5 highly relevant user testing sessions via these platforms. Which is definitely not so much money compared to the value it will get you.

And in the next chapter - User testing best practices 2/2 - I will try to explain why 5 users are totally enough for a user testing session and also how an ideal testing session should look like.