Can A Founder's Charisma Lead to False Positives?

How to direct users to react to the product and not the interviewer in user tests

Today I met with a founder who is getting too many false positives in user tests because she's too charismatic.

Wait, is this a real problem?!

Actually, it's a pretty common problem. Founders tend to get ahead by being fantastic at pitching. Many founders have an aura of charisma and magnetism, especially about their product.

As a result, people who talk to them about the product can get socially pulled in, but then fail to adopt or quietly churn off once they discover that the product itself doesn't deliver on the energy of the conversation. And in user tests, it means they say Hell Yes without actually needing the thing.

The problem is that, even though test is theoretically "objective", there has to be a dialogue portion—especially if the test involves interviewing, or if it's an enterprise sales call. And if the founder has to explain the product to the user at any point, well, then it's really all over.

The Challenge of Accuracy in User Testing

The biggest technical challenge of accurate qualitative testing is getting the user to respond TO THE PRODUCT instead of SOCIALLY TO THE INTERVIEWER.

This is the double edged sword of qualitative testing: Get true responses TO product? It's gold; a tiny amount is insanely valuable and gives you tons of direction and depth.

Get social interaction ABOUT the product? It's totally useless and actively misleading.

So how do you get the user to respond only (or primarily) to the product while the interviewer is in the room?

Tips to Get User Responses About the Product

1 - The Prototype

Create a prototype that does not require the interviewer to say a single word to explain what it does. Chiefly, this involves creating a sort of "fake" explainer page that you wouldn't actually have in a live product and putting it before the main experience.

For example, you can have an app store mockup with fake rating/reviews and info, and then a 1-2-3 how it works page. The point is to proxy the research and onboarding they would have done to get to the product quickly in a single handout or two.

2 - User Testing Script

Block out the first half of the test for pure user-led interaction with the product, without (much) input from the interviewer. Have the user talk out loud with their authentic stream of consciousness while they do this.

Set the user's expectation in advance that you won't be allowed to talk to them during this portion (other than to occasionally say "pause" and make quick changes, or flip through screens / prototypes), and that you'll have a dialogue in the second half so that they don't feel like your distance is off-putting.

Be really warm and effusive at the beginning to welcome them, but DON'T explain the product or interview at all. Then, during the test, use a neutral tone and maintain social distance.

(Comment for exact scripts for this.)

3 - Pay in Advance

Try a test sometime where you pay a low-income user before the test, and then after, and watch the difference in their feedback quality. Seriously.