Interviewer Bias In User Research & Steps To Conquer It

Interviewer Bias In User Research & Steps To Conquer It 1

Context is powerful. If you’ve lined up an interview with someone because they match a set of selected criteria, you’ve already made the decision that, on some level, you know that person.

Whether it’s for research, journalism or a job interview, that pretext is the lens and construct through which you’ve chosen to understand and empathise with that person. And interview bias is that lens, or at the very least its outcome.

But even without a context for a discussion, or a strong ability to put bias aside when starting an interview; a first impression will do most of the work on its own.

In truth, by actually talking to the person at all you are already fighting an uphill battle against bias within seconds of starting the conversation. Bias that you as a user researcher can’t afford. Bias that as a journalist might diminish the truth in your article. And as a manager could cause an imbalance in the workplace.

In this article I’m going to talk about interviewer bias in both its technical and non-technical forms; as well as tactics for professional interviewerrs to circumvent some of the damage that their own bias could be causing.

What Is Interviewer Bias, Actually

Let’s first look at the technical description of interviewer bias. The Oxford Reference database (as operated by the Oxford University Press Association) defines interviewer bias as follows:

[Interviewer Bias] is a distortion of response related to the person questioning informants in research. The interviewer’s expectations or opinions may interfere with their objectivity or interviewees may react differently to their personality or social background. Both mistrust and over-rapport can affect outcomes. 

Interviewer bias – Oxford Reference Database

Now what this definition references from a practical point of view is the potential for two separate forms of interference with the interview process:

  1. Any number of actions and behaviours made by the interviewer that cause the interviewee to react and respond in ways that conform with the interviewer’s bias. Examples of this could be:
    1. using certain language, phrases or leading questions 
      1. outwardly expressing one’s own beliefs or answers to questions
    2. using certain vocal tenor, or inflections to imply a presumed answer
      1. Pretexting questions with implicative phrases, such as “I’m sure you know this already but…”
    3. using non-neutral body language that establishes a mood or projects onto the conversation
      1. dressing or provoking intrigue through appearance or presentation
  1. A prejudiced perspective on data from an interview or interviews. Examples of this could be:
    1. dismissing a person or data from a particular interview based on your personal impression of the interviewee
      1. Gender or racial bias, etc.
    2. not recognising or dismissing patterns and key insights from interview data as it fails to align with your own point of view
      1. highlighting or finding implied patterns and insights in data that align with your point of view even if not explicitly in the data
    3. not reviewing interview data at all and instead relying on your memory or impressions from an interview

Now within these practical states of interference, and therefore distortion of outcomes, we have implied a long list of potential bias types. I’ll talk more about different bias types later in the article, but the overarching construct of implied social bias is perhaps best defined by the term “halo effect”.

To continue to quote the Oxford Reference database…

[Halo Effect] refers to a common bias, in the impression people form of others, by which attributes are often generalized. Implicitly nice people, for example, are assumed to have all nice attributes. This can lead to misleading judgements: for example, clever people may falsely be assumed to be knowledgeable about everything.

Halo effect – Oxford Reference Database

As if we didn’t already know, the halo effect and the studies around it are confirmation that regardless of who you are, first impressions are an ever present, ever powerful influence in our lives. 

It’s impossible to really say how much time a first impression takes; with study findings ranging from 5-30 minutes for a job interview, to 1/10 of a second for social interaction.

The potentially dangerous part of these discoveries is that we have no control over it. Humans have, in the general sense, failed to build a social mechanism that hinders our instinctual biases. And because humans consciously reflect on the past, in the way other animals do not, it takes much longer for us to reprogram a first impression.

So if you literally cannot help yourself from having bias in an interview, what can you do?

How To Identify Interviewer Bias In Yourself

The first step in identifying interviewer bias in yourself is accepting that you have interviewer bias at all. 

To be able to work with your interviewer bias, you’ll need to fast track through the 5 stages of grief: through denial and anger, past depression and bargaining, and right to the end: Acceptance.

If you can accept that you yourself are agist, racist, gender biased, homo/hetero-phobic and have any other worldy combination of socially biased perspectives; then you have a good platform to build from. 

Because the sad part of this world is that we all have these biases, we all use stereotypes and generalisations to try and understand, judge or empathise with people when we interact with them. But if we don’t accept that we might have these, either through ignorance or claiming to stand above them, we are most prone to suggestions and actions that are in the best case a mild bias, or in the worst case, offensive.

As an interviewer, and particularly a researcher, your role is to ask questions and listen to your interviewees without judgement; something that you have to accept is nigh on impossible, before you can learn to work with it.

How To Identify Interviewer Bias In Others

For this article, when we refer to “others” we’re really talking about your colleagues.

Whether it’s other people who do interviews, editors and clients, people that listen and take notes, or even people responsible for transcribing or coding data during analysis.

In most research work environments, identifying bias in others can often be very difficult. In fact, you’d often hope it’s difficult, as overt bias early on will have other implications.

For example, if a research project is designed around, influenced by, and makes no effort to challenge the overt bias of a client, the results of that project will often only confirm what that client assumed to know anyway. The client may then disregard the work as a waste of time/money, and refuse to acknowledge research as an important part of their process. These situations are common in industry research; a kind of self-fulfilling prophecy for clients who feel this work is unnecessary.

But the bias of people involved in the project may not even appear until the very end of a project, wherein the only real outcome is what one person has been saying all along. If this has happened and the data to support the claim is strong, then while the work may have missed things, it at the very least confirmed an assumption. However, if the data to support the initial claim is weak, but remains the only real insight from the work, it’s likely the project was off the rails from the beginning.

In point of fact though, as researchers we are often trying to suppress our own bias – partially, fully, overtly or not. The respect most people have for the scientific process means that they attempt not to judge too much. And this type of bias, this subtle influence of our assumptions is extremely difficult to see in colleagues until it’s too late; in fact, if you ever even see it at all.

So what can you do about it?

Working With The Bias Of Colleagues, Friends And Clients

There are a number of ways to insulate qualitative work from the influence of your colleagues.

A tactic that some researchers use, and is now a standard process for the research ops team at Interviewerr, is a simple extension of what you will be doing anyway as a researcher: you interview people.

Whether it’s your colleague and a client; a team and company, or you alone and an overseeing professor, the first interview you design for a project is an interview for you and your colleagues to identify, notate and hold accountable your own biases and assumptions about the project.

It doesn’t take much, or need much, but even if you’re writing questions for yourself and getting a roommate to ask you them; the best way you can counter your own bias is to use your interviewing skills to weed them out.

At Interviewerr these biases and patterns found are recorded, noted, reported to everyone involved, and then checked during each stage of the project from script development to analysis and reporting.

Sometimes it’s as simple (or perhaps as difficult…) as saying: “We know this is a bias in the team. Is this question informing an assumption, or leaning into confirmation instead?”

Working With Your Accepted Biases During An Actual Interview

If you’ve accepted that you have biases, there are three acts of self awareness you can use to improve and monitor your interviewer bias.

1. Maintain a state of active self review.

While reflective self review is, well, reflective, and therefore “after the fact”; active self review is a state of constant consciousness and self evaluation.

Although arguably the most effective tactic for improving your biases in life, this is very difficult to do at all, much less during an interview…

As an interviewer you’re juggling a lot, everything from the schedule, time and duration, to the comfort of the interviewee as well as any colleagues in the room, your script, the notes you are taking, etc etc, etc.

As part of that dance researchers tend to be, on average, quite self aware. But can you do more? Can you do better? Can you consciously analyse and control your own haptics, your voice inflection, your facial reactions, even the wording choice of your questions?

This is not in an attempt to change who you are, but to identify traits, words and phrases in yourself that may (intentionally or not) trigger specific reactions in your interviewee. The ability to do this well is both a power for good as well as evil as it’s the root to manipulation as much as it can be a root to empathy.

Regardless of your own abilities, there are limits to everyone’s ability to do and improve this set of skills, so it’s good to have a vehicle for this to work within, thus our second act…

2. Quite literally play a role.

Playing a role means to create a character or persona for your interview that you adopt and embody for its duration.

It is, in short, to be an actor on a very small and intimate stage, wherein your goal is not to make the audience laugh or cry, but instead to allow them to be entirely and completely who they are and experience how they feel about things without your interruption or interference.

Now, this character could be designed in your current self image, or reflect what you feel to be your best properties. But we find it works best when you can use the first act (active self review), to identify your biases and then act the part of someone (you) who has no such biases.

Now many of you may think, “I do that anyway!” And if so, that’s great! I would hope so.

I want to emphasise it here because in our evaluation of investigative conversations between mentors and subjects, interviewers and interviewees, we’ve seen that making the conscious decision to play a role often means avoiding the compulsion to represent your own truth (due to ego or habit). It also means you don’t feel like you’re lying because you are playing a role for someone else’s benefit, not yours.

The trick in this work is not to lie about who you are, but to not let who you are prevent someone else from representing their truth. So making the conscious decision to play that role allows you to put aside your instincts and feelings and be fully present for your interviewee.

3. Add an additional layer to your analysis process wherein you start by redefining your hypothesis.

Unlike points one and two, the third is a very quick framework for researchers to use pre-analysis or even during one-off interview debriefs.

The concept is simple. Before starting the analysis of interview data, particularly the tagging and coding of raw audio data, take 30 minutes to do the following with anyone involved in the analysis process.

  • Review your original hypothesis for the study or interview(s)
  • Write down 1-3 assumptions on what you’ll find/discover about each hypothesis
  • Write down what you think you absolutely won’t find about each hypothesis
  • Write down a list of things you know you learned from interviews that you didn’t previously know about the topic
  • Write down the number of things you hope to find in the data that would be new insights for you
  • Formulate additional drafted hypotheses of things you don’t expect to find but answers to which could, maybe, possibly, be hidden in the data

Now I’m going to call myself out here. Many researchers would argue that the above process could potentially lead to a close-minded view of the data. Instead of removing bias, perhaps this process enforces it?

In short, this checklist is a way of consolidating the risk of bias to a short process at the start of the analysis process as opposed to opening the entire analysis process up to the risk of bias throughout.

I’m sure a number of researchers will have comments and arguments about this concept, and I hope you’ll all contribute to the comments and help evaluate, test and adjust this process!

The Crux

“Am I creating bias in my own research?” is a question we get all the time at Interviewerr.

In fact, in a study we did with 60 researchers, at least 10% of researchers brought this up as a concern they had in their work, independent of the questions asked (which were not on that topic).

In fact, I personally believe that if you haven’t asked yourself this question, you may be missing entire areas of discovery in your work!

However, this concern we have for our own bias is something we should learn to address constructively. Researchers who experience this doubt and struggle to justify the results of their research need to be sure that their results were not primarily driven by their own ideals.

And that’s ultimately why we wrote this article. The point is not to sell a fixed ideal of bias-resistance research, but rather to present options that may help some of you overcome the barriers to achieving confidently unbiased outcomes.

Regardless of whether or not you agree with the suggestions we’ve made, I hope you’ll contribute to the comments and help us find the best way forward.

Thanks for this article goes out to the entire Interviewerr team, our beta community, and the teams at Growth Mechanics and Silicon Rhino for their knowledge contributions.

Leave a Reply

Your email address will not be published. Required fields are marked *