We are terrible at picking which people will become great co-workers in the long run. We review resumes and look for relevant experience, while combing through overstated or poorly explained projects. We attempt to test technical competence which took years to build through a few hours of observation. We ask for references, and pretend that feedback from specific named sources is remotely unbiased.
Beyond attempting to see if a candidate can accomplish tasks, we also need to make certain they have the right leadership qualities for our position.
Me - So what's the main reason you'd like to transfer teams?
Employee - I can't work with co-worker anymore. They're constantly screwing up. They never admit that it's their fault when their code breaks, then they fingers at their co-workers who missed their bugs. In general it's making life on the team miserable.
Employees need to be able to get along with co-workers, graciously admit mistakes, give credit when it's due, and at the very least, not make the work environment worse through their presence.
Finally, we need to accomplish the above without mistakenly bringing personal bias into the process. Beyond illegal bias (e.g. gender, or race), we need to be careful about our biases regarding education, grammar, the types of companies someone has worked at, and someone's choice to spend the previous year surfing instead of working. We are all biased in a thousand ways, and we need to be cautious about which evaluations are valuable, and which are not.
Interviewing is hard. It takes dozens of interviews to feel remotely competent. It takes hundreds of interviews to begin to feel some aspects of mastery. Amazon invented a bar raiser position to bring mastery and experience into every interview loop.
What is a Bar Raiser?
A bar raiser is an experienced person added to an interview loop to ensure that it does not go horribly off the rails.
Stated more optimistically, a bar raiser is responsible for ensuring that a candidate has a great experience, and Amazon makes a great hiring decision.
The bar raiser is an experienced leader. Their common sense and judgment is trusted by their leadership team, and they have the responsibility to correct or modify elements of an interview process. Rather than a checklist of responsibilities, a bar raiser impacts the entire process from end to end. It is not a clear defined role, rather it's a leadership position which requires an understanding of what ownership means.
Pre-Bar Raiser
Prior to a Bar Raiser being involved, a few steps have taken place.
A resume has been screened to start, because generally you would not conduct any type of screen without knowing a candidate's background. Often these days you're getting a better formatted version of someone's LinkedIn profile.
After a resume has been flagged as interesting, someone conducts a screen. The primary purpose of this screen is to validate that a full interview should take place. This is for the benefit of the candidate as well as the interviewers.
Recruiter - Hey, your interview is next, right Dave? You're the third interviewer in the loop?
Me - Yeah, why?
Recruiter - Something went wrong with the initial screen. The candidate did terribly in the first interview, and are bombing the second interview right now. The current interviewer says the candidate has started to cry.
Me - Ok, I'll talk to them and see if they'd like to leave early.
No one wants a candidate to have a terrible time. It's one thing to make a mistake or two while interviewing. We all regret the answers we give in the spur of the moment, and I've received dozens of emails from candidates after interviews explaining that they have a much better answer now. However, interviewing for a position you're clearly not qualified for can be a terribly embarrassing experience. The screen, often done over phone or video chat, is a way of ensuring that the candidate will be prepared.
One way of phrasing the qualification is that we say that a candidate should not have all not-inclined in their full loop. If we believe they have a shot of being hired, and would likely pass one or more in-house interviews, that should be considered a pass. A screen is not a "I think we should hire this person", it's a "I think there's a chance this person could pass an interview" decision.
Pre-Interview
Once we’ve verified someone is generally qualified, we setup an interview. Interviews vary in size, but we often have five interviewers on a loop. One of the interviewers would be the hiring manager, one interviewer would be the bar raiser, and three would be there to help assess the candidate. Each interviewer is assigned what are called "competencies” by the hiring manager to prevent too much overlap as well as ensure we assess as broadly as possible.
As the loop is being set up, the bar raiser surveys the competencies to ensure that they're the right ones for the role. Hiring managers are often not as experienced as the bar raiser on the loop, so the bar raiser often acts as a consultant to help ensure the right things are covered.
Me - I see in your competencies, you assigned "culture fit" to one of the interviewers. Could you explain what you intend by that competency?
Manager - We have a great team culture. We hang out after work together. We sometimes play sports, sometimes go out drinking. In general I want to make certain future team members will fit into our group.
You can't filter hiring based on someone's interest in drinking, how much fun they'd be to hang out with, or if they'd fit into your team culture. The bar raiser is responsible to watch for incorrect or risky competencies being assigned, and help educate the hiring manager on what we do and don't screen for. Even if "culture fit" sounds innocent, it has a number of dangerous side effects.
<The remainder of the article below is only visible to free or paid subscribers. Subscribe for free to read more!>
The bar raiser also looks for potential mistakes in the loop setup. For example, an interview loop for a level 7 manager, utilizing an level 6 manager as an interviewer. Or perhaps a coding competency assigned to a manager. They would also flag a risk if the interviewers are all relatively new to Amazon. When in doubt, the bar raiser works with the interviewers and manager to ensure that the loop makes sense.
During the Interview
The bar raiser is often rumored to be the hardest interview. The bar raiser is often the most experienced interviewer. This should not add difficulty, it just means they will be more consistent. They will work to ensure that the candidate is having a great experience, and that the candidate has an accurate evaluation.
They will make certain they connect personally with the candidate, giving a little background on themselves. They will ensure the candidate has a break if needed, and time for questions at the end of the interview. They will explain the next steps for the interview loop, and clarify any questions the candidate might have around process.
Me - How are things going so far? You're 2 interviews into your loop right?
Candidate - I think I screwed up that last interview. They asked me what a binary tree was. I know what they are, I just completely mind blanked and didn't answer the question.
Me - Don't stress. There are 5 interviews for a reason. A single bad interview doesn't break things. Just take a deep breath, drink some water, and we'll have a lovely chat. No need to worry. Heck, why don't you tell me about binary trees briefly, and then we can move on.
Anyone with a few months of experience can make a hard interview, it takes hundreds of interviews to make a comfortable experience which pulls out critical details about someone's work history and abilities.
A bar raiser works to be efficient with their time. They will move a candidate along if the words they're saying are not useful. They'll ask probing questions to get at the important details of the competency they're questioning. They get as many data points as they can in the short period of time available. If they are comfortable with a competency, they move on. If they need more details, they'll probe decisively.
Me - Tell me more about how you gave that employee feedback?
Candidate - I told them that they were underperforming.
Me - Give me an example of what you might have said to indicate that they were underperforming.
Candidate - I said something like 'On that project, you missed the due date.'
Me - Did you say anything else? Did you literally say just those words, or would there have been more?
Asking clarification questions and probing for details can ensure that an experienced interviewer can get the data points necessary, preferably without leading the candidate to the right answer by mistake.
If possible, a bar raiser tries to make a candidate feel as if they've done well in an interview, even if they haven't.
Me - So explain why you chose to use a triple nested loop to solve this problem?
Candidate - Computers are fast these days, efficiency doesn't matter.
Me - Oh I see, that's interesting. Thanks! Lets move on to my next question.
There's no advantage to someone feeling like they're failing, and it can impact someone's performance if they begin to lose their confidence. Even if someone is repeatedly doing poorly, it is the job of an interviewer to ensure that no one has a miserable time. It's stressful enough to interview, no need to make it worse if you can avoid it.
A bar raiser is also expected to take excellent notes. As a role model for other interviewers, they're expected to take high quality notes on the questions they've asked, the answers, and their own evaluation of those answers.
Post Interview - The Debrief
After all the interviewers have conducted their interviews, a debrief is held. In this meeting, everyone reads each other's feedback, and makes the hiring decision. This meeting is run and owned by the bar raiser. The bar raiser is here to ensure that the right decision is made, as well as act as a coach and mentor for interviewers to ensure that they are all becoming better interviewers over time.
The bar raiser holds a high standard for written and verbal feedback, and will give coaching to interviewers who are not meeting expectations.
Once everyone has read feedback, the bar raiser facilitates a discussion between all the participants. The goal is to have the participants come to an agreement on what areas the candidate did well, and in which areas they present a risk. When in doubt, the bar raiser helps guide the discussion through the role descriptions, leveling guidelines, leadership principles, and other sources of truth to understand where the bar is, and how the candidate performed.
Interviewer - I'm worried about their ability to move at our fast pace.
Me - Sounds like you're talking about bias for action. What specifically in the interview lead you to believe they'd have a challenge with having bias for action?
Interviewer - Just the way they answered questions, they were slow.
Sometimes we obtain valid data points on a competency. Perhaps a candidate says that they hate to make decisions quickly, they prefer to think for a few weeks before making any decision. Sounds like they might have a challenge at a company like Amazon. Other times, our personal biases show, and we get an impression of a candidate which may not have data or facts supporting it. The bar raiser needs to separate facts from feelings, and valid data points from conjecture.
In the end, the hiring manager and the bar raiser both need to agree to hire a candidate before they're given an offer. If either of them do not want to hire the candidate, the candidate is not given an offer. The bar raisers are known for having 'veto' power, but in reality it's about the hiring manager needing to partner with the bar raiser to make the right decision.
Results of the process
Even when we've conducted an excellent interview process using experienced interviewers, success in the workplace is not guaranteed. I've seen candidates who barely made it through the interview process turn out to be star employees within a few months, and those who flew through the interview process with flying colors end up not being successful.
I've also seen successful employees on one team at Amazon move to another team, only to find themselves struggling. I’ve seen an employee unable to be productive on one team, move to another group and become a top performer.
Success is a magical mixture of how well you mesh with your co-workers, your relationship with your manager, your work (how interesting you find it, how well it utilizes your talents), your personal life (are you getting enough sleep? Stressed at home or energized?), and a myriad of other factors. It is impossible to predict where and when someone will be successful.
