editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Definition In hiring, references are a candidate’s former colleagues or supervisors who can speak to the candidate’s skills or past job performance. Candidates usually directly identify a set of references for the company to contact; but for referral candidates there also exists an implicit assumption that the individual who referred them will act as a reference.
If you are a hiring manager, you know that feeling of uncertainty: A candidate may seem incredibly promising, perhaps even passed a technical interview with flying colors. But no one on the team has worked with them before. What will they really be like to work with? What will make them successful? In what situations have they performed well or poorly in the past?
References allow the hiring team to collect the information needed to bolster confidence about the hiring decision. Checking references is an essential part of hiring well and wisely. Unfortunately, prospective employers often conduct reference checks informally, rather than as an established part of the hiring process. When rushed or performed poorly, reference checks can produce misleading or unfair results. But there is an effective way to give reference-checking the attention it deserves.
caution It’s not usually a good idea—or thoughtful—to ask for references until someone is fairly far into the hiring process.** Checking references when you’re unlikely to hire someone is a waste of both the social capital of the candidate (who is asking a favor of the reference). and the time of everyone else involved.
dangerAlways obtain a candidate’s prior consent before reaching out to anyone at their current employer. Most candidates are job hunting without their current employer’s knowledge, and attempting to get a reference from a current employer could put the candidate in a very tough position or even get them fired.
When making important hiring decisions, companies of all types look for the most telling and predictive signals possible, and that includes the opinions of people who know the candidate’s work well. This signal is just as important for junior hires as it is for senior hires.
story “Many people treat references as just a filter or a way to potentially screen out candidates they may otherwise have hired. But references can be both a positive or negative signal. I once had a candidate who did OK on his interviews—no red flags, but definitely would not have been hired solely based on interview performance. I happened to closely know one of his professors, so I reached out and received a really glowing reference. I trusted that reference more than the interview feedback, so we hired that candidate and he went on to do really well.” —Ozzie Osman, lead author of this Guide; co-founder, Monarch Money
controversyInformal polls of people who have made many hires reveal that when properly considered, signals based on extensive past work experience are far more reliable than what can be obtained during a few hours or days of interview processes. But prevailing opinions about references differ wildly,** and the available research also isn’t conclusive.** One aspect of references that nearly everyone agrees on is that they are another point in the interview process where bias and noise can influence the outcome. For hiring teams, it helps to be aware of the challenges and to aim to make reference checks as structured as possible.
Most commonly, references serve four purposes:
To identify “red flags” serious enough to make the team decide not to move forward with a candidate. Mark Suster refers to this as seeking any “disconfirming evidence” that the hiring team might have missed in the interview process.
To identify the strongest or best-fitting candidate when the hiring team has multiple candidates who are otherwise equally qualified. (This is a lucky situation for the hiring manager!)
To gather more information on how to make the candidate successful in their new position. Hiring managers often overlook this purpose of references, but the resulting ideas can be especially valuable to manager and new hire alike.
To help in closing the candidate. This too is an oft-overlooked use of the reference call. Often, references will speak to candidates about how the call went, and a positive call can play in your favor: “I just got off the phone with Sonia—that role sounds really awesome!”
Candidate back references may provide the hiring team with confidence in a candidate early in the funnel. This is typically someone known to the team who strongly vouches for the candidate’s skill or fit for the role. Although this is a less common use of references, in some situations, it can convert a candidate who looks like an unlikely fit on paper (such as a graduate student or a person coming from another industry) into a more promising one who is worth advancing further in the funnel.
importantAs with any new meeting, it’s wise to treat every reference call as a potentially long-term relationship. For hiring teams, this means respecting and building trust with references just as you would with candidates. References can become part of your network: If the reference has common interests or related skills, this is a chance to get to know them and give them more information about your company and what excites you.
candidate Respecting and building trust with references also applies to candidates; this helps maintain lasting relationships that are not transaction-based alone.
In Who: The A Method for Hiring, the authors recommend that the hiring manager conduct four references and other team members conduct an additional three, for a total of seven. Two to four is more typical, however. Conducting only one will naturally provide a skewed perspective.
Almost any candidate who’s not fresh out of school will be able to give two or three professional references. (Note that when recruiting from colleges and universities, candidate references may be professors rather than former colleagues or bosses.) If a candidate is hesitant about giving references, it may be a sign that either they are not really interested in the job, or they have had trouble building relationships they can now draw upon.
While it’s tempting (and logistically easier) to ask candidates for references from previous managers, hiring teams will get a more complete picture by taking the time to talk to former colleagues. Harvard Business Review reported the results of a study that found that former managers are more likely to focus on whether a former employee met deadlines, was proactive, and was organized and efficient, while former colleagues provided insights into whether candidates were helpful, collaborative, and listened well.
candidate Candidates applying for jobs feel the stress of reference checks as well. You’ve done great (you hope) on an interview, and the hiring manager calls and asks you for references. Does that indicate how far you are in the process? Who should you pick as a reference? What questions will the hiring manager ask the references, and what will they say about you?
As a candidate, when you ask an old colleague or manager to be a reference, you’re asking for a favor. Usually it is gladly given, but the best references are often busy people. If you’re excited about a job and think you have a serious chance of getting it, tell the reference so that they know their investment of time will make a difference.
On the other hand, occasionally companies will stall for time with a candidate they’re not likely to hire by asking for references. Conversely, they may ask too early in the process, before they’re certain the candidate is a serious contender. If you’re afraid this is the case, it’s reasonable to ask your hiring manager or recruiter to tell you how far along you are in the process so that you can give your references a realistic idea of when they might receive a call.
If you’re applying for many positions, it’s also thoughtful to wait until you can narrow down the set of options so a reference doesn’t have to have more than a couple calls.
Hiring teams usually conduct reference checks over the phone. A few larger companies have in-house recruiters send forms or emailed questions, but these are usually impersonal and may be less effective for the hiring team than one-on-one conversations. References have a hard time quantifying or evaluating others’ skills using checkbox-style and similar form answers that can make it feel like an answer might (intentionally or unintentionally) hurt the candidate. A conversation between industry colleagues is more comfortable and can allow helpful nuances and details to emerge. Usually a candidate can make reference introductions to the hiring team, so setting up a call will likely be the easy part.
Although it’s not universal practice, the best person to do reference checks is most likely the person making the hiring decision. It’s even better if that person is also the candidate’s future manager.
caution At larger companies, references are often done by HR staff or recruiters, but they may not capture certain valuable information if they don’t know the exact needs of the role. Because HR staff and recruiters often are juggling multiple pipelines, they may be more likely to forget details gleaned from the reference process by the candidate’s first day on the job.
Good preparation for checking references includes taking a little time to research the reference’s background; this provides information to help put anything they say into context. Make sure you understand what their relationship to the candidate is or was. Plan what you want to discuss. What areas of the candidate’s experience can this reference speak to? About what are they credible?
The hiring team doesn’t need a fully prepared, rigid script, but will benefit from having a clear agenda and approach in mind, rather than calling off the cuff. Assuming the best of the reference will produce the most forthright conversation. Most people prefer to be honest, yet it’s helpful to expect that many references won’t want to speak ill of the candidate or hurt their chances of getting a job.
General guidelines to follow are to say hello, explain who you are, confirm they’re ready to talk, and thank the person for their time. Be friendly to set them at ease. Confirm how long they are available to talk—a hurried or interrupted reference call will be unproductive.
A reference call might look like this:
Establishing trust. (3-5 minutes) Ask if they’d like a very short background on you and your company and why you’re talking to the candidate. Be brief. Imagine you’re in their shoes. What would you want to know? Give them enough information on yourself that they can get a sense of who they are talking to. (Are you a founder? Manager? Technical? Business? What domain have you worked in during your career?) To build trust and rapport, discuss any other common interests or areas of overlap.
Establishing mutual purpose. (2-3 minutes) Remind them that the goal of the call is to measure fit between the candidate and the role, based on the experience the reference has had working with the candidate in the past. Ask if that sounds good, and pause and listen for anything they might want to tell you up front.
Asking questions. (20+ minutes) Ask questions that you’ve planned, but follow up on things the reference says that may lead to useful insights.
caution Throughout the discussion, be careful to listen exactly to what the reference says, and not confuse it with how they say it. You don’t know the reference, so it can be easy to be thrown off by communication style.
important Take notes. You’ll want them later.
First-time hiring managers usually know or are told that they need to check references and may eagerly solicit or set up the calls. But at some point the reality of the challenge becomes apparent: How exactly do you get helpful and reliable guidance about one person you don’t know, from someone you know even less? And even more challenging, how do you do it in a short window of time, like a 30-minute call? Surely any reference who is a friend of the candidate will just say positive things anyway—how do you know if you can trust this person’s judgment? Worse, you may have heard that legal considerations make most references largely perfunctory and a waste of time,* so why spend the time in the first place?
When preparing your questions for a reference call, it’s often best to sequence from basic, factual questions to more high-level and subjective questions. This way, you can let the reference ease into the discussion with answers to simple questions, that are closed—that is, have one possible answer. In other words, ask about job title and technology stack before you ask about communication skills.
Start by asking a little about the reference themself, how well they know the candidate, and how long they worked together and in what context.
What was the role of the candidate at the time the two worked together? Did one report to the other? Were they on the same team? How did the team fit into the larger organization?
Did the candidate’s role, responsibilities, or title change over the course of their working together?
Then the questions can be a little more advanced:
Ask anything you need to know about the product the candidate focused on, or the nature of the work, such as the tech stack (if you didn’t ask that at the outset) or the key challenges of the team at the time.
What was the candidate known for being good at on the team?
Communication skills and style, including any specific needs or aspects that would be good for a future manager to know.
What others on the team thought of the candidate.
Ask about fit:
Any kinds of work the reference thinks the candidate is a great fit for.
Any kinds of work the reference thinks the candidate is a less than ideal fit for.
Any other things they can add or suggest to make the candidate successful.
For all these questions, be specific, but open-ended. Listen carefully, take notes, and follow up on details.
danger Some key pitfalls include:
Phrasing questions in a way that encourages the reference to take a bluntly negative position, or makes them feel they’re being critical of the candidate. This will cause most references to shut down.
Asking for overly precise or numeric rankings of skills. This is a good way to get a subjective, inflated answer. Instead, focus on comparisons with the team the candidate worked with, or on clear but qualitative characteristics.
Asking vague questions where answers will give little signal.
Asking closed questions or feeding answers to the reference. Give them space to suggest or provide detail.
Being biased by your own opinion of the candidate, good or bad. Confirmation bias means hearing only what confirms what you already think. Listen attentively. Ask neutral questions that elicit real answers. Ask for clarification wherever you’re unsure what the reference meant.
Rushing the call. Be mindful of time, but without enough depth and context, you’ll get no useful information.
“What can you tell me about Anne’s work?”
This question is way too general and will result in equally general answers. Only specific questions that yield specific answers will be helpful in the hiring process.
“So what was it like working with Anne?”
Again, this is too general a question.
“How would you rate Anne’s ability in algorithms on a scale of 1 to 10?”
Very few references provided by a candidate will give a low number on a quantitative rating. In addition, no matter how precisely you phrase the question, one person’s 7 is another person’s 9. Without knowing the reference really well, the answer is almost always useless.
“Does Anne know Python and JavaScript?”
This is a low-signal question because “know” is far too vague—it can mean anything from an expert to someone who studied languages in college. The best way to learn the answer to this is to test for it.
“We notice she was really strong at Python and database management. Did you notice that too?”
This gives the reference little room except to agree or disagree, and feeds them your own bias toward the candidate’s skills.
“What would you say are Anne’s greatest weaknesses?”
This rarely elicits candid information; most people are not willing to answer that question about a colleague to a stranger.
“Can you tell me how long you were at Zenith and what your role was there?”
The answers can provide essential understanding of what the reference knows about the work environment they shared with the candidate.
“Can you explain Anne’s role at the time you worked together? How closely and for how long did you work together?”
Follow-up: “How big was the team? And how did the team fit into the overall organization?”
The answers signal how accurate the reference’s take on the candidate will be.
“What was one of the more valuable pieces of work Anne did while on your team? Can you give me an example?”
Follow-up: Ask for details and have them walk you through Anne’s notable accomplishment.
“Anne worked for three years on your team at Zenith Labs. How did her role evolve during that time?”
If the candidate grew in responsibility, this is a positive sign you’ll want to know more about.
Follow-up: If the answer is no, it’s worth noting and asking if the reference can say why. This could be related to the candidate, but could also have been due to organizational or structural factors.
“Can you talk a little about her communication skills? How did they compare to others on the team?”
It’s much easier to discuss how someone compares with others on the team than to assign numeric rankings or make judgements of “good” or “poor” skills. From there, you can start to dig deeper into specific skills and projects that person worked on.
“One skill that’s very important to us is database administration. Anne tells us she has experience in that. Can you tell me about her work in that area?”
Follow-up: “Who else did database administration on your team? How did their work compare?”
“You and Anne rebuilt the entire Widget Deconstruction mechanism over a period of about a year, correct? What parts of that did each of you do, and why?”
“You say her team was eight engineers. Usually there are specific people on a team who are highly respected or known as the go-to person for certain problems or questions. Were there any areas where Anne was that go-to person who was most experienced or most skilled?”
Follow-up: “Can you give examples?”
Follow-up: “How did her skill in Python compare to the others on the team?”
“Would you say Anne is most suited to a highly technical individual contributor role? Or do you think she would be ready for and enjoy filling a tech lead position on a small team?”
Particularly if you know the reference is close with the candidate, questions that focus on how the candidate can be helped to succeed and what makes them happy may encourage candidness
“We noticed during interviews that Anne is unusually quiet and at times hesitated to convey her ideas. We weren’t sure if this was her style day-to-day, or just interview nervousness. Can you say anything about how she communicates with a team? How about one-and-one?”
“What sort of team and working style would be ideal for Anne to be successful and excited about her work?”
Follow-up: “Are there any types of work or teams you think would be a less than ideal fit?”
“If you were hiring, are there roles for which you would hire Anne again? What kinds of roles?”
The kinds of roles the reference recommends can reveal a lot about Anne’s strengths and preferences.
In addition to talking with the references a candidate has listed, hiring teams may also solicit back-channel references (or back references)—that is, people the candidate has worked with but not listed as references. Hiring teams often source back-channel references without the candidate’s knowledge by checking the candidate’s LinkedIn connections or consulting with others who may know them.
Some consider back-channel references to be more reliable than candidate-sourced references, as a back-channel reference will likely have some association with the hiring team and so be more inclined toward greater fairness and honesty. Hiring teams use back-channel references far more commonly for senior hires. And in some industries and geographic areas (like Silicon Valley), the social graph among senior candidates, executives, investors, and former colleagues is dense enough that hiring teams almost certainly will use them.
The best approach for soliciting back-channel references is usually different from what is used for candidate-supplied references. In the case of the back-channel reference, usually the person on the team who best knows the reference will reach out, in any format that feels suitable. A short email might say something like:
Subject: Feedback on Anne Jackson?
Charlie, we’ve been making a few hires here at Acme Megatronomatics and we’re considering Anne as a candidate for Senior Forward Deployed Widget Architect. We’ve enjoyed getting to know her and are considering making her an offer.
Is there anything you can share about her suitability for this role? Would you have time for a call? I’m free today after 3 p.m. and tomorrow after 2 p.m. If that’s not possible, anything you can share by email is also helpful.
Thank you in advance!
Best, Barbara
cautionWhen soliciting back-channel references, it’s important to be mindful of a candidate’s privacy and trust. . This can entail informing candidates ahead of conducting back-channel references so that they are not surprised when they hear you’ve been asking about them and so you don’t put their current employment at risk. Advance notification might look like this: “In addition to the references you gave us, we may contact other people we know who may know you. Is there anyone that you wouldn’t want to be aware that we’re considering working together?” You can also mention the person you’re talking to by name (“I’m catching up with Charlie this week—do you mind if I ask her about your time working together?”), but in that case, you need to be mindful of the reference’s privacy, since she may not want you to tell the candidate you two talked.
When interpreting reference feedback, consider where the person is coming from. If the reference trusts you, they are more likely to give you candid feedback. If they are closer to the candidate, they may not want to risk any negative feedback being attributed to them, or they may feel the relationship colors their ability to provide objective feedback.
In general, references will be more positive than negative. Some recommend mentally compensating by discounting positive comments by 30% and amplifying negative comments by 30%. While this is an arbitrary number, it does highlight the importance of perspective. You can take outrageously positive or negative comments with a grain or two of salt.
Next, think about the context in which the candidate and reference knew each other. How closely did the reference and the candidate work together? How equipped is the reference to judge the candidate’s work? For instance, “I’ve managed over a dozen engineers and Anne is the strongest I’ve had on my team by far,” is a lot more compelling than, “Anne has been great to work with, but that was my first job out of school and I’ve only been here a few months.”
You also may consider whether the reference’s criteria for assessment is consistent with your own. For instance, traits that lead a candidate to success (or hold them back) at a large company may differ significantly from traits that lead to success at a much smaller company.
importantThe limited nature of reference checking makes it more subject to possible noise and bias than interviews. You may have spoken to the reference on a bad day. People grow and change—would you give yourself a glowing reference for every job you’ve held, and would those references all be reflective of the type of work you can do now?
Find out how veteran recruiter Jose Guardado learned the hard way about the power of a (slightly) negative reference comment in “How Reference Checks Can Go Wrong: Managing subjectivity in reference feedback,” up on the Holloway blog.
As with any part of the hiring process, references aren’t a perfect source of information. Ultimately, a negative reference for a candidate should definitely give you pause, but if you believe in the candidate based on other data points, it doesn’t need to completely disqualify them. You can always follow up with the candidate to see if you can figure out if anything has changed since the time they knew the reference.
Anyone who’s run interview processes for enough candidates will be able to recall instances in which a smart manager made a bad call. Whether it’s hiring or passing, there’s no way to know whether this decision is the right one for the long run, but somehow a decision has to be made. It’s worth recognizing that no matter how careful and structured your hiring process is, it can never involve 100% certainty, because it is a human process and we are neither mind-readers nor oracles. But the more thoughtful you are about how decisions are made and what factors to consider, the more confident you can be.
candidate Knowing how a company you’re interviewing with makes hiring decisions will help you have a sense of your timeline with that company, and make the (often stressful) process less opaque. For instance, Google tends to take longer to extend an offer, since offers require both hiring committee and senior leader approval. When interviewing with a company, you can ask your recruiter or interviewer how that company makes hiring decisions and why.
How a company makes hiring decisions is usually related to the company’s decision-making philosophy in general. Netflix has a culture that values trust, freedom, and “diffuse accountability,”* and thus allows teams (and managers) to make their own hiring decisions. Google, on the other hand, believes that well-designed committees can make better decisions than giving a manager unilateral power. This philosophy usually extends beyond hiring to promotions and even technical and product decisions. Because hiring philosophy and strategy may be aligned with how a company or team runs day-to-day, it’s often a good idea (and fair) to share with the candidate how this decision will be made—it helps set them up for success and gives them important insight into the company.