editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
First-time hiring managers usually know or are told that they need to check references and may eagerly solicit or set up the calls. But at some point the reality of the challenge becomes apparent: How exactly do you get helpful and reliable guidance about one person you don’t know, from someone you know even less? And even more challenging, how do you do it in a short window of time, like a 30-minute call? Surely any reference who is a friend of the candidate will just say positive things anyway—how do you know if you can trust this person’s judgment? Worse, you may have heard that legal considerations make most references largely perfunctory and a waste of time,* so why spend the time in the first place?
When preparing your questions for a reference call, it’s often best to sequence from basic, factual questions to more high-level and subjective questions. This way, you can let the reference ease into the discussion with answers to simple questions, that are closed—that is, have one possible answer. In other words, ask about job title and technology stack before you ask about communication skills.
Start by asking a little about the reference themself, how well they know the candidate, and how long they worked together and in what context.
What was the role of the candidate at the time the two worked together? Did one report to the other? Were they on the same team? How did the team fit into the larger organization?
Did the candidate’s role, responsibilities, or title change over the course of their working together?
Then the questions can be a little more advanced:
Ask anything you need to know about the product the candidate focused on, or the nature of the work, such as the tech stack (if you didn’t ask that at the outset) or the key challenges of the team at the time.
What was the candidate known for being good at on the team?
Communication skills and style, including any specific needs or aspects that would be good for a future manager to know.
What others on the team thought of the candidate.
Ask about fit:
Any kinds of work the reference thinks the candidate is a great fit for.
Any kinds of work the reference thinks the candidate is a less than ideal fit for.
Any other things they can add or suggest to make the candidate successful.
For all these questions, be specific, but open-ended. Listen carefully, take notes, and follow up on details.
danger Some key pitfalls include:
Phrasing questions in a way that encourages the reference to take a bluntly negative position, or makes them feel they’re being critical of the candidate. This will cause most references to shut down.
Asking for overly precise or numeric rankings of skills. This is a good way to get a subjective, inflated answer. Instead, focus on comparisons with the team the candidate worked with, or on clear but qualitative characteristics.
Asking vague questions where answers will give little signal.
Asking closed questions or feeding answers to the reference. Give them space to suggest or provide detail.
Being biased by your own opinion of the candidate, good or bad. Confirmation bias means hearing only what confirms what you already think. Listen attentively. Ask neutral questions that elicit real answers. Ask for clarification wherever you’re unsure what the reference meant.
Rushing the call. Be mindful of time, but without enough depth and context, you’ll get no useful information.
“What can you tell me about Anne’s work?”
This question is way too general and will result in equally general answers. Only specific questions that yield specific answers will be helpful in the hiring process.
“So what was it like working with Anne?”
Again, this is too general a question.
“How would you rate Anne’s ability in algorithms on a scale of 1 to 10?”
Very few references provided by a candidate will give a low number on a quantitative rating. In addition, no matter how precisely you phrase the question, one person’s 7 is another person’s 9. Without knowing the reference really well, the answer is almost always useless.
“Does Anne know Python and JavaScript?”
This is a low-signal question because “know” is far too vague—it can mean anything from an expert to someone who studied languages in college. The best way to learn the answer to this is to test for it.
“We notice she was really strong at Python and database management. Did you notice that too?”
This gives the reference little room except to agree or disagree, and feeds them your own bias toward the candidate’s skills.
“What would you say are Anne’s greatest weaknesses?”
This rarely elicits candid information; most people are not willing to answer that question about a colleague to a stranger.
“Can you tell me how long you were at Zenith and what your role was there?”
The answers can provide essential understanding of what the reference knows about the work environment they shared with the candidate.
“Can you explain Anne’s role at the time you worked together? How closely and for how long did you work together?”
Follow-up: “How big was the team? And how did the team fit into the overall organization?”
The answers signal how accurate the reference’s take on the candidate will be.
“What was one of the more valuable pieces of work Anne did while on your team? Can you give me an example?”
Follow-up: Ask for details and have them walk you through Anne’s notable accomplishment.
“Anne worked for three years on your team at Zenith Labs. How did her role evolve during that time?”
If the candidate grew in responsibility, this is a positive sign you’ll want to know more about.
Follow-up: If the answer is no, it’s worth noting and asking if the reference can say why. This could be related to the candidate, but could also have been due to organizational or structural factors.
“Can you talk a little about her communication skills? How did they compare to others on the team?”
It’s much easier to discuss how someone compares with others on the team than to assign numeric rankings or make judgements of “good” or “poor” skills. From there, you can start to dig deeper into specific skills and projects that person worked on.
“One skill that’s very important to us is database administration. Anne tells us she has experience in that. Can you tell me about her work in that area?”
Follow-up: “Who else did database administration on your team? How did their work compare?”
“You and Anne rebuilt the entire Widget Deconstruction mechanism over a period of about a year, correct? What parts of that did each of you do, and why?”
“You say her team was eight engineers. Usually there are specific people on a team who are highly respected or known as the go-to person for certain problems or questions. Were there any areas where Anne was that go-to person who was most experienced or most skilled?”
Follow-up: “Can you give examples?”
Follow-up: “How did her skill in Python compare to the others on the team?”
“Would you say Anne is most suited to a highly technical individual contributor role? Or do you think she would be ready for and enjoy filling a tech lead position on a small team?”
Particularly if you know the reference is close with the candidate, questions that focus on how the candidate can be helped to succeed and what makes them happy may encourage candidness
“We noticed during interviews that Anne is unusually quiet and at times hesitated to convey her ideas. We weren’t sure if this was her style day-to-day, or just interview nervousness. Can you say anything about how she communicates with a team? How about one-and-one?”
“What sort of team and working style would be ideal for Anne to be successful and excited about her work?”
Follow-up: “Are there any types of work or teams you think would be a less than ideal fit?”
“If you were hiring, are there roles for which you would hire Anne again? What kinds of roles?”
The kinds of roles the reference recommends can reveal a lot about Anne’s strengths and preferences.
In addition to talking with the references a candidate has listed, hiring teams may also solicit back-channel references (or back references)—that is, people the candidate has worked with but not listed as references. Hiring teams often source back-channel references without the candidate’s knowledge by checking the candidate’s LinkedIn connections or consulting with others who may know them.
Some consider back-channel references to be more reliable than candidate-sourced references, as a back-channel reference will likely have some association with the hiring team and so be more inclined toward greater fairness and honesty. Hiring teams use back-channel references far more commonly for senior hires. And in some industries and geographic areas (like Silicon Valley), the social graph among senior candidates, executives, investors, and former colleagues is dense enough that hiring teams almost certainly will use them.
The best approach for soliciting back-channel references is usually different from what is used for candidate-supplied references. In the case of the back-channel reference, usually the person on the team who best knows the reference will reach out, in any format that feels suitable. A short email might say something like: