editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
caution Interviewing is an active process that requires the interviewer to be completely engaged. This requires that interviewers not be on their phone or check email or Slack. If you take notes on a laptop, turning off notifications will prevent interruptions. Candidates can tell when the interviewer is distracted, and it makes their performance worse. Likewise, the interviewer’s active engagement at all times will ensure they are collecting the clearest possible signal.
Being engaged with the candidate also ensures the interviewer feels like a human being to the candidate. An interviewer who sits silent and stone faced will intimidate the candidate, who will then underperform.
At the same time, effective interviewers will guard against giving the candidate unintended hints. This is a matter of some judgment, but it is likely that if a candidate is constantly pausing and looking at the interviewer, they’re fishing. If it’s clear they’re focused on the problem at hand, small positive signs at points of breakthrough can reinforce progress.
It’s essential for an interviewer to take some form of notes during an interview. These notes can be turned into a formal write-up, ideally as soon as possible after the interview itself. New interviewers should typically budget about the same length of time for their write-up as they spent on the interview, although with a great deal of practice and good in-interview note-taking, the time commitment will go down by half or more.
Good notes capture the questions that were asked and give a high-level description of what happened during the interview, including both candidate answers and any key moments in the discussion.
When interviews include coding questions, a complete report will capture the candidate’s written code, to allow the hiring team to objectively evaluate the results and consistently calibrate the process. A complete and final write-up will include a high-level assessment of the code quality.
Some interviewers are able to capture more detailed quotes while the interview progresses. This makes reconstructing the interview and creating a good write-up easier and can be particularly useful for folks whose schedules preclude a full write-up immediately following the discussion. However, note that good interviewers will only take down this level of detail if they can simultaneously stay engaged with the candidate.
The first step to collecting signal is making sure the interview is moving along on schedule.
Timing in interviews is critical. If the candidate isn’t moving quickly enough through the full range of questions, the interviewer may find it necessary to intervene. This may include asking the candidate to skip over less essential parts of the discussion so as to stay on time.
Interviewer boredom can be a clue that the interview is going off track in one of these ways:
The candidate is stuck and needs a hint. One useful tactic is to set them at ease by explaining, “Even if you don’t know the solution right away, I’d like to hear how you are thinking about the problem, so feel free to share your possible approaches as you work through it.”
The candidate is talking too much about something they have already made sufficiently clear. In this case, the interviewer may tell the candidate that since they definitely understand the topic, they can progress to the next question.
The interview as a whole is off course and needs to be pivoted.
Near the end of the interview, you may have to choose between letting the candidate finish working through what they’re on and leaving time for selling and any candidate follow-on questions. If the candidate’s evaluation hinges on what they do in these last few minutes, it’s best to give them that time; but you may still need to help them reach a good stopping point so that their mind doesn’t stay in the question during subsequent interviews.
I’m not interviewing for the right answer to the questions I ask. Instead, I want to see how the candidate thinks on their feet and whether they can engage in collaborative problem solving with me. So I always frame interview questions as if we were solving a real-life problem, even if the rules are a little far-fetched.Eric Ries, author of The Lean Startup*
The best interview questions are short and clear. Short and clear questions ensure candidates answer the question you meant, not just those they’ve previously thought about and are primed to respond to.
Good interviewers use a balance of open and closed questions. Open (or “open-ended”) questions, such as “How would you approach that challenge?”, allow the candidate to determine the direction of the discussion and show how they handle ambiguity. Open questions do not have a yes, no, or otherwise simple answer. Closed questions have specific answers, such as “How many months did you spend on this project?” These questions are good for gathering basic facts or seeking clarification.
Interviewers may need to probe into ambiguous or obfuscated answers, which are often signs that the candidate either doesn’t know the answer or is trying to hide something (perhaps avoiding an admission of fault, in a behavioral interview). Ambiguity requires specific or direct follow-up questions.
Sometimes a candidate may say something that just doesn’t make sense given the context shared in the interview. For example, they may describe previous work experience in a way that sounds problematic to the interviewer, but the candidate doesn’t acknowledge the problem. This sometimes happens because the candidate may not recognize that they’ve left out key context or details. This situation requires further questions, even leading ones, to clarify whether the candidate behaved correctly but just failed to communicate clearly during the interview. Any time a possible exculpatory explanation exists, a good interviewer will pursue it to avoid later questions about whether their assessment was correct.
Eric Ries stresses, “No matter what question you’re asking, make sure it has sufficient depth that you can ask a lot of follow-ups, but that it has a first iteration that’s very simple.” This gives you the opportunity to probe further into what the candidate may or may not know. Ries distinguishes between degrees of “not-knowingness” that give you additional signal:
The candidate doesn’t know, but can figure it out. Perhaps it’s been some time and they can’t recall all the details, but they have a strong intuition that leads them in the right direction.
They don’t know, but can deduce it given the key principles. If you fill the candidate in on the basic rules, can they reason from there? Would that change the way they approach the problem at hand?
They don’t understand the question. Ries notes that “Most questions require a surprising amount of context to answer. It doesn’t do you any good to beat someone up by forcing them through terrain that’s too far afield from their actual area of expertise.” If it is their area of expertise, you know you’ve hit a point to consider in your evaluation. If it’s not, it’s best to move on—it’s not fair or efficient to ask them to keep struggling through at that point.
A key part of extracting signal is making sure the candidate is able to put their best foot forward. An interviewer’s job entails setting up the candidate to shine—but it’s up to the candidate to actually shine. While the interviewer/interviewee relationship has an inherent adversarial component, if it’s clear that the interviewer is trying to set the candidate up for success, it can de-stress the process. Strategies might include outlining the flow of the interview before it begins, making it clear what the interviewer is looking for, and clarifying how the candidate can best provide answers. Giving appropriate guidance and hints can help get the candidate unstuck.
Much like the process of scaling the question, good hints are metered out carefully, starting with more general hints and gradually getting more specific. An interviewer who solves the problem for the candidate definitely doesn’t intend to hire them, so hints that answer for the candidate aren’t appropriate unless it’s clear they are not a fit. It’s also important to time hints properly—not too early, before the candidate has time to think; and not too late, when the interview runs overtime. This timing will vary by question (some questions take longer to answer than others), but if a candidate is clearly stuck for more than a few minutes, it’s usually time to get them unstuck. Also note that letting a candidate flounder for too long restricts what information you can gather as an interviewer. A candidate stuck for 45 minutes on a single aspect of a question might have performed brilliantly on the rest of the question, but for you to discover that, you’d have to give an appropriate hint so that they can move along.
Hints can take multiple forms:
Nudging a candidate toward or away from a particular approach that they mentioned. This is especially important early in the question, so that the candidate doesn’t work on a mis-scoped problem. In this scenario, you might say something like, “Don’t worry about how this would work when stored on disk; you can focus on solving this in memory.”
Giving the candidate general hints toward a solution. You might say, for example, “You mentioned binary trees; you might want to explore that direction further.” When they’re running out of time, these kinds of hints help a candidate get unstuck from a bad path that you believe they would eventually discard anyway.
Giving the candidate general feedback on the solution. This might sound like the following: “I think you may have a bug” or “It seems like there are cases this solution doesn’t handle.” These hints give the candidate a chance to discover the gaps independently and also allow the interviewer to assess the candidate’s ability to reason about their solution.
Giving the candidate specific feedback or direction. You might say, “What happens if you pass the argument 8 to this function?” These kind of hints are certainly strong, and might indicate some concern that the candidate isn’t able to track down the problem in question, but they give the interviewer a chance to validate that if a bug does turn up in a work setting, the candidate will be able to find it. These hints usually won’t invalidate a coding interview because this mimics the way most people run their code to find issues in real-life situations.
Telling the candidate exactly why something is wrong and asking them to address it. “If your solution were to handle multiple connections simultaneously, it would have race conditions. How could you fix this?” This kind of hint can either be part of making a question harder, like if the initial problem statement didn’t include these constraints, or can be a sign that the candidate missed something critical to the problem. This kind of hint is a last attempt to see how the candidate handles the problem. Is their response, “Oh, of course, that was silly of me,” followed by an immediate explanation of how to fix it, or do they seem confused by why that is an issue in the first place? If they follow up with the former, you’re back on track.
For the sake of candidate experience, if at all possible, you’d like the candidate to get to a satisfying stopping point by the end of the interview. Being half-finished with a problem will carry over and negatively impact the rest of their day. Given that coding questions are particularly noisy, it’s most helpful to make the signal you get from each one be as independent as possible. If the interview has gone south, and it’s clear the candidate cannot recover, the next goal will be to lead them to that satisfying conclusion using whatever hints are needed, or even by explaining the problem directly.
Not every company collects feedback on the interviewers and process from candidates who have gone through an interview loop. But doing so can be very helpful in determining pitfalls in your process, especially when it comes to the candidate experience. The key to collecting candidate feedback is to do so only if you have the intention of using it to improve your process. (See Diversity and Inclusion in Tech for more.)
You might send something like a “candidate experience survey” to each candidate directly after their interview process, regardless of whether they received or accepted an offer. A second alternative is to send surveys out in batches once a quarter or at some other interval.
The survey might ask a combination of qualitative and quantitative questions. Quantitative questions ask candidates to rank the experience or a subset of the experience on a scale of 1–10 or to rank how likely they would be to interview at the company again, on a 1–5 scale. Qualitative questions might include:
“How was your experience?”
“Would you recommend this experience to someone else?”
“Was there anything your recruiter or hiring manager could have done better?”
“Anything else, positive or negative, you’d like to share?”
If your company has a recruiting function, you may wish to ask some targeted recruiting questions.
caution It’s critical to keep candidate feedback anonymous.
When interviewers know what to expect from interview questions—what makes an answer “poor,” “fair,” “good,” or “excellent”—they are less likely to let noise and bias slip into their evaluations. Rubrics are systems that make it easier for interviewers to provide and discuss feedback on candidates, because everyone will be working from the same set of expectations.
A rubric is a set of guidance, usually written, for evaluating candidate’s answers to interview questions. Included in this guidance may be examples of answers at different quality levels, or prompts to help the interviewer perform their assessment. Rubrics may also provide interviewers with a series of questions to use, typically of increasing difficulty.
In a structured interview, rubrics are essential to keep interviewers on track and ensure an effective and fair process. Even with a rubric, it can be tricky to specify exactly what a good or bad performance looks like—there’s always variation in each individual and room for human judgment or interpretation. You might think of rubrics as a starting point to help foster fair and productive interviews and evaluations.