editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
The onsite interview is a series of interviews held at the company’s office for several hours to a full day. Onsites can be a crucial part of the interview experience for candidates and companies alike because they offer an extended opportunity for assessment on both sides. Onsites typically include pair interviews, pair programming, live coding, debugging, non-coding technical interviews, and nontechnical interviews.
controversy Some companies have a multi-day process where the candidate comes back onsite several times. This can slow down your hiring process and lead to a bad candidate experience; but for early-stage startups or key roles, the opportunity to spend more time with the candidate may be worth the risk.
The pros of onsites are numerous—they provide the highest signal you can get. The human factor is important here: candidates get a real sense of the day-to-day output and environment of a company, and if you train your interviewers well, the onsite allows for a more conducive environment for comfortable communication. Onsites are very important for selling the job to candidates, who will get to experience the team firsthand, usually for the first time. Most candidates would prefer to interview in person, though in some cases it is essential to offer a remote option.
Some interviews provide much higher signal in person than they would over the phone. Technical non-coding formats like architectural interviews are nearly impossible to conduct when you don’t have a shared whiteboard to work from. Behavioral interviews are easier to assess when you can also read a person’s body language and facial expressions.
danger A risk of onsites is that they can be a source of bias; they require more evaluative efforts from interviewers than things like coding challenges do, and interviewers tend to feel more comfortable rendering verdicts based on false signal. For example, handwriting or communication style can color how an interviewer interprets code on a whiteboard. But few people are actually qualified to make such judgements. Interviewers might think, “This person was nervous at the whiteboard,” and infer that the candidate is inexperienced or has a reason to be nervous. Such judgments are usually wrong and tend to be based on unconscious biases. Interviewers tend to uprank nervous-seeming candidates who look like them or have a similar background, while penalizing or downgrading candidates for nervousness when they are underrepresented or have a different background. It is crucial to train interviewers and use structured interviews, to keep candidate performance from being a function of who is conducting the interview or what mood they are in at the time.
caution Many believe coding-intense interviews provide the noisiest signal in onsite interviews. Ideally, an interview loop includes multiple sessions and multiple technical formats to avoid a situation where one bad interview tanks a candidate. In general, coding questions can also be the most tiring for candidates because they are being asked to solve new problems rather than talk about previous work. Discussing previous work gives you some of the best signal on a candidate’s coding ability and their problem-solving skills. On the other hand, if you have too many questions or formats, you will wear candidates down. It’s important to measure out the day, with easier, more straightforward interviews coming earlier. Coverage of other technical skills (such as deep diving into a prior project) and nontechnical interviews can come later in the day.
story “To get in more questions (better signal) without tiring the candidate, design the questions so that they flow together. For example, you might ask them to first build a simple API. Then add rate-limiting. Then maybe solve an algorithm problem that comes from efficiently rate-limiting across multiple machines. Then maybe add a frontend client. The thematic connection of the problems allows the candidate to stay focused on one codebase. This is not super common across the industry, but it’s something that we’ve seen work really well for us.” —Ammon Bartram, co-founder and CEO, Triplebyte
Lara Hogan provides a template you can use to determine the list of signals you need to gather to make a hiring decision, and assign those signals to people in different interview slots. This is a great place to start planning out who will cover what for your onsite interviews. Marco Rogers adds even more detail to his post that includes an onsite interview schedule template, notably including a second interviewer in each session and capping the total number of sessions at four.
Finally, having some kind of wrap-up at the end of the day with a hiring manager is an important part of giving the candidate some closure on the day and a chance to ask questions and hopefully meet their future manager. Additionally, given how important the manager-team member relationship is, this is an important part of building toward a successful close. You can also learn a lot about a candidate’s perspective through the questions they ask, and while you will be giving them time during the other interviews to ask questions, they may have new questions and insights by the end of the day and may be feeling more comfortable.
Technical one-on-one interviews often follow a simple form. This general overview illustrates how most in-person interviews are conducted:
Pre-interview prep. Most interviewers will require at least a few minutes before the interview to get into the right mindset, make sure they understand what questions to ask, and what the expectations for the candidate are. This need not take more than 15 minutes, but new interviewers may experience nervousness leading up to the interview throughout the day.
Greeting. Welcome the candidate and set them at ease—they may be experiencing high stress, information overload, or energy drain. A positive greeting gives them a chance to settle into the conversation and get used to the interviewer. Energy and enthusiasm are infectious; interviewers who bring genuine excitement to the interview can help a tired candidate show what they are capable of. Ask if they need to use a bathroom or would like something to drink. It’s the human thing to do. Making them comfortable is not only the right thing to do, it will also make it easier for you to assess their actual capabilities.
Interviewer introduction. It’s both helpful and just friendly for interviewers to let the candidates know who they are before diving into the details. It’s awkward to talk to someone you know nothing about, and intros build some initial rapport while providing a chance to highlight one or two things the interviewer likes about the company.
dangerSome interviewers might try to make conversation without realizing that it is against the law to ask certain questions in interview settings. Common illegal interview questions include asking about marital status or age.
Introductory questions, usually short and easy. Most interviews start with an easier warmup question or questions, which could be a simpler version of a difficult question that will be asked later. The warmup allows the interviewer to adjust their approach based on whether things are going better or worse than expected. Some candidates won’t realize the initial question is not the hard part of the interview and will over-solve or over-think it. Good interviewers help the candidate understand the flow of the interview so they can manage the time and nudge the candidate away from being unnecessarily thorough or careful in this phase.
You might start with a version of the question where a simple solution would work. For example, you may pose a coding question where the first version of the problem can be solved with storing everything in memory.
Follow-up questions, usually meatier. A gradual increase in difficulty ensures that candidates can be assessed on a spectrum of how far they were able to get, not whether they had a single “aha” moment or got stuck on something far too hard. The meatier follow-up question or questions should consume half or more of the overall interview time.
Using the example above, once the candidate understands the problem and has something working, you can introduce new requirements that up the challenge, like increasing the amount of data to a degree where it will no longer fit in memory.
Questions from the candidate. The final question-and-answer section allows candidates to learn more about the company and provides insight into what the candidate is concerned or excited about. It’s important for interviewers to write these down for use in the closing process.
Tell them what will happen next. This is the time to clarify the next steps in the process, including the approximate timeline. Helping the candidate estimate when they will know if they’re moving forward can help reduce stress, leading to a better candidate experience. For instance, for phone screens or after the last session in an onsite, it’s important to let the candidate know that there will be a later follow-up and when they can expect to hear back. In between onsite interviews, the candidate should know when and where they can meet the next interviewer—whether they should wait in their current interview room or whether they will be escorted to another room.
dangerIt’s important for the interviewer to refrain from sharing their assessment with the candidate during or immediately after the interview. Even when the interviewer feels certain of the outcome, the interview is just one data point in the process, and their opinion might change when they see the full set of feedback from colleagues or take some time to reflect. It’s helpful to remember that the period during and immediately after an interview can be stressful for candidates, and that stress and adrenaline may cause them to be less likely to properly interpret (or handle) any feedback. Thus, an appropriately neutral but generally positive closing is to thank the candidate for their time and indicate that you enjoyed the discussion.
importantIt is, however, equally important to deliver a response in a timely manner. If you take too long to give a candidate any feedback at all, they may read the silence as impolite disinterest on the part of the company or convince themselves they don’t want the job anymore.
In the late stages of the process, candidates often ask about the actual solution to a coding problem they were presented with or whether there was a better approach than the one they took. Time permitting, answering neutral technical questions like this is fine and can lead to a more engaged technical discussion that gives the interviewer more signal—a candidate that is truly interested in better ways to solve a problem is usually a more thoughtful engineer.
In pair interviewing, two interviewers simultaneously work with and assess a single candidate. Typically, one interviewer leads the interview while the other observes. Microsoft’s Developer Division and Twitter, among others, use pair interviewing exclusively.
confusion Some companies or teams use a form of pair interviewing to onboard new interviewers; this is called shadowing.
Add another colleague into the mix as a second interviewer. I’ve found that it not only brings a higher fidelity and signal to the conversation, but also breaks down the flow and nature of the conversation. Sometimes a person will be talking to the candidate, and other times, they’ll address their colleague. I find questions and answers bouncing between three points opens up the discussion and evaluation much more.Marco Rogers, veteran engineering manager*
Pair interviewing can yield more complete signal from an interview—if one interviewer notices something that the other interviewer doesn’t, they can collaborate to assess what the outcome should be. They may help reduce bias, by bringing two perspectives to the same situation. This format can also help to root out bias on the candidate side.
story “When pair interviewing with a male colleague, I’ve sometimes seen candidates only speak to or make eye contact with him. This is behavior that likely wouldn’t have been seen were I interviewing solo—it would be super unlikely for a candidate to ignore the one person talking to them. Pair interviews can help to reveal if a candidate treats one member of the pair noticeably differently, which can be useful signal for how they might treat different teammates.” —Ryn Daniels, Senior Software Engineer, HashiCorp
Pair interviewing also creates quality related feedback loops, since interviewers can share their thoughts with one another on how the interview was conducted. That said, pair interviewing effectively doubles the amount of interviewer time spent per candidate, and it is wise to evaluate whether it is worth that extra cost.
Pair interviewing does add extra interviewer load, but interviewers seem to like it since it gives them increased confidence in their assessment and a mechanism for learning and feedback—a bit like paired programming.Jan Chong, Senior Director of Engineering, Twitter*
The likelihood of success in pair interviews may depend on the specific pairing. For example, having a pair made up of a manager and a more junior person can be problematic. Risk is lower if the junior person is shadowing the manager and therefore is not expected to question the candidate directly. But if they’re meant to be partners in the interview, the junior person may feel intimidated by the manager and be unwilling to risk asking the wrong questions of the candidate. Structured interviewing and careful planning can mitigate this risk; each person in the pair can be assigned a tack to take, or an area to focus on, or specific questions.
A pair of developers who are of equal rank and similar disposition may work best; variant personality style causes greater risk. When a candidate is focusing hard on answering questions, having to switch between, say, a low-energy interviewer and a high-energy interviewer can be extremely difficult and tiring. This does not set the candidate up for success.
important It might seem intimidating to a candidate when more than one interviewer walks into the room—you can mitigate this by letting the candidate know in advance that you will be conducting a pair interview.
A panel interview is one in which three or more interviewers simultaneously evaluate a candidate.* Typically, companies select interviewers to represent a variety of perspectives regarding the open position.
Panel interviews have downsides for the candidate experience. Having to address multiple interviewers at once can feel like an interrogation; introverts in particular usually do better in one-on-one settings. One upside is that panel interviews give the candidate a chance to meet more of the team. But for most people, panel interviews are intimidating.
That said, there are a few possible benefits of this format for the company.
One company benefit is that in a very consensus-based hiring process, a panel interview makes it possible for more people to be involved. It may also be useful when multiple hiring managers are looking for different types of candidates or are trying to route a candidate to the right role or team. However, with everyone asking different questions, the likelihood that the interview will get deep enough for a strong signal is low.
Panel interviews can be beneficial for reducing bias, when answers can be gauged by multiple people of varying backgrounds. However, it’s important not to misrepresent the diversity of your company or overburden underrepresented employees.
caution Although it may seem that panel interviews can give signal on how a candidate will work in a group, panels may not reflect the team the candidate would work with or may lack the dynamic of a collaborative team, particularly if the group feels artificially put together, or no one has specific questions to ask that they couldn’t get signal on in another format. Inadequately trained panelists may give off a vibe that they do not want to be there, hurting the candidate experience and downgrading their perception of the company. It’s hard enough for candidates to adapt between different types of people at the same time, especially in a situation that is already stressful. So if you decide to use this format, it helps to make sure each panelist has a clearly defined reason for being there.
In the end, most people just hate presenting in front of groups. Panel interviews often do not meet the candidate experience principle.
A similar, and often preferable, alternative is to have the candidate give a presentation to the team. One hiring manager* told us that a team he worked for tried presentations and ended up scrapping it because the signal they got was of so little value. One hire they made based on the person’s presentation ability ended up not being a strong fit for the role; they had gotten signal on the wrong kind of communication ability, one that wasn’t relevant to collaborative work. While engineers do sometimes present ideas to groups, like in design reviews, it’s a relatively infrequent activity and usually less formal than a typical presentation.
On the other hand, presentations are fairly common for research or scientist positions, because the work requires conference presentations. Similarly, if the role you’re hiring for requires similar work as a core component—product managers or a sales position—a presentation evaluation makes more sense.
A whiteboard interview is a technical problem-solving assessment that takes place in real time and typically involves a candidate writing code and sometimes diagrams on a whiteboard while onsite.* A similar kind of interview can be done in person with pen and paper.
controversy Whiteboard coding questions are controversial in the industry and their merits are hotly debated.* Whiteboard coding questions do not accurately mimic the environment that a candidate will usually work in—they have no access to a compiler, editor, or reference material as they normally would. They also bias toward candidates with fresh, detailed memory of a language or library and against developers who prefer a more iterative model of writing and testing code incrementally. Reacting to these potential flaws, Hiring Without Whiteboards provides a popular, extensive, collaboratively sourced list of companies that don’t use whiteboard coding.
important Looking at whiteboarding more broadly, however, can dissolve what appear to be inherent constraints of the format. As more than one senior engineer put it to us: “No one is good at coding on a whiteboard.” Accepting that can be the start of something interesting. If hands-on coding tells you what an engineer knows and how they use their knowledge, whiteboarding interviews may better cover the why—why does the engineer make each choice? When conducted well, whiteboarding gives signal on an engineer’s communication and collaborative abilities, as well as how they think through problems and theorize solutions.