Hands-on Coding Interviews

You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.

Hands-on coding interviews include all formats where a candidate uses a computer and not a whiteboard. They are often part of an onsite, but can also be conducted remotely.

In a hands-on coding interview, an interviewer will give the candidate a technical problem; the candidate will typically use a code editor to solve it. This provides signal on whether a person has done this kind of work before, the nature of their coding style and problem-solving style, and what practical set of skills they have. Because most programmers are more comfortable with writing code than completing a task on a whiteboard, hands-on interviews move along more quickly, allowing you to go deeper and get more volume of signal.

Pros and Cons of Hands-on Coding Interviews

Are hands-on coding interviews better than whiteboard interviews? It’s complicated!

Some advantages of hands-on coding interviews include:

• Technical candidates are more familiar with the coding experience, which may be particularly relevant for candidates who rely heavily on the features of their IDE.

• Hands-on coding interviews are based on a more natural style of writing and debugging code incrementally, which allows the candidate to display confidence in their work.

• The candidate can use reference materials.

• The candidate can provide an exact transcript of the code at the end of the interview.

• Because hands-on interviews are live, they offer little room for cheating.

The primary downsides of this approach are:

• Not all candidates have laptops, so the company may need to provide them.

• Laptops may have hardware or software issues that delay the interview; whiteboards rarely do.

• Unless the company provides scaffolding, the candidate may need to write a bunch of boilerplate code to get started. If the company does provide scaffolding, it likely needs to do so for a large number of languages, and the logistics of getting the scaffolding to the candidate can present a hurdle. (One possible alternative is to have the candidate modify or debug an existing codebase, allowing the collection of additional signal on other skills relevant to software engineering like reading and working with new code.)

• The candidate may get caught up in such non-essential elements of the problem as making sure their code compiles or looking up the exact right library function, rather than focusing on solving the more fundamental aspects of the problem. While these are also skills required of an engineer, they’re more basic than the translation of ideas to code, and detract from evaluating that.

• Most programmers do not perform their work in front of other people; the pressure of having someone watch them code can be enough to throw candidates off their game.

• It can be somewhat awkward for the interviewer to watch the candidate work without staring over their shoulder. While this can be mitigated by using an app like Coderpad, the editing experience when using such an app will not be what the candidate is used to.

Those downsides notwithstanding, for the right kind of question, having people code on a laptop can offer a better candidate experience. Hands-on, synchronous work more closely mirrors a real working environment, with back-and-forth between the parties, and with no expectation that a candidate do something perfectly without feedback.

While online challenges screen for basic coding skills, synchronous, hands-on formats are closer to assessments, because they allow you to measure how a candidate reasons their way through problems. This results in better signal on a candidate’s talent, not just their skill or knowledge base. Interviewers can course correct, and can increase the difficulty of the question or questions as the interview proceeds, depending on how the candidate is performing. Signal is also more accurate because candidates can’t ask friends or look up answers.

Compared with online coding challenges, hands-on coding interviews are more expensive; they require an interviewer to be present in person, over the phone, or in a shared screen environment, as well as the cost of transporting the candidate to an onsite. They also require far more training and preparation from interviewers.

Compared with the whiteboard environment, where everyone is expected to fumble (or to hesitate or be more playful in how they use the whiteboard), candidates are expected to be more precise and efficient when working in a code editor. Some fumbling might be expected for people earlier in their career, who may perhaps be not as familiar with different coding environments or who haven’t spent thousands of hours coding—especially when working with an interviewer looking over their shoulder. That doesn’t mean a less senior person isn’t right for the role you’re hiring for. If the person can be trained on the job, how much they fumble in hands-on work doesn’t matter as much; the interviewer’s focus then becomes evaluating talent over skill.

Some companies opt to constrain candidates on language or environment, putting candidates in a place they may not be familiar with. Language requirements are normal, but environmental requirements are rarely standard unless you use a specific coding environment like iOS or Android. Barring role requirements, restricting language or environment is an artificial constraint that will give you poor signal on their ability and may cause you to lose a third to half of your candidates.

On the other hand, unconstrained choice on language and environment means interviewers must prepare more thoroughly and may receive more variable signal. Has a candidate made a smart language choice, like using a typed vs. untyped language? If they use a typed language, errors will be apparent; if they use an untyped language, the interviewer needs to know how to find the errors the candidate might make.

To remedy this, it’s best to train interviewers to ask the candidate doing the coding to talk them through their work, rather than just watching the candidate deliver the code. “If anything, more talk, less code,” as Scott Woody put it to us. This can be harder than it seems, especially when the interviewers are junior engineers who want to appear accomplished and impressive to candidates. It’s very reasonable for an interviewer to say, “I haven’t seen a solution like this before. Can you talk me through it?”

​story​ “I tend to recommend that interviewers try to take an attitude of ‘How would I interact with someone I was pairing with or mentoring?’ This cuts down on seemingly adversarial attitudes that can make candidates nervous. It’s hard to be at your best when it feels like someone is wanting or waiting for you to fail.” —Ryn Daniels, Senior Software Engineer, HashiCorp

​story​ “It’s gauche to judge someone purely on language expertise. You can learn a language if you’re a good engineer. That said, if they need to be an expert in Go, go ahead, force them to use Go in all their interviews. That’s appropriate. What I find more often is an implicit expectation that you’re an expert in whatever language you use. In reality, we’re all searching the internet for help with our work. That’s just more realistic.” —Scott Woody, former Director of Engineering, Dropbox

In short, in the best case, hands-on coding can give you fine-grained, practical signal. What do they know, and how do they use that knowledge? As with other formats, building clear rubrics and communicating the signal you’re trying to get from a hands-on interview can help avoid judgments based on work style or approach.

​candidate​ We suggest Yangshun Tay’s Tech Interview Handbook Cheatsheet, especially when preparing for hands-on interviews—it includes how candidates can explain their work in a hands-on. There are also helpful reminders here for preparing for other types of interviews, like phone screens.

Some Hands-on Interview Formats

A live coding challenge is an exercise or work sample that provides the company with content for assessment that mimics the work a candidate might do on the job. Within live coding challenges, there are two variations: a company may ask a candidate to work on a project on their own while onsite, or the candidate may be assessed in a pair programming interview, in which the candidate interacts directly with the existing team on a project.

Pair programming and live coding make the most sense when the role is for a coding-only position. With senior positions, which require a lot more than coding ability, these formats might be a waste of time—there are other ways to evaluate how someone thinks through code and approaches problems, and how they lead others.

The challenge is to design something close to what the candidate would be doing on the job; giving a puzzle on recursive algorithms when the role will not require that kind of work will not give you relevant signal. It also has to be possible to complete the challenge in a reasonable amount of time.

With pair programming and live coding, candidates can’t cheat. The skills demonstrated are practical. The candidate may have told you that they know what a singleton pattern is, but can they write one or recognize one in code? Live work helps to track how the candidate applies their knowledge, which a candidate can help with by walking the interviewer through what they’re doing in detail. This may be especially important for recent graduates.

Live coding presents the same kinds of challenges as other hands-on formats—most engineers do not spend their time coding in front of others. The pressure and awkwardness of doing so can be challenging enough to throw off a candidate and give you false signal on their abilities. You’ll see how quickly and efficiently the candidate produces work, but only to a point; the pressure of the live environment can hobble some engineers. Live work is a skill in and of itself. Dealing with that anxiety or not being affected by it measures something different from coding ability.

​story​ “An interview is as contrived as a blind date. When you meet someone by accident, glorious music plays and you get to see the real them. A lifetime movie is made. When you go on a blind date or a Bumble mixer, it’s so predatory and awkward. Same thing with live coding. If you start failing, you’re going to spiral out of control because of the stress. In real life, someone corrects you, a bug is filed, you go home, you come back the next day and you fix it. With the coding experience live, you make a mistake, EVERYONE knows it, and you can’t reset all that well unless you’re very controlled.” —Aaron Saray, engineer

​story​ “I’ve seen companies with open-source efforts tackle a ticket with the applicant. It’s interesting because it requires the interviewer to find a ticket before each interview that is appropriate, and if the project isn’t sufficiently large, that’s hard to do for multiple candidates; the ticket will always be different. But it gives two options: either they can pair on the open ticket or, if it’s a senior candidate, they can review an existing PR and make adjustments.” —Laurie Barth, Staff Engineer, Gatsby

Debugging interviews are exercises in which candidates are asked to find solutions to real-world bug problems in codebases. Debugging is a critical task for most engineering roles, and an interview of this type can give insight into how the candidate reasons and works with their tools. These interviews also test for how well a candidate can read others’ code and make changes that respect existing designs.

Debugging may give a narrower slice of insight into the candidate than a take-home, which usually entails a mix of debugging and building.

The challenges of debugging interviews are mostly logistical. Debugging interviews require a lot of time on the part of the interviewers to develop the right question and build up scaffolding code in a language and environment with which the candidate is relatively familiar.

Note that some debugging interviews may be “great in theory, bad in practice.” As many as a third or even half of your candidates just won’t be able to solve the challenge. The more standardized language and development of iOS or Android environments make them a possible exception.