editione1.0.8
Updated August 24, 2022You’re reading an excerpt of The Holloway Guide to Technical Recruiting and Hiring, a book by Osman (Ozzie) Osman and over 45 other contributors. It is the most authoritative resource on growing software engineering teams effectively, written by and for hiring managers, recruiters, interviewers, and candidates. Purchase the book to support the author and the ad-free Holloway reading experience. You get instant digital access, over 800 links and references, commentary and future updates, and a high-quality PDF download.
Aline Lerner provided major contributions to this section.
Most hiring teams make decisions about whom to advance through the top of the funnel based on candidate resumes. Especially for inbound candidates, the resume may be the primary or only source of signal. Companies with strong brands and lots of inbound applicants rely on resumes to filter large numbers of candidates very quickly. Candidates selected via outbound sourcing or referrals usually get a chance to stand out in other ways.
Companies often screen LinkedIn profiles alongside or in place of resumes. Both allow filtering based on a candidate’s pedigree—where they’ve worked and where they’ve gone to school. Companies may also lean on different sources of early signal, like GitHub repositories, personal websites, and online challenges, any one of which can supplement, or in rarer cases, replace a traditional resume.
While hiring teams often view resumes as the primary means of making top-of-funnel decisions, filtering candidates by their resumes is noisy, biased, and inefficient. Although little academic research exists on the predictive use of resumes in software engineering hiring specifically, research has been done on the function of resumes across all industries. An 85-year meta-analysis found that resumes (along with other traditional assessment methods like interviews) are very poor predictors of subsequent job performance.
Engineering managers tend to harbor the suspicion that engineers are better at reading technical resumes than recruiters: “We’re domain experts! We don’t need to rely on proxies and can actually tell what the candidate can do!” But even the most experienced, qualified engineers aren’t always on the same page when it comes to what makes a good technical hire. In fact, the more experience you gain, the higher the risk that you may be stuck in your own way of seeing things. This makes the static data of a resume a particularly poor alignment tool.
Interviewing.io recently conducted a study* in which researchers removed all personally identifying information (name, contact information, dates, et cetera) from a set of resumes, and showed them to hundreds of recruiters and engineers. For each resume, researchers asked participating recruiters and engineers just one question: “Would you interview this candidate?”
On average, participants correctly guessed which candidates were strong* 53% of the time, and there was no statistically significant difference between engineers and recruiters. Moreover, and even more importantly, people’s errors weren’t consistent. In other words, everyone disagreed about what a good candidate looked like in the first place.*
Source: interviewing.io
Academic and industry research has consistently proven that bias is a common problem in resume filtering as well. In a 2003 study on resumes by the National Bureau of Economic Research, experimenters submitted identical resumes to a series of help-wanted ads in the Boston and Chicago areas where the only difference between resumes was candidate name; some names were thought of as traditionally white and others as traditionally Black. Resumes belonging to candidates with “white-sounding” names received 50% more callbacks. These results were repeated in a 2016 study at Harvard Business School, targeting 16 metropolitan areas and including traditionally Asian names. Another study showed bias along class and gender lines in law firms.
candidate As a candidate, you will increase your chances of being judged by your strongest attributes by doing everything in your power not to be filtered by your resume alone. If you have a resume with no name-brand schools or companies, the reality is that you’ll face stiff odds at companies with many applicants who do. And even if you have a degree from an elite university and employment with a top company, you’ll still have more luck advancing through the hiring process at any company if you can get into the funnel later, via a warm introduction, referral, or any other connection you can find. A few things to consider when building a resume:
Attention to detail. Avoid typos, poor organization, or other easy-to-fix formatting problems. These can mean automatic rejection. Many people looking at a resume think, “If you can’t write one page without four typos, how are you going to do other work where attention to detail matters?”
Provide context. When describing work, include context on your role (and how it fit into the team) and the business impact it had so reviewers know why it was important and how you performed. An ineffective, mechanistic description would be: “Worked on Java backend of the AcmeTronix 7000.” Something like this is far better: “Was main developer of backend of Acme’s e-commerce platform used for $1M/month in transactions. Led team of four.”
Look forward. Highlight what you want your job to be, by writing your resume to show your aspirations, not just what you factually have done in the past.
Online challenges (or coding challenges) are tests that a company sends to prospective candidates in order to screen them for specific criteria early in the hiring process, usually before phone screens or more in-depth interviews. Online challenges typically involve a small set of predetermined programming problems. Either the company or a third-party company will have created them, and the hiring team will expect candidates to complete them in a specified amount of time and without any assistance.
When candidate volume is high, screening mechanisms like online challenges can make sense; they are easier for companies to scale and assess than other types of evaluations, and they are less biased than traditional screening mechanisms like resumes. Hiring teams use online challenges to identify unqualified leads.
Companies can learn a lot about candidates by giving them a homework assignment in which they have to write code to solve a problem. These technical screens are almost always scored automatically—once the candidate writes and submits code, the assessment system runs it against a bunch of tests to verify that the code outputs are correct and that the code runs efficiently.
story “At Dropbox, we would go to a campus and would get a thousand people who would apply from that school for an internship, but we’d only have five interviewers on the campus. We would have them use Hackerrank X (a code-screening software) to tackle a simple problem, because what we didn’t want to do was destigmatize the really good people in the pool from doing it—it was basically, ‘can you code or not?’ We were trying to differentiate between classroom students and engineers.” —Scott Woody, former Director of Engineering, Dropbox
Examples of widely used assessment tools include HackerRank, Codility, CodeSignal, and Qualified.io.
Because these assessments take minimal time and effort for companies to give out, they’re an appealing approach to making hiring a bit more fair. The idea is that if someone can pass a coding challenge, regardless of how they look on paper, they should get to advance to an interview. In some cases, online assessments can replace reliance on traditionally poor-signal screening formats like resumes: there is less bias in testing someone’s code output than in placing worth on where they went to school.
Still, there is value asymmetry in online challenges.
Value asymmetry is a state of affairs in which one party derives greater value from a transaction than the other. Value asymmetry in hiring and recruiting can arise when companies require candidates to expend significant effort in the interviewing process, such as asking a candidate to complete a lengthy coding challenge that provides little benefit to them. Value asymmetry can lead to frustration, disaffection, and candidates dropping out of the hiring funnel.
If your company is a household name, you can get away with having value asymmetry in your process because candidates value the chance to work for you so highly. But there are very few companies that are household names; and until yours is one, it’s wise to weigh whether it makes sense to ask candidates to do something up front without giving them anything in return.
Source: Aline Lerner
To combat value asymmetry, companies often choose to send coding assessments just to candidates that they perceive as less marketable; that is, those that don’t look good on paper. One pitfall is that even when those candidates do well in these assessments, they may not get to move to the next step—either because they made a mistake on the online assignment or because after they pass, a recruiter may revisit the candidate’s resume and notice a practical reason that disqualifies them (like location).
caution It’s wise not to make the common mistake of assigning the same online coding challenges for all developer roles. Typically (1) frontend or fullstack, (2) data science or machine learning, and (3) devops roles all are different enough they should be assessed with different challenges. Effective hiring teams test for and assess candidates’ ability to fulfill the requirements of the specific role all along the funnel.
story “Online challenges are incredibly irritating. They’re things like, ‘Create a map of 8 by 14, and then write a program that navigates that in as little moves as possible.’ They’re asking you to build out chess boards for web developer positions. I’m a great programmer, but I’m not a computer scientist. I fail these things repeatedly. I know the people who can pass these kinds of challenges, and they’re the worst programmers I’ve ever worked with. For every one company that uses these, there are four or five that don’t—and I’ve worked for them instead.” —Aaron Saray, engineer
Many companies with high applicant volume at the top of the funnel use some kind of online challenge. They are automated, skills-based, and less biased than using resumes. If you choose the right platform, such challenges can increase hiring efficiency. But they will never give you a complete picture of a candidate’s ability to do a job, and high potential for false negatives remains, especially if you make the mistake of assigning challenges that have little or nothing to do with the skills required for the role. These are screens meant to disqualify poor fits, and are not useful for qualifying further.
The same principles for collecting signal from resumes apply to LinkedIn profiles, if the profiles are sufficiently filled in.
caution An incomplete or out-of-date LinkedIn profile presents a challenge. It’s common for people with significant experience to have a minimalistic profile.* As a recruiter or hiring manager, you may want to ask a candidate if their LinkedIn is current . At the same, demanding that they update it before entering your process can create unnecessary friction. It’s wise to be flexible.
With an ever-growing number of engineers contributing to open source, GitHub profiles and GitHub contributions have become a potentially powerful complement (or even a partial replacement) to LinkedIn profiles. Looking at someone’s code output in the form of an open-source project or GitHub repo can work well at the top of the funnel, if the candidate has produced a large volume of code—effort on the candidate’s part is small (they just have to show you where to look), and signal is high. (Later on, you may ask the candidate to dive deeper with you on one or two projects, as part of a past work sample interview.)
While GitHub stars and published code are powerful signals, it’s worth remembering that most engineers don’t have meaningful projects they can publish. A notable exception is students or other nontraditional candidates who deliberately publish projects on GitHub.
If an engineer has built a personal website or online resume it can be helpful to look at the site itself, both for links to projects and as a demonstration of frontend design or engineering, as appropriate.
Cover letters (and whether to include them when you’re a candidate or consider them when you’re an employer) have historically been a bit of a contentious topic, mostly because of the purported time they take reviewers to read. As a recruiter or hiring manager, you’ll be able to tell quickly if it’s a generic, copy-pasted form letter. If it is, the letter can’t tell you much. But if it’s not, then it’s absolutely worth reading—the candidate just gave you a huge window into who they are and how they communicate, in a way that a resume cannot. If the letter reads like a thoughtful, passionate human wrote it, then wise hiring teams will seriously consider talking to the candidate. This holds true even if you’d be on the fence just judging from their resume alone.
The converse doesn’t always hold true, however. If a candidate looks competent but isn’t over the moon about your company yet and hasn’t gone out of their way to show their enthusiasm, that’s not a reason to reject them. If you feel strongly that they’d be great in the position, it’s best to let them know you want to talk—you’ll have plenty of chances to sell them later.
Despite changes to technical hiring practices, related hiring practices have been slow to catch up—especially when it comes to filtering candidates through the top of the funnel. Companies typically set up hiring processes to gate-keep rather than sell. But the demands of the market dictate that technical hiring be more about selling than filtering. The difficult part for companies isn’t sorting through a sea of candidates to figure out who’s “the best” or who’s worth engaging with.* Rather, technical hiring is a sourcing problem. A successful effort begins with getting candidates interested enough to talk to you in the first place and remain engaged as they go through the hiring process.
At a high level, selling is no different in the context of recruiting than it is in more traditional sales of goods and services. You talk to your customer (the candidate), ask questions, listen closely, and learn about their past, what their pain points are, and their hopes and dreams—and then you weave a carefully crafted narrative about how working at your company can actually deliver on those things.
Selling starts with writing great job descriptions that focus on how the role will improve the candidate’s life. This may include learning new things, gaining more responsibility, helping to fix problems in the world that matter to them, and so on. Selling ends with compelling offers tailored to the candidate’s needs. And in between, a good process will treat every candidate like a unique individual, never taking for granted that they have plenty of other options.
How is this approach different from what most employers are doing?
Read a typical job description, noting the ubiquitous laundry list of sometimes literally impossible requirements, and you’ll see that the typical hiring process is set up to exclude rather than attract. Many people involved in technical recruiting and hiring take for granted the fact that the people whose applications cross their desk want to work at their company. The common belief is that it’s not worth the time to engage with anyone who doesn’t fit an “ideal” mold.
Given the data available on the internet, a recruiter could reach out to every engineer at Facebook or Google who went to MIT or Stanford. That data is already out there, and between LinkedIn, Clearbit, Entelo, other candidate search aggregators, with a little bit of quick scripting, doing this isn’t that hard. What’s hard is finding the ones who want to talk to you, and finding the right fit between candidate and company. Companies who understand this are orders of magnitude more successful than ones that don’t, and the implications of a candidate-friendly market are what drive wise tactics at every stage of the modern technical hiring process.
Another important reason to develop a better top-of-funnel process is that, despite the volume of candidates out there, traditional hiring practices really only serve candidates who meet a specific small set of qualifications. Every company pushes pedigreed candidates—those with four-year degrees from top schools, who have already worked for a FAANG company—through their funnel, leaving candidates from nontraditional backgrounds feeling like they don’t hold a lot of cards. This is not a failure on the part of those candidates, but rather represents maladaptive hiring processes that smaller companies copy from the bigger, more established companies.
No matter where talented engineers come from, they won’t join your company unless someone sells them on the idea. This requires optimizing the time you spend on the candidates who want to talk to you. Filtering well at the top of the funnel becomes even more important, so that you do not waste time on candidates who won’t engage. There are a number of things you can do to sell candidates on joining your team when you are making the effort to diversify your pipeline. Each of these methods will help all potential employees feel supported.
This represents a big departure from the more traditional filtering approach, where you would try to save time by cutting candidates who aren’t skilled enough to make it through your process. Instead, this approach entails thinking about how you might gauge whether a candidate is likely to want to work for you. Maybe they feel alignment with your mission (or just the more nebulous alignment with your general approach to problems, like disrupting an antiquated, inefficient market space with new technology). Or perhaps they feel excitement about your stack, a history of thematically relevant past projects, or something else. Of course, the hiring process will always contain deal-breakers like whether or not the candidate has the appropriate level of seniority, specific skills, or location; but once those are off the table, the main question isn’t “does this candidate seem likely to fail my interview process based on their qualifications?” but rather “would this candidate be excited to work on the stuff we’re working on?”
A new wave of tools and credentialing approaches has arisen in recent years to help meet demand for talent in an increasingly competitive hiring market. Assessment tools like online challenges and hiring marketplaces can surface and filter engineers based on skill, rather than pedigree. Educational institutions like bootcamps train new engineers and connect graduates directly with employers. Companies can use these tools in conjunction with resumes, but it may be possible to effectively broaden the applicant pool by ditching the use of resumes in hiring altogether.
Software engineering credentialing is changing, and fast. Enrollment in undergraduate computer science programs is growing linearly (and not at all quickly at the very top schools companies recruit from most). On the other hand, enrollment is growing exponentially in software engineering-focused alternative programs/bootcamps and MOOCs. Those who can find a way to surface talent using other credentialing means are going to have a serious advantage for some time.
Aline Lerner provided major contributions to this section.
Most recruiters spend 7.4 seconds on a resume before deciding whether to advance a candidate. This is troubling because a resume is a collection of weak and imperfect signals—some positive, some negative, and some just informative. The purely informative signals can’t wisely be used for screening, but they do provide more information about a candidate for later discussion.
A well-formed resume is like a map of where someone can dig later in the interview process.Scott Woody, former Director of Engineering, Dropbox