Interviewing engineering executives.

executive (39), management (203)

Earlier I wrote about getting hired as an Engineering executive, and it’s perhaps even more important to discuss the opposite question: how should you interview and evaluate Engineering executives? As an Engineering executive, you may not directly run one of these searches, but you’ll likely be asked for advice about how to run them, and may be asked to design the process to hire your successor.

The key topics I want to explore are:

  • Avoiding the unicorn search
  • How interviewing executives goes wrong
  • Structuring your evaluation process
  • Focusing on four areas to evaluate engineering executives

These topics will prepare you to conduct an engineering executive search that culminates in hiring a leader that can support your company today, and will help you avoid bogging the executive team down in a multi-month process that is ambivalent about potential candidates.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

While most companies struggle to evaluate incoming executives effectively, there’s a second category that are effective at evaluating executives but nonetheless fail to hire effectively because they want a rare intersection of skills. For example, I once saw an engineering executive search that wanted someone with experience leading a large Engineering function, with deep go-to-market and Product experience, deep domain exposure to a narrow infrastructure engineering domain, cultural alignment with consensus-based decision making, and a sufficiently strong motor to skip consensus-making to accelerate company processes.

There is always some candidate who fits any mold you define, but hiring them gets very challenging. Identifying and hiring them is even harder once you acknowledge the breadth of the error bars inherent to this process. Worse, you usually can’t go back to reactivate a candidate after you’ve passed on them, so even if you later realize an earlier candidate was excellent, that prospect will have already passed you by. If you run a narrow search for too long, by the time you open the search up, you may have already rejected your best potential candidates.

My biggest advice for avoiding the unicorn search is to get the search’s sponsor, generally the CEO for an Engineering executive, to spend time before the formal search talking to seasoned Engineering executives to assess the profile. These don’t need to be folks you could hire, and your goal isn’t really to hire them, rather it’s to listen to their feedback on the profile you want to hire and what could make your opportunity sufficiently compelling that a qualified candidate would accept it. This will take a few weeks, but will save you months of time in the long run.

How interviewing executives goes wrong

Typically, the hiring loop for a software engineering role starts out messy and is slowly refined into an effective hiring process as you make more successful hires. This post-hire calibration process is particularly important to reduce the number of false positives and false negatives in your interviewers feedback. Executive searches only make one hire, while evaluating for a very broad role, which makes these loops even harder to calibrate.

These loops are further made challenging because there’s rarely someone wholly qualified to assess the potential engineering executives, who are being hired to be the most senior technical leader at the company, but despite that gap you’ll have many folks with strong opinions about who should be hired. Combining these messy incentives and challenges, most companies bouncing between two interview formats:

  • Vibes and backchannel: hiring is heavily weighted on a small number of discussions along with backchannel references who provide input on candidates previous work. These processes generate very little direct signal, which means that internal colleagues often don’t feel bought into the hires. Even the candidates themselves may not feel particularly evaluated either, which may cause them to decline the offer.
  • Broadchurch: hiring incorporates a wide range of internal interviewers. This might include an interview with the CEO, Product, Design, People, Finance, Marketing, Sales, along with four or five folks from Engineering. Introducing this many interviewers, many who will be unpracticed at interviewing for this role and may rely on an entirely informal interview, will generate numerous false negatives, often anchoring evaluation on the perspectives of folks without clear evaluation criteria and limited exposure to the role you’re hiring.

My experience is that neither of these are particularly effective at evaluating candidates, with the former accepting too many candidates and the latter rejecting candidates randomly. Further, these experiences leave the candidates themselves skeptical of the company’s decision making.

Structure for evaluating executives

Fortunately, it’s straightforward to design a reasonable process that’s comparable to most engineering executive evaluations and avoids some of the common missteps. That’s not to say that this process is perfect, but rather than most obvious changes introduce at least as many problems as they solve. I recommend starting with:

  • Recruiter screening: generally executive searches are run through an executive recruiter, an executive recruiting firm, or a VC recruiting firm. They should lightly filter for candidates’ interest in the role and their plausibility for getting an offer. My experience is that executive recruiters are excellent at this filtering as long as you listen to them!
  • CEO chat(s): make sure the CEO and candidate could work together well, with a focus on the candidate’s understanding of the opportunity and the core challenges. Don’t lean out of the challenges: the best candidates know the challenges exist and will be skeptical if you try to avoid or downplay them. Have as many of these as necessary to build conviction that the candidate is plausible and engaged.
  • 2-3 interviews with executive peers: have two to three executive peers interview. These interviews should explicitly cover the topics discussed in the next section, and should have a documented rubric for assessing candidates. A written rubric will particularly reduce the risk of false negatives and false positives, which unstructured executive interviews often introduce.
  • 30 minute presentation with a 30 minute Q&A: a short presentation given by the candidate that’s focused on their understanding of what they’d need to do in their new role is an excellent way to assess whether the candidate is listening throughout the process, and whether they have the executive acumen to operate within your organization. Avoid the temptation to expand attendees, and instead reuse the executives and CEO who have already met the candidate.

    Introducing more attendees will randomize evaluation rather than improve evaluation. For example, bringing in the Chief Marketing Officer (CMO) for the first time will often cause the CMO to observe that the presentation missed several key marketing needs, which is somewhat expected if the candidate hasn’t met anyone yet from Marketing. That’s a randomizing signal. If the CMO is indeed a key stakeholder then they should be one of the executive peers included in the previous step rather than added to the presentation.
  • Perform rigorous backchannel references: find three to four individuals who have worked directly with the individual in question for an extended period of time. I’m generally ambivalent about backchannels, but in the case of executives I believe the risk of hiring a poor executive is high enough that it’s an essential step. Candidate supplied references are not very high signal at this level, because the candidate will prepare their reference with talking points, including how to answer questions around gaps.
  • 2-3 interviews with members of Engineering: assuming the other steps have gone well, then end with several interviews with engineers and engineering managers. Your primary goal here is to build commitment to the candidate from within the Engineering team, but you also want to listen for any major concerns from Engineering. Your interviewers should be running a structured interview with explicit areas of evaluation. It’s not ideal, but acceptable, to get some lukewarm signals at this stage, as long as you don’t get any major concerns. It’s very rare to hire any new manager whose team doesn’t have some concerns.

At this point in the process, either go to offer or decide not to extend an offer. Resist the temptation to hedge. Delaying sends candidates a bad message, and there’s rarely additional information out there that will change your mind for the better.

Four areas of evaluation

There are an unlimited number of skills to assess executives on, and ultimately there are more skills than you can viably assess. I recommend drilling in on these areas:

  • Executive skills: are they an effective listener and communicator? Do they have the fundamental skills expected of an executive at this level, such as operating to a financial plan, supporting a single or multi-business unit organization, running a hiring or performance process, etc? You’ll get a signal on this from the presentation, sessions with peer executives, and from backchannel references.
  • Role and company specific skills: every executive role you’re hiring for is aimed to solve a handful of specific problems at your company, and you should assess on those dimensions. In some cases this is improving partnership between Sales and Engineering, in other cases it’s improving Engineering velocity, and in others it’s partnering more effectively with peer executives. Identify whatever these are, and ensure that you explicitly cover them in either the CEO or peer executive sessions.
  • Engineering functional expertise: depending on how you’ve scoped your engineering executive role, you’re going to want some sort of functional expertise. In some companies this is deep on running a scaled organization, guiding new product development, partnering between Engineering and commercial functions, technical architecture, or even infrastructure. Whatever it is for the role you’re hiring for, you should ensure that either the engineering interviews or the peer executive interviews cover these points.
  • Historical performance and behavior: use your backchannel references to get an accurate understanding of the candidate’s true performance and behavior over time. There are effective executives who leave a trail of angry peers behind them. Similarly, there are very ineffective but beloved executives who remain far too long at companies that they serve poorly. You can assess self-awareness by asking about these directly, but you can only assess actual performance by talking to folks who were there. Good executives can spin even the worst performance into something positive, which means you simply cannot rely on them to self-evaluate.

You’ll note that I’ve not provided a checklist of skills to evaluate against. This is deliberate, because the role of a strong engineering executive is exceptionally broad, and reducing it to a list of skills will distract you from evaluating what is particularly valuable to your search. Most current CTOs and VPs of Engineering out there are the wrong fit for your role at your company. Evaluate on the specifics, not the universal.

Summary

You now know how to avoid the unicorn search, avoid bringing too many interviewers into the process, and how to evaluate the particulars of what you need rather than anchoring too heavily on vibes. Even with all of that in mind, these are still difficult searches. Don’t get discouraged if it takes you five or six candidates before you find someone you’re excited about, this is a natural part of learning how to hire a new executive role. Conversely, do get worried if you’re not excited after talking to ten-plus candidates; that probably means your search is going a bit off the rails.