Interviewing senior engineering leaders.

A friend recently reached out for advice on interviewing and hiring senior engineering leaders. I’ve spent a good deal of time on this topic over the last couple years, starting with partnering with Laura Hilton to design Stripe’s interview loops for engineering leadership, and more recently going through a search for my own role. Leadership hiring is particularly interesting as a window into an organization’s psychology: for the highest stake decisions, do they turn to structure or to intuition?

For many topics I can write the algorithm that I’m confident will lead to a pretty good outcome, but this definitely isn’t one of them. Rather these are just the notes of what I’ve seen so far.

For clarity’s sake, I’m defining “senior” as managing an engineering organization of 30+ folks, likely with a Director-plus title at a pre-IPO company or the VPE/CTO at a smaller startup. These ideas will apply to folks outside of these narrow lanes, but will require some adaptation.

Why this is hard

While hiring can be difficult in general, I’ve found hiring at this level to be challenging in a few particular ways. First and foremost, companies want so much from these hires: someone who has managed a team of 300+, has experience in a particular niche, and has worked at a trendy company. These intersecting requirements empty out the candidate pool, leaving behind a small number of folks that are highly sought after.

Further, you’re hiring for a skillset you’re missing, which makes it hard to know exactly how to evaluate the candidates. Worse, while these folks will fit many aspects of your culture to some extent, you’re also hiring them to bring a new influence into your culture, which may generate mixed results from an overly rigid approach to assessing culture fit.

Neither can you rely on hiring volume to teach you what you’re looking for, as you’ll hopefully only hire one such candidate at any given stage of company size and complexity. By the time you hire another CTO, you’ll be looking for a very different set of skills.

Finally, senior folks who invest in preparation tend to interview really, really well. Well enough that it’s very difficult to pierce the veil and get an accurate read on who they are. This is especially true if you rely on unstructured and intuitive evaluation.

The Standard Process

Well, there isn’t really one standard process, but the majority of what I’ve seen and heard of compresses into something fairly uniform. The interviews that you’ll generally see in these processes are:

• Topgrading where they dig into your accomplishments and trajectory over the course of your career. These look for career acceleration across each role. Note that most places won’t call these topgrading, and may not even be aware their interview is influenced by topgrading.
• Deep-dives to ask about your experience in relevant topics, such as building an engineering brand, hiring, structuring organizations and so on.
• Group presentation to execs, peers or team about a relevant topic. One frequent topic is around your first ninety plan.
• Group Q&A to evaluate your ability to speak extemporaneously, as well as to get the room excited about working with you.
• Working session with would-be manager or peers on a relevant problem.
• Fit conversations with critical stakeholders, including the folks you’ll manage, to learn about you and understand your approach, beliefs and style.
• References that you provide to learn about you from your coworkers.
• Backchannel references to learn about you, ensuring they don’t only hear from folks particularly positively predisposed towards you.

The majority of processes I’m aware of are some combination of those, over the course of six to twelve distinct interviews, emails, phone calls and texting threads.

My sense is that this approach does a sufficiently good job of assessing folks against the hiring company’s current needs, but often struggles to assess “around the corners” of role evolution over the next year or two. The general lack of structure also, in my opinion anyway, generally does a better job of identifying gaps rather than strengths. I often struggle to reconcile interview performance with someone’s considerable successes, and I’m certain at least a fraction of that is due to this process extracting insufficient signal.

Are backchannels unethical?

As an aside, when I’ve chatted with folks about the standard process, the most contentious area is around the acceptability of backchannel references. I’ve been a long-time rejector of backchannel references, and can recite all the arguments against them.

Backchannels disadvantage the very folks who already have structural inequalities hindering them at each step. The sort of folks who are called intimidating where others might be described as bold; threatening where others might be described as boisterous. They can also be little more than gossip if you chat with folks who don’t work closely with the individual being back-referenced. They can expose the candidate’s job search in a way that creates considerable undue grief. Further, they represent a distrust in your ability to evaluate folks well – shouldn’t you trust in your process more?

I hear all of that criticism against the backchannel, and I agree with it: backchannels introduce considerable bias along with all the other issues.

However, I still think they’re necessary when hiring for executives. Rather than skipping backchannels, I think your obligation is to do them well. Listen to feedback very carefully and knowing full-well that it’s packed with bias. Ensure you ask folks who worked with the individual closely. Let the candidate know that you’re planning to do backchannel references and try to incorporate their guidance. (That said, as a candidate I’ve always accepted that this will happen without folks letting me know. It’s just “how it works” in my experience.)

Getting to the heart of things, I think backchannels are essential because executives are given immense opportunity to impact the lives of folks in an organization. Executive candidates are, with some exception, far more privileged than the folks they’ll soon be responsible for, and you owe it to the team to ensure that the person you’re entrusting them to hasn’t left a trail of ethical concerns behind them. Inconveniencing one relatively privileged individual to prioritize the needs of dozens or hundreds of relatively less privileged folks strikes me as totally reasonable behavior.

That said, I’d love to find better ways to acquire backchannel signal with fewer downsides. If folks have found better approaches, please send them over.

Demonstrating expertise

The approach that we took in Stripe’s most recent senior engineering interview refresh was to focus on folks demonstrating their skills rather than describing their skills. We brainstormed the activities that experienced managers were spending their time on, and designed interviews to emulate those activities. This approach is used to some extent in “the standard process” presentation interview, but we wanted to go much further.

Much of senior leadership is editing and providing advice to folks on your team, so we tested an interview where folks had to give feedback on a poorly written set of team goals. Another common activity is reviewing an organizational health survey, something along the lines of CultureAmp, so we synthesized an organizational health dataset and asked them to analyze it, identify themes and propose how they’d address each of those themes.

We also retained several roleplay interviews, which have long been a mainstay in Stripe’s manager interviews. These roleplays focused heavily on remaining empathetic while giving clear, direct feedback.

I found this approach gave a very clear sense of how directly and seriously leaders had engaged with these specific areas that we considered integral to successful senior leadership. They’re particularly good at extracting depth (relative to asking folks to describe a time they did something along those lines) and they get signal even if the candidate is an unstructured communicator (while still capturing the signal that they are an unstructured communicator).

This demonstration of expertise approach is certainly not the only sort of interview you should do, but personally I think it works quite well and I hope to shift more of my future interviews that direction.

What else?

It feels like there must be better techniques out there, so I asked, and got some interesting perspectives:

• Yvette Pasqua proposed more topics for demonstrating expertise like translating business strategy into technology strategy, navigating your P&L statement, mediating complex conflict
• Rushabh Doshi shared his post on Hiring Engineering Leaders, which links to Martin Casado’s Hire a VP of Engineering, and in particular pushes back on backchannel references
• TechTello suggested evaluating against the role a year out (not the role today), and digging into their willingness to take unpopular opinions, something negative they’ve learned about your company and how they’d try to change it, dig into your company values, design a layoff, and their questions around the company and role
• Paul Robinson referenced a post he wrote about The Problems of the CTO Role, which includes a short section on hiring focused on understanding what sort of CTO/VPE you wish to hire

I’m sure there are other good approaches out there as well.

Recommendation

Pulling all of this together, my general recommendation for interviewing senior candidates is to identify the tasks they’ll need to perform, design a loop to have them demonstrate mastery of the related skills, check for alignment on company values, and keep doing backchannel references.