Irrational Exuberance

Solving the Engineering Strategy crisis.

Sat, 23 Sep 2023 09:00:00 -0600

These are speaking notes for my October 4th, QCon talk in San Francisco.
Slides for this talk.

Over the course of my career, I’ve frequently heard from colleagues, team members and random internet strangers with the same frustration: the company doesn’t have an Engineering strategy. I don’t think this problem is unique to Engineering: it’s also common to hear folks complain that they’re missing a strategy for Product, Design or Business. But, whereas I don’t feel particularly confident speaking to why so many companies are missing a clear Business or Product strategy, I’ve come to have some clear opinions about why so many engineering organizations don’t have a written strategy.

I’ve been fortunate to be involved in architecture at many companies, including designing several iterations of Stripe’s approach to architecture (which taught me some lessons). From that experience, I’ve tried writing about this topic quite a few times:

Magnitudes of exploration documented a public version of Stripe’s Engineering strategy
Write five, then synthesize presents a methodology to drive Engineering strategy while operating as an individual contributor
Writing an engineering strategy describes how Engineering executives can lead Engineering strategy at their company
I also collected engineering strategy resources written by others into Staff Engineer’s appendix

In this talk, I hope to pull those ideas together, into a unified theory of Engineering strategy, with a particular emphasis on how you can drive strategy even if you’re not the company’s CTO. Another way to think about this talk, is that I hope to “Solve the Engineering Strategy Crisis” that so many people keep emailing me about.

What I will talk through

In this talk, I’ll work through five topics around engineering strategy:

Eng strategy is honest diagnosis + practical approach
It’s useful (↑dev velocity, ↓friction)
It’s everywhere, although rarely written
Written strategy is much more effective
You can advance strategy at your company

What is Engineering strategy?

Whenever I think about strategy, I start from Richard Rumelt’s Good Strategy, Bad Strategy, which three pillars of effective strategy:

Diagnosis - a theory describing the nature of the challenge. This is trying to identify the root cause(s) at play, for example “high work-in-progress is preventing us from finishing any tasks, so we are increasingly behind each sprint” might be a good diagnosis
Guiding policy - a series of general policies which will be applied to grapple with the challenge. Guiding policies are typically going to be implicit or explicit tradeoffs. For example, a guiding policy might be “only hire for most urgent team, do not spread hires across all teams.” If a guiding policy doesn’t imply a tradeoff, you should be suspicious of it (e.g. “working harder to get it done” isn’t really a guiding policy, the relevant guiding policy there might be “work folks hard and expect high attrition”)
Coherent actions - a set of specific actions directed by guiding policy to address challenge. This is the most important part, and I think the most exciting part, because it clarifies that a strategy is only meaningful if it leads to aligned action

I’ve found that definition extremely useful, and Rumelt’s views have shaped how I think about Engineering strategy. In particular, I believe that Engineering strategy comes down to two core components:

Honest diagnosis that engages with the reality your organization’s current needs and challenges
Practical approach to move forward while addressing the circumstances raised in the diagnosis

Sure, that sounds nice, but what does that mean? To clarify that a bit, let’s work through an example scenario. This is a scenario that many folks have experienced in their career:

You join a new company
Your team works in a Python monolith to build the Widget product
Your CTO hates monoliths, mandates service migration
You join a team building a brand new Hammer product in a new service
2 years later, your old team and Widget are still in the monolith
You have no idea how to share code between Widget and Hammer

I believe this sequence of events keep reoccuring because of bad strategy, and is preventable with good strategy. Lets work into the components of strategy to look at how strategy could cause and prevent this scenario from happening.

Starting with “honest diagnosis” and in particular, looking at what a bad honest diagnosis would look like for this scenario. (For the record, I don’t think “dishonest” is the opposite of an “honest” diagnosis, they tend to be “bad” rather than “dishonest.”)

Here’s a bad diagnosis:

“We can migrate from our monolith to services in three months.”
“We’ve derisked our approach by moving a meaningfully complex component out of our monolith.”
“We’re willing to invest heavily in migrating to services, even if it means slowing down product velocity in the short term.”
“We are willing to expand our Developer Tools team to build new tools for services in addition to supporting our existing monolith.”

OK, but then let’s briefly consider what a good diagnosis might look like:

“We can migrate from our monolith to services in three months.”
“We’ve derisked our approach by moving a meaningfully complex component out of our monolith.”
“We’re willing to invest heavily in migrating to services, even if it means slowing down product velocity in the short term.”
“We are willing to expand our Developer Tools team to build new tools for services in addition to supporting our existing monolith.”

Disappointingly, this is the same list in both cases. In a small startup with only one simple product, you probably can migrate from a monolith to services in a few months, maybe even less. In a larger startup, that’s almost certainly impossible.

An honest diagnosis is a reality-based assessment of your circumstances. Nothing is universally honest. (Neither is anything universally bad.)

Once you find a reality-based assessment to inform your honest diagnosis, the second half of your strategy, a practical approach. The most important thing to keep in mind is that a practical approach makes explicit tradeoffs that acknowledge your real constraints, for example, here are some good approaches, even if they are a bit painful to write:

“We want to migrate to services, but are unwilling to staff Dev Tooling more, so the migration will happen in 12 months after tooling gets finished.”
“We don’t adopt additional programming languages, even if we prefer them, because we don’t have capacity to support them.”

What makes these good is not that they’re beautiful, ambitious statements of how we work. These are not loft “engineering values”, they are specific acknowledgments of how you’ll navigate your constraints.

Thinking back to our scenario with Hammer and Widget products, our practical approach might look like:

Expand Developer Tooling team by 2 engineers for next year
Those additional engineers will focus on tooling for services
Before committing to our services migration, we’ll validate by moving the Widget product to a service, and operating it as a service
If we can’t exceed monolith productivity within Widget, we’ll migrate back
No other products are allowed to spin up new services until we’ve validated the Widget migration was successful and a significant improvement (as measured by % of product eng team’s time spent on features combined with number of major Widget product ships relative to last year)

Once again, tragically, a practical approach depends on your company and your circumstances. You could write the same exact practical approach and have it go very badly indeed, which is why senior leaders often fail when they reapply familiar strategies at new companeis.

Hopefully you’ll accept the definition of “engineering strategy = honest diagnosis + practical approach”. Next, is to try to convince you that this definition is actually useful.

Engineering strategy is useful

Let’s start making the case for engineering strategy by talking through some practical examples of enginering strategy that I’ve encountered in my career.

Stripe – “We run a monolith in a monorepo.”

Diagnosis:

We work in a business with dynamic external forces–regulators across each country, numerous financial partners like banks, and growing enterprise customers–that change frequently and unexpectedly
We integrate with thousands of external financial infrastructure that are filled with bad, inconsistent, buggy technology and numerous human-driven processes
We have a meaningfully complex financial platform (e.g. money movement) internally that our other products (e.g. Stripe Connect) are built on

Approach:

We need our entire risk budget to respond to external changes
We reduce technology risk by running a Ruby monolith in a monorepo
Our developer tooling team invests heavily in running Ruby and our monorepo at scale
Exceptions to the above are narrow and rare (data engineering, tokenization environment)

Impact of Stripe’s strategy:

Innovation budget (mostly) went into product, not infrastructure
Avoided the decade-long journey into (micro)services that distracted most contemporaneous technology companies
Narrow technology landscape made it possible to concentrate investment into technologies like the Sorbet (static typing for Ruby) without an outsized investment with developer tooling

Calm – “We’re a product engineering company.”

Diagnosis:

We’re spending a lot of time arguing about adopting new technologies
We seem to be adopting new technologies out of interest in using and learning about new technologies
We have a long-running services migration, but only small infrastructure and platform components have been moved out. All product engineering code remains in our monolith
Our developer tooling team is split between supporting monolith and service workflows

Approach:

1.. We are a product engineering company 2. We adopt new technologies to create valuable product capabilities 3. We do not adopt technologies for other reasons 4. We write all code in the monolith unless there is a functional requirement that makes it extremely difficult to do so 5. Exceptions to the above are granted exclusively by the CTO, who will approve in writing in the #engineering channel

Impact of Calm’s strategy:

We stopped arguing about technology investments
We exited several engineers who didn’t want to follow our strategy
Combined, this meant we could consolidate our tooling investments into our TypeScript monolith
We started spending our innovation chips on product enhancements, culminating in ML-powered algorithm to determine best content for each user based on their behavior, UI to allow content team to self-service content management rather than require engineering support, and so on
This was initially viewed, by some, as making it “less fun”, but ultimately meant we spent a lot more time having doing fun work that both stretched us as engineers and helped our users

Uber – “We run our own hardware.”

Diagnosis:

Uber was going through a period of rapid geographic expansion
Some of those geographies lacked a meaningful cloud presence
We were operating at a scale, X0,000s of servers, where economic impact of 20-30% lower cost of ownership from managing our own hardware was meaningful
We were willing to incur the cost of not having access to useful cloud

Approach:

Run exclusively on our own hardware in dedicated colo space
Do not store data or compute in the cloud
It’s OK to do networking (e.g. TLS termination) on cloud, along the line of a Point of Presence (POP)
Any cloud experiments beyond POPs will require CTO approval

Impact of Uber’s strategy:

We were able to enter, and remain within, regions that cloud-reliant competitors would be unable to maintain operations within in the case of shifting data locality regulatory changes
Concretely, we were able to spinup datacenter in China in ~6 months, without colocating our US or EU data
(Aside – this was very painful, I don’t recommend it)
We did a lot of Not Invented Here (NIH) to replace common cloud tooling
(Life is tradeoffs: even good strategies have undesirable consequences!)

Why do these strategies work?

These strategies are effective for a few reasons:

Many interesting properties only available through universal adoption (“we run our own hardware”)
Concentrate tooling investment onto smaller space (“we run in a mono repo”)
Reduce energy lost on conflict (“we are a product engineering company”)
Control your innovation budget (all three)
New hires, especially senior new hires, forced to engage explicitly with strategy rather than having option of ignoring it (all three)

This is the power of making explicit, consistent tradeoffs across an entire organization.

Absence shows value as well

In addition to arguing the value of strategy from these positive examples, it’s easy to find negative examples where a missing or inconsistent strategy caused a great deal of pain:

Digg’s 3+ year migration to V4, onto a 100% new codebase with a new database, new frontend, new backend, and new algorithms. Honest diagnosis about challenges, but highly impractical approachs
Stripe’s introduction of Java had unclear evaluation criteria, took years to assess effective. Rooted in inaccurate diagnosis about problems at hand
Uber’s invested heavily in competing routing technologies, causing significant friction. Rooted in simultaneous following conflicting approaches without aligning on approach

I’m sure you can think of examples from your careers as well!

Strategy is everywhere. Written strategy is rare

Interestingly, Uber and Stripe are well-known technology companies, and I wrote a bit above about their technology strategies were, but neither were particularly proactive at writing their strategies down.

I’ve come to believe that:

Most companies do have an engineering strategy
Awareness of that engineering strategy is often inconsistent
It’s very rare for a company to have a written engineering strategy

This is the first really important takeaway from this talk: you can solve half the engineering strategy crisis by just writing stuff down.

We’ll get to solving the other half in a second.

Written strategy is more powerful

There are probably an infinite number of reasons why written strategy outperforms implicit strategy, but a few that I’ve seen matter in particularly important ways are:

You can get feedback on it
You can make updates to it
You can explain why you made updates to it!
You can clarify points of confusion
Nuance is important, and almost impossible in unwritten strategy
It democratizes technical decision making beyond a small caste of architects
You can hold people accountable for not following it
New hires can learn proactively rather than “fail their way into learning”

You can drive Engineering strategy

Two primary ways:

From below: how you can rollout strategy without being the CTO engaging
With above: how you can rollout if the CTO’s bought in

Top-down

This strategy is a modified version of the one describes in Writing an engineering strategy. At it’s core, the thing to recognize is: it’s easy to get CTO buy-in if you write the strategy that the CTO wants.

To do that:

Align up frequently, and take time to debug their feedback
Be trustworthily curious: folks know you’ll listen hard to understand their point
Be pragmatic rather than dogmatic
Have a track record of Doing The Work to build buy-in
Frame it as a low-risk experiment, “We’ll try for 3 months then reevaluate”
Let CTO decide how to break ties

If you’re reading this and your biggest thought is, “My CTO will never let me do this”, then 7 out of 10 times, I promise you that either you’re not writing the strategy that the CTO wants. The other 3 out of 10 times, there’s some internal conflict that the CTO just isn’t willing or able to resolve, which is a bit trickier, but you can approach via the next strategy.

Bottom-up

The approach to bottoms-up rollout is described in Write five, then synthesize:

Write 5 design docs
Synthesize those design docs into a “narrow strategy”
Do the above five times, until you have 5 “narrow strategies”
Synthesize those five into a “broad strategy”
You just wrote a really good engineering strategy

This approach definitely takes a long time, but I’ve seen it work a number of times. Even if your current strategy has some gaps in it, birthing it into an explicit strategy document will always make it much easier to address those gaps.

Recap

Here’s what we talked about:

Eng strategy is honest diagnosis + practical approach
It’s useful (↑dev velocity, ↓friction)
It’s everywhere, although rarely written
Written strategy is much more effective
You can advance strategy at your company

Within those topics, the two disappointingly straightforward steps that you can talk to solve the engineering strategy crisis are:

Writing down the existing strategy
Using either tops-down or bottoms-up approach to improve the quality of your existing strategy

This might not be what you were excited to do when you wrote about getting more strategic in your annual goals, but it’s what actually works.

Drafted Eng Executive's Primer!

Mon, 04 Sep 2023 04:00:00 -0600

Back in late April, I mentioned that I was working on a new book, The Engineering Executive’s Primer, with O’Reilly. I wanted to share a few notes on progress!

First, there’s a cover, shown above in this post’s image, and also in the right rail (or bottom footer if you’re reading on a smaller device). I’m quite excited about the cover, which is simple and imperfect. There is nothing pure about being an executive; it’s mostly about balancing opposing forces to the best of your ability, and I think the cover captures some of that. The map underneath the cracks is an early map of San Francisco’s Golden Gate Park (if you want further proof, try searching for “Stow Lake” whose label you can see peeking through in the crack on the right side).

Second, I’ve done a lot of writing. I’ve been sharing early chapters with the “executive” tag, which now has 28 posts, all except one of which are from this year. Every one of those is an idea that I intended for the book. Some will be in the book exactly as is (well, almost exactly, they all still need some editing), others have been trimmed down to asides to include within other chapters, and just a couple of them didn’t end up fitting (e.g. the post on creating executive LinkedIn profiles was top of mind for me as I was reworking mine for the job search that helped me connect with Carta, but there’s no advice I can write about any tool that’s truly evergreen advice–tools change too often).

At this point, I am nominally done writing, although what I really mean is that I’ve finished the first draft. There’s still quite a bit of editing, including incorporating feedback from an amazing group of tech reviewers (ty Jasmine, Julia, Kevin, Tanya, Uma, and Virginia), which I hope to finish over the course of September.

From there, there’s copy editing, perparing the book for printing, actually printing the book, and so on, but most of that won’t require much direct involvement from me. That means, we should be on track for the digital version being complete by the end of this year, and the physical release by June, 2024.

This is my third book, and I’d say that I have a pretty clear sense of how to write this sort of book, so it hasn’t been a particularly tortured experience pulling it together. It certainly helped that I had a couple months winding down at Calm before starting at Carta, which gave some space to focus on outlining and writing the book. I’m pretty sure I couldn’t have written this while ramping up at a new job if so much of it hadn’t already been pulled together. In particular, the chapters that I think are exceptionally good were all written by the time I started, including Writing an engineering strategy, which I hope will be the enduring piece from this book. (Perhaps that’s wishful thinking, as it’s a topic I’ve been trying to land for a long time now.)

Alright, now I’m off to edit, prepare for a talk on engineering strategy at QCon San Francisco in October, and continue my work at Carta.

Performance & Compensation (for Eng Execs).

Sun, 03 Sep 2023 07:00:00 -0600

Uber’s original performance process was called “T3B3” and was remarkably simple: write the individuals top 3 strengths, and top 3 weaknesses, and share the feedback with them directly in person. There was a prolonged fight against even documenting the feedback, which was viewed as discouraging honesty. On the other side of things, there are numerous stories of spending months crafting Google promotion packets that still don’t get their authors promoted. Among those who’ve worked within both Uber and Google’s promotion processes, there are advocates and detractors, and absolutely no consensus on what an ideal performance process looks like.

Compensation is a subtly different set of problems, but similarly there are no universally appreciated compensation processes out there. Highly structured, centrally orchestrated compensation systems often converge on most folks at a given level receiving similar compensation, even if their impact is quite different. More dynamic compensation systems disproportionately reward top performers, which introduces room for bias.

Because there’s no agreement on what performance or compensation process you should use, you’ll likely end up working within a variety of systems. This post digs into:

The conflicting goals between those designing, operating, and participating in performance and compensation processes
How to run performance processes, including calibrations, and their challenges
How to participate in a compensation process effectively
How often you should run performance and compensation cycles
Why your goal should be an effective process rather than a perfect one

Every one of these systems is loaded with tradeoffs and traps that you’ll need to be wary of, and after finishing this post, you should be prepared to plot the right course for your organization through them.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Conflicting goals

Going back to Uber’s T3B3 performance process–where you told someone their top and bottom three areas for a given half–what’s most remarkable was its radical simplicity. It was focused exclusively on providing useful feedback to the recipient. To this day, I find that clarity of purpose very remarkable, and genuinely rare.

Most performance and compensation systems have far less clarity of purpose, because they try to balance many priorities from many stakeholders. Your typical process at a given company is trying to balance all of these goals:

Individuals: want to get useful feedback to grow, want to get promoted as soon as possible, want to maximize their compensation, and they want to do it quickly.
Managers: want to provide fair and useful feedback to their team, want to promote their team as appropriate (and in alignment with the various commitments they’ve made to their team), want to provide appropriate compensation (and, once again, do so in alignment with the various commitments they’ve made to their team). Oh, and they also want to do it quickly as well!
People (or Human Resources) Team: want to ensure individuals receive valuable feedback, create a “floor” for quality of feedback that individuals receive, document feedback that can be used in performance management and legal scenarios later to support the company’s perspective, need to demonstrate performance process for compliance purposes (e.g. SOC2 requires annual performance reviews), and create structured input to include in calculating compensation.
Executives: decide who to promote based on inconsistent evaluations across managers, optimize allocation of a fixed compensation budget to meet organizational objectives, and minimize the impact of inexperienced and misaligned managers on promotions and compensation.

I’ve never encountered, or heard of, a process that solves all these problems elegantly. My informed guess is that there simply isn’t any process that works with hundreds of people that isn’t a bit laborious to operate within. There’s also no way to flawlessly balance the goals of objective, consistent outcomes and recognizing exceptional individuals.

There’s a lot of room for improvement in these processes, and they can absolutely always be improved, but the tension in these process is inherent to the participants’ conflicting goals. These conflicting goals are real, fundamental, unavoidable, and must be kept in mind as you make decisions about how your process works.

Performance & Promotions

I’ll start out talking about performance processes, including promotions. Your baseline performance process is each manager providing written feedback for each of their direct reports, including a decision on whether to promote them, but there are quite a few details and variations to consider.

The first variations to consider are whether to include peer and upward feedback. Upward feedback is a constrained problem, as each person should only have one manager. In the worst case, asking for upward feedback generates low-value feedback, often because the individual doesn’t want to criticize their manager, but it doesn’t take up too much time.

Peer feedback can take up a significant amount of time, particularly for highly connected individuals who may be asked to provide peer feedback on ten or more individuals. This is usually accompanied with the advice that you can decline peer feedback requests if you get too many, but many individuals find it difficult to decline peer feedback requests, even if they know they should.

More importantly, my experience is that peer feedback is very inconsistent, and I’ve come to believe that each’s team’s beliefs are the value of peer feedback determine whether the feedback is actually useful. I’ve managed teams who feel peer feedback is too uncomfortable to give honestly, and those teams have provided useless peer feedback: in those cases, it’s not worth collecting peer feedback. I’ve also managed teams who believed feverishly in the value of peer feedback, and those teams generated insightful, valuable feedback. As such, I prefer to lean towards empowering managers to make the decision on collecting peer feedback for their team. Often this is a policy decision enacted for the overall company, and in that case it’s not a battle I’d pick.

Levels and leveling rubrics

Agreeing on performance ratings and who should be promoted is nearly impossible without written criteria that describe the dimensions of expected performance for each level. However, before we can talk about leveling rubrics, first we have to talk about levels.

Most companies have paired titles and levels, such as:

Entry Level Engineer (Level 3)
Software Engineer (Level 4)
Senior Software Engineer (Level 5)
Staff Software Engineer (Level 6)

The specific levels vary widely across companies (there many sites that show how levels differ across companies), and what is a “Level 3” at some companies might be a “60” at another, and a “601” at a third. There is no consistent leveling standard across companies. It’s fairly common for Software Engineering levels to start at “Level 3”, as companies use levels across many functions, and often reserve “Level 1” for entry-level roles in roles with fewer entry requirements.

Titles vary even more widely across the industry, and there certainly isn’t a universal standard to adopt. If you are in the position of setting titles for your company, I recommend using the fairly typical progression of Entry-Level Software Engineer, Software Engineer, Senior Software Engineer, Staff Software Engineering, and Sr Staff Software Engineer. If you’re tempted to experiment with new titles, note that the downside is that it makes your hiring process more complex since you have to explain what the titles mean, and you will lose some candidates who are worried the non-standard titles will harm their career trajectory.

Once you establish Engineering’s titles and levels, the next step is documenting the leveling rubrics that describe expectations for operating within each level (again, there are a variety of sites that collect publicly available leveling rubrics from many companies). This can be a very sizable endeavor, and I’d recommend skipping the hardest part by picking a reasonably good one that’s available online, creating a working group to tweak the details, and then refining it after every performance cycle to address issues that come up.

Additionally, I’d emphasize a few things that I’ve learned the hard way over time:

Prefer concise leveling rubrics over comprehensive ones: there’s a strong desire for leveling rubrics to represent the complete, clear criteria for being promoted. The challenge, of course, is that many folks are exceptionally good at gaming specific criteria. For example, Stripe’s promotion criteria included mentorship, and I encountered folks who claimed to mentor others because they scheduled a meeting with that person, unrequested, and said that constituted mentorship.

Concise rubrics require more nuanced interpretation, but attempts to game rubrics mean that all options in practice require significant interpretation. You can respond to each attempt at gaming with even more comprehensive documentation, but your rubrics will quickly become confusing to use, more focused on preventing bad behavior than providing clear guidance for the well-intentioned.
Prefer broad job families over narrow job families: a classic executive decision is whether Site Reliability Engineers and Software Engineers should have different leveling criteria. Let’s say you decide that yes, separate criteria would be more fair. Great! Shouldn’t you also have separate rubrics for Data Engineers, Data Scientists, Frontend Engineers, and Quality Assurance Engineers?

Yes, each of those functions would be better served by having its own rubric, but maintaining rubrics is expensive, and tuning rubrics requires using them frequently to evaluate many people. Having more rubrics generally means making more poorly tuned promotion decisions, and creating the perception that certain functions have an easier path to promotion. I strongly recommend reusing and consolidating as much as possible, especially when it comes to maintaining custom rubrics for teams with fewer than ten people: you’ll end up exercising bespoke judgment when evaluating performance on narrow specializations whether or not you introduce a custom rubric, and it’s less expensive to use a shared process.
Capture the how (behavior) in addition to the what (outcomes): some rubrics are extremely focused on demonstrating certain capabilities, but don’t have a clear point of view about being culturally aligned on accomplishing those goals. I think that’s a miss, because it means you’ll promote folks who are capable but accomplish goals in ways that your company doesn’t want. Rubrics–and promotions–should provide a clear signal that someone is on the path to success at the company they work in, and that’s only possible if you maintain behavioral expectations.

My final topic around with levels and leveling rubrics is that you should strive for them to be an honest representation of how things work. Many companies have a stated leveling and promotion criteria–often designed around fairness, transparency and so on–which is supplemented by a significant invisible process underneath that governs how things actually work. Whenever possible, say the awkward part out loud, and let your organization engage with what’s real. If promotions are constrained by available budget and business need, it’s better to acknowledge that than to let the team spend their time inventing an imaginary sea of rules to explain unexpected outcomes.

Promotions and calibration

With leveling criteria, you can now have grounded discussions around which individuals have moved from one level to another. Most companies rely on managers to make a tentative promotion nomination, then rely on a calibration process to ratify that nomination. Calibration is generally a meeting of managers who talk through each person’s tentative rating and promotion decision, with the aim of making consistent decisions across the organization.

In an organization with several hundred engineers, a common calibration process looks like:

Managers submit their tentative ratings and promotion decisions.
Managers in a sub-organization (e.g. “Infrastructure Engineering”) meet together in a group of 5-8 managers, including the manager responsible for all of the sub-organization (e.g. “Director of Infrastructure Engineering” who the other managers report into), to discuss each of the tentative decisions within their sub-organization.
Managers reporting to the Engineering executive meet together with the Engineering executive, and re-review tentative decisions for the entire organization. In practice, this is too many folks to review in detail, so this round typically focuses on promotions, top performers, and bottom performers.
The Engineering executive will review the final decisions with the People team, and then align with other executives to maintain some degree of consistency across organizations. They’ll also review how the proposed ratings and promotion decisions will impact the current company budget.

The above example has three rounds of calibration (sub-organization, organization, executives), and each round will generally take three to five hours from the involved managers. The decisions significantly impact your team’s career, and the process is a major time investment.

The more calibrations that I’ve done, the more I’ve come to believe that outcomes depend significantly on each manager’s comfort level with the process. One way to reduce the impact of managers on their team’s ratings is to run calibration practice sessions for new managers and newly joined managers, to give them a trial run at the process before their performance dictates their team’s performance outcomes.

Another way is for you, as the functional executive, to have a strong point of view on good calibration hygiene. You will encounter managers who filibuster disagreement about their team, and you must push through that filibuster to get to the correct decisions despite their resistance. You will also find managers who are simply terrible at presenting their team’s work in calibration meetings, and you should try to limit the impact on their team’s ratings. In either case, your biggest contribution in any given calibration cycle is giving feedback to your managers to prepare them to do a better job in the subsequent cycle.

While most companies rely on the same group to calibrate performance ratings and decide on promotions, some companies rely on a separate promotion committee for the later decision, particularly for senior roles. The advantage of this practice is that you can bring folks with the most context into the decision, such that Staff-plus engineers can arbitrate promotions to Staff-plus levels, rather than relying exclusively on managers to do so. The downside is that it is a heavier process, and often generates a gap between feedback delivered by the individual’s manager and the decision rendered by the promotion committee, which can make the process feel arbitrary.

Demotions

The flipside of promotions are demotions, often referred to via the somewhat opaque euphemism, “down leveling.” Companies generally avoid talking about this concept, and will rarely acknowledge its existence in any formal documentation, but it is a real thing that does indeed happen.

There are three variants to consider:

Demotion with compensation adjustment: for example, your level goes from Senior Engineering Manager (L6) to Engineering Manager (L5), and your compensation is adjusted to be appropriate for an Engineering Manager (L5). Equity grants are, of course, particularly messy to adjust in this way.
Demotions without compensation adjustment: as above, your level goes from Senior Engineering Manager (L6) to Engineering Manager (L5), but your compensation is not adjusted down to match the new level. This is good for you, but in most compensation systems you will exceed (or be close to exceeding) the pay band for the previous level, which means you will see very limited adjustments going forward.
Title demotion without level adjustment: your title goes from a Senior Engineering Manager to Engineering Manager (L6), while maintaining the same level (L6). This means compensation will keep treating you in the same way, but organizationally you’ll be treated as a member of the lower level, e.g. not publicly considered a Senior Engineering Manager, not included in forums for Senior Engineering Managers, and so on.

All of these approaches are a mix of fair or unfair, and come with heavy or light bureaucratic aftereffects to deal with going forward. These bureaucratic challenges are why most companies try to avoid demotions entirely. Further, the concept of “constructive dismissal” means that demotions need the same degree of documentation as dismissals. It’s certainly not a time saving approach.

I avoided demotions entirely for a long time, but I have found demotions to be effective in some cases. First, there are scenarios where you mis-level a new hire. They might come in as a Staff Engineer (L6), but operate as a Senior Engineer (L5). In that scenario, your options are either to undermine your leveling for everyone by retaining an underperforming Staff Engineer–which will make every promotion discussion more challenging going forward–or to adjust their level down. I’ve done relatively few demotions, but few is not zero. I have demoted folks in my organizations, as well as those I directly managed, and the outcomes were better than I expected in every case where outright dismissal felt like the wrong solution.

Floor for Feedback

When you’re designing processes, I think it’s helpful to think about whether you’re trying to raise the floor of expected outcomes (“worst case, you get decent feedback once a year”) or trying to raise the ceiling (“best case, you get life changing feedback”). Very few processes successfully do both, and both performance processes focus on raising the floor of delivered feedback. This is highlighted by the awkward, but frequent, advice that feedback in a performance process should never be a surprise.

Because performance processes usually optimize for everyone receiving some feedback, it’s unwise to rely on them as the mechanism to give feedback to your team. Instead, you should give feedback in real time, on an ongoing basis, without relying much on the performance process to help. If you’re giving good feedback, it simply won’t help much.

This is particularly true as your team gets more senior. If senior folks are getting performance feedback during the performance process, then something is going very wrong. They should be getting it much more frequently.

Managing other functions

One of the trickiest aspects of performance management is when you end up managing a function that you’ve never personally worked in. You may be well calibrated on managing software engineer’s performance, but feel entirely unprepared to grade Data Scientists or Quality Assurance Engineers. That’s tricky when you end up managing all three.

What I’ve found effective:

Leave behind your functional biases (e.g. “QA is easy”) that you may have developed earlier in your career.
Don’t be afraid to lead, even if you don’t know the area well. You are the functional area’s executive, and if you don’t push on performance within the function, no one else will.
Learn the area’s fundamentals: watch them in their workflows, read the foundational texts, attend the tech talks, speak to domain experts in and outside of your company, and so on.
Find high judgment individuals internally to lean on, validate ideas with, and consult for input. Be careful how you pick those individuals, as it can go wrong if you lean on individuals that the team doesn’t respect.
Prioritize hiring a functional leader who can operate as the area’s quasi-executive. Ultimately, you will never have enough time to become an expert in each area you work in, and that problem will only compound as you move into more senior roles at larger companies.

This certainly is tricky, but don’t convince yourself that it can’t be done. Most executives in moderately large companies are responsible for functions that they never worked in directly.

Compensation

As an Engineering executive, you will generally be the consumer of a compensation process designed by your People team. In that case, your interaction may come down to reviewing the proposed changes, inspecting for odd decisions, collecting feedback from senior managers about the proposals for their team, and making spot changes to account for atypical circumstances.

That said, I have found it useful to have a bit more context on how these systems typically work, and I will walk through some of the key aspects of how these processes typically work:

Companies typically build compensation bands by looking at aggregated data acquired from compensation benchmarking companies. Many providers of this data rely on companies submitting their data, and try to build a reliable dataset despite each company relying on their own inconsistent leveling rubrics. You’ll often be pushed to accept compensation data as objective truth, but recognize that the underlying dataset is far from perfect, which means compensation decisions based on that dataset will be imperfect as well.
Compensation benchmarking is always done against a self-defined peer group. For example, you might say you’re looking to benchmark against Series A companies headquartered in Silicon Valley. Or Series B companies headquartered outside of “Tier 1 markets” (“Tier 1” being, of course, also an ambiguous term). You can accomplish most compensation goals by shifting your peer group: if you want higher compensation, pick a more competitive peer group, if you want lower compensation, do the opposite. Picking peers is more an art than a science, but it’s another detail to pay attention to if you’re getting numbers that feel odd.
Once you have benchmarks, you’ll generally discuss compensation using the compa ratio, which expands to “comparative ratio.” Someone whose salary is 90% of the benchmark for their level has a 0.9 compa ratio, and someone who has 110% of the benchmark for their level has a 1.1 compa ratio.

Each company will have a number of compensation policies described using compa ratios. For example, most companies have a target compa ratio for new hires of approximately 0.95 compa, and aim for newly promoted individuals to reach approximately 0.9 compa at their new level after their promotion. Another common example is for companies to have a maximum compensation of 1.1 compa ratio for a given level: after reaching that ratio, your compensation would only increase as the market shifts the bands or if you were promoted.
Every company has a geographical adjustment component of their compensation bands. A simple, somewhat common, implementation in the United States is to have three tiers of regions–Tier 1, Tier 2 and Tier 3–with Tier 2 taking a 10% compensation reduction, and Tier 3 taking a 20% reduction. Tier 1 might be Silicon Valley and New York, Tier 2 might be Seattle and Boston, and Tier 3 might be everywhere else. Of course, some companies go far, far deeper into both of these topics as well, but structurally it will be something along these lines.

Whatever the compensation system determines as the correct outcome, that output will have to be checked against the actual company budget. If the two numbers don’t align, then it’s almost always the compensation system that adjusts to meet the budget. Keep this in mind as you get deep into optimizing compensation results: no amount of tweaking will matter if the budget isn’t there to support it.

Whatever the actual numbers end up being, remember that framing the numbers matters at least as much as the numbers themselves. A team that is used to 5-7% year over year increases will be very upset by a 3% increase, even if the market data shows that compensation bands went down that year. If you explain the details behind how numbers are calculated, you can give your team a framework to understand the numbers, which will help them come to terms with any surprises that you have to deliver.

How often should you run cycles?

Everyone has strong opinions about the frequency of their company’s performance cycles. If you run once a year, folks will be frustrated that a new hire joining just after the cycle might not get any formal feedback for their first year. If you run every quarter, the team will be upset about spending so much time on the process, even if the process is lightweight. This universal angst is liberating, because it means there’s no choice that will make folks happy, so you can do what you think will be most effective.

For most companies, I recommend a twice annual process. Some companies do performance twice a year, but only do promotions and compensation once a year, which reduces the overall time to orchestrate the process. There’s little evidence that doing more frequent performance reviews is worthwhile.

The only place I’ll take a particularly firm stand is against processes that anchor on each employee’s start date and subsequent anniversaries. For example, each employee gets a performance review on their anniversary of joining the company. This sort of process is very messy to orchestrate, makes it difficult to make process changes, and prevents inspecting an organization’s overall distributions of ratings, promotions or compensation. It’s an aesthetically pleasing process design, but it simply doesn’t work.

Avoid pursuing perfection

In The Engineering executive’s role in hiring, my advice is to pursue an effective rather than perfect hiring process, and that advice applies here as well. There is always another step to improve your performance or compensation process’ completeness, but good processes keep in mind the cost of implementing each additional step. Many companies with twenty employees provide too little feedback, but almost all companies with 1,000 employees spend most of their time on artifacts of performance that could be devoted instead to giving better feedback or on the business’ underlying work itself rather than meta-commentary about that work.

As an executive, you are likely the only person positioned to make the tradeoff between useful and perfect, and I encourage you to take this obligation seriously. If you abscond this responsibility, you will incrementally turn middle-management into a bureaucratic paper-pushing function rather than a vibrant hub that translates corporate strategy into effective tactics. Each incremental change may be small enough, but in aggregate they’ll have a significant impact.

If you want to get a quick check, just ask your team–particularly the manager of managers–how they feel about the current process, and you’ll get a sense of whether the process is serving them effectively. If they all describe it as slow and painful, especially those who’ve seen processes at multiple companies, then it’s worth considering if you’ve landed in the wrong place.

Summary

This post has covered the core challenges you’ll encounter when operating and evolving the performance and compensation processes for your Engineering organization. With this background, you’ll be ready to resolve the first batch of challenges you’re likely to encounter, but remember that these are extremely deep topics, with much disagreement, and many best practices of a decade ago are considered bad practice today.

The Engineering executive’s role in hiring.

Sun, 27 Aug 2023 07:00:00 -0600

Everyone in an engineering organization contributes to the hiring process. As an engineer, you may have taken pride in being an effective interviewer. As an engineering manager, you may have prioritized becoming a strong closer, convincing candidates to join your team. As a more senior manager, you will have likely shifted focus to training others and spending time with candidates for particularly senior roles.

As an engineering executive, your role in the hiring process will shift once again. You’ll continue to make some key leadership hires yourself, but you’ll spend more and more time designing and debugging your overall interview process.

In this post, we’ll cover:

Establishing your overall hiring process, including job descriptions, rubrics, hiring loops, and so on
How many executives focus so much on perfect hiring processes that their processes fail
Your role in monitoring and debugging the hiring process
Helping close key candidates across your organization
Options for leveling candidates appropriately
How managing headcount is a key part of managing hiring
How to train your hiring managers to avoid challenges like pursuing non-existent unicorn candidates
Calibrating hiring inside and outside of your network, along with considering company-internal candidates
Evaluating building an engineering brand for your company
Deciding whether to introduce a hiring committee process
Remembering that the system serves you, not the opposite

After reading through, you’ll have a clear plan for structuring your overall hiring process, as well as your specific role within that new process. You’ve spent much of your career serving a hiring process, and now you need to create a system that serves you.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Establish a hiring process

Unless you’re joining an extremely early-stage company, the engineering organization will already have some sort of hiring process in place. Unless there’s widespread agreement that the current process isn’t working, you should participate in the existing process to get a feel for how it works.

It’s almost always the case that you can adapt the existing process to accomplish your goals rather than starting from scratch, and incorporating what already exists will both simplify retraining the team on the new process and build good-will with the folks who built the previous process.

Regardless of where you start, your final process should include every one of these components:

Applicant Tracking Systems (ATS): a good ATS is the core mechanism for coordinating your interviewing process. Although many early companies try, running an effective hiring process without an ATS is time intensive with limited return: don’t try it. There are enough reasonable options out there that I won’t recommend any one in particular.
Interview loop documentation: every role should have a documented interview loop that covers the interviews, the trained interviewers for each interview, and links to each interviews’ definition and rubric..
Leveling framework: articulate how you level candidates based on their interview performance and prior experience. In particular, describe when you level candidates in your process.
Interview definition and rubric: define an explicit problem or set of questions to ask for each interview. Then add an explicit rubric for evaluating that candidate’s answers. My experience is that it’s preferable to be very consistent on which questions to ask. For example, using the same programming problem for all candidates’ programming interviews.

A frequent pushback is that candidates will cheat by learning the problem from previous candidates, which is certainly possible. However, I’ve found the risk of cheating is still lower than the risk of poor signal due to solving inconsistent problems. (Furthermore, it’s usually pretty clear which candidates are cheating. Make sure you have additional sections for them to complete if they go fast, and note if their ability to solve those sections degrades in a surprising way.)
Hiring role definitions: every interviewer, hiring manager, and recruiter will engage with your hiring process using assumptions built on the prior processes they’ve worked in. This will often lead to disagreement between hiring managers and recruiters about who’s responsible for the closing process, who has input on the offer’s compensation details, and so on. The best way to avoid this is being very explicit about who is responsible for what.
Job description template: you should create a baseline template for job descriptions, with a consistent structure and background on your organization, benefits, and mission.
Job description library: hiring managers should use the job description template to write a job description for each role they hire. These job descriptions should be aggregated in a shared folder where they can be reused rather than reinvented. This also simplifies keeping descriptions updated as you refine shared components.
Hiring manager and interviewer training: finally, the last component of an effective hiring process is a clear mechanism for training your interviewers. The most effective process I have seen is having new interviewers shadow several trained interviewers, combined with one reverse-shadow interview where an experienced interviewer shadows (and gives feedback to) the new interviewer.

There are certainly other approaches to consider, including training materials or classes, but I’ve found that many interviewers simply don’t listen in those training, whereas shadowing and reverse-shadowing is much harder to fake your participation.

If you’re joining a relatively scaled engineering organization, it’s likely that most of these will already exist, and that you can quickly formalize the few undocumented portions. On the other hand, if you’re joining a smaller organization, it’s quite possible that you’ll start from a place where none of these materials exist. In the latter case, I’d aim to introduce one or two components at a time over the course of a year: going too fast will overwhelm the team, but isolating changes will lead to retraining fatigue from the team as their hiring process changes repeatedly.

Pursue effective rather than perfect

The two biggest errors that executives make in designing their hiring processes are not designing a process at all—hopefully addressed by the preceding section—and designing overly heavy processes that make it ineffective to hire. The latter is particularly challenging to notice, because you’ll often believe you are optimizing the process when you’re actually slowing it down.

The three clearest indications that you’ve over-optimized your hiring process are:

Recruiters hire fewer than five engineers per recruiter per quarter (excluding the scenario where they are constrained by headcount)
You frequently need new interview loops or new interview questions
It routinely takes more than two weeks from a candidate’s first interview to you making an offer to that candidate

Each of these indicate a process that’s consuming a lot of energy without generating much impact. Often the cause is an indecisive executive who adds steps to find clearer signals, which generally obscures reality rather than clarifies it. I’ve also seen this caused by well-meaning, structured thinkers who are trying to replace biased human judgment with more structured approaches. Neither of these are inherently bad ideas, and it’s through inspecting the above indicators that you can check whether you’re really improving your process or if it just feels like progress.

As the responsible executive, I recommend you require a high bar for each extension to your hiring process. Even if individually they make a great deal of sense, in aggregate they will quickly make the process too cumbersome to operate. Each specialized interview loop, each additional interview to design and train the team on, each approval step, each movement from an accountable individual to a committee–all of these will improve quality, but often in a way that leads to worse outcomes as the process grows heavy. If the current process works, even if it’s not ideal, push the team to work with it rather than extend it. You should certainly modify the process when it’s wholly broken, or when you can improve the standard path for everyone, but stay wary of specializations, customization, and the bespoke.

Monitoring hiring progress and problems

Once you’ve built the hiring process, your job as an executive is generally to monitor and debug it, rather than serve within it. This is particularly true after Engineering grows past 100 members, at which point you’ll be directly involved in the process for a small fraction of the senior-most hires.

Here are the mechanisms I’ve found effective in monitoring and debugging hiring, in the order that I’d consider rolling them out if I joined a company without much oversight:

Include Recruiting in your weekly team meeting: your Engineering leadership team should have a weekly team meeting, and I strongly encourage including a tech recruiter in that meeting. Their presence makes it possible to troubleshoot recruiting topics quickly and transparently. Some topics may not be particularly interesting to the recruiters, but that’s true for some members of most standing working meeting.

In particular, this is by far the easiest place to change hiring priorities without anyone feeling left out of the loop.
Hiring review meeting: meet once a month with the Engineering recruiting lead, and talk through their priorities, as well as any problems that have come up. Keeping this meeting small, typically just the two of you, means you can troubleshoot difficult issues that may be hard to discuss in your team meeting.
Visibility into hiring approval: although you should likely not be approving every hire in your organization, it’s extremely valuable to have a single place where you can see all the approvals. Often this is a private chat with Engineering’s hiring managers and recruiters, where each offer is approved.
Out-of-band compensation approval: this is discussed more below, but similarly to seeing all hiring approval, it’s even more helpful to be an approver on all atypical candidate offers. This gives you visibility into the places where your standard operating process isn’t working for some reason.
Monthly hiring statistics: have Recruiting report on hiring statistics for each role they’re currently hiring. It’s particularly helpful to understand throughput (hires per recruiter), time to hire, offer rate, and acceptance rate. Those four metrics, cohorted by each role, should be enough for you to identify the next set of questions to dig into.

There are, of course, always more meetings and tools that you can introduce. I’d recommend starting with a couple and going from there. As you’ve probably picked up by now, my experience is generally that you can go faster by making incremental changes than by introducing massive changes, even when your goal is transformation.

Helping close key candidates

Sometimes executives insert themselves as a final interview in the hiring process, even after their organization becomes quite large. Executives who do this tend to swear by it as an essential step, where only they can ensure the quality bar for hires remains exceptionally high. However, it tends to significantly slow down the hiring process, and even executives who believe in the most strongly will eventually scale back on this practice.

However, while it’s unscalable to remain as an interviewer across all loops, it is particularly valuable to remain engaged in helping close senior candidates. As an executive, you should be able to tell Engineering’s story and how it contributes to the larger company story, and why that makes for interesting work. You’re also best placed to address strategic concerns the candidate raises.

The approach I’ve found helpful here is three-fold:

Letting the recruiting team know that you’re available for sell calls with good candidates.
Updating the Staff-plus and Engineering Manager hiring loops to offer you as a sell call by default for candidates who ask for it. Many candidates won’t, but some will, and these are the candidates who will shape your company the most quickly after they’re hired.
Inserting yourself into the selling process for particularly high importance roles. Even if you don’t say something new, often the same message from an executive will carry more weight, and your involvement is a clear signal to the candidate that they’re considered an important hire.

I’ve only found this counterproductive in two scenarios. First, some executives add a sell call as a mandatory part of the hiring process, which often creates more friction than it’s worth. There are candidates who are excited to accept without meeting you, and for them the additional sell call will slow things down, and executives are particularly painful to schedule. Second, there are some executives who are exceptionally bad at selling their organization. A friend once did a sell call with an executive who turned out to be watching online videos during the call, which unsurprisingly did not make them feel like a valued candidate.

Leveling candidates

Within the hiring process, the two most contentious topics tend to be determining compensation details for each candidate, and determining the candidate’s level. You can’t determine appropriate compensation for a candidate without knowing their level, so we’ll start there.

The first question to answer is when you level candidates in your process. The obvious answer is that you level candidates after seeing their interview performance, but there are a few issues with that. Most importantly, you likely want to conduct a different process to evaluate very senior candidates than to evaluate early career candidates. At a minimum, you’d want the interviewers to be different, as it’s relatively rare for a panel of mid-level interviewers to decide a candidate is Staff-plus, and you likely wouldn’t be confident in their evaluation even if they did.

Generally I recommend provisionally leveling candidates before they start the bulk of your interview process. For example, you might provisionally level them after they complete the technical phone screen, allowing you to customize the remainder of their process for that provisional level. You can then finalize the leveling decision as part of deciding whether to make an offer. I recommend relying on a simple provisional leveling heuristic such as a combination of technical phone screen performance and years of prior experience. This is far from perfect, but there’s generally enough signal there to determine the range of plausible levels.

The final leveling decision should be guided by a written leveling framework, which looks at the candidate’s holistic interview performance to determine a level. Part of that framework is handling disagreement around leveling, which is particularly common. The most common approach is that:

Final leveling decision is made by the hiring manager then escalated for approval.
Approval is done by the hiring manager’s manager for levels at or below your career level (also known as “terminal level” at some companies, this is Senior Software Engineer at many companies), and approval for more senior levels is done by the Engineering executive.

Some companies, particularly larger ones, rely on a committee rather than individual hiring managers for these decisions. My experience is that committees appear less biased, but generally introduce a bias of their own, and are less efficient than wholly accountable individuals. The counter-balance is that at a certain scale, it’s simply impossible to centralize these decisions without significantly slowing down your hiring process. I recommend introducing committees only after relying on individuals has proven too slow at your rate of hiring.

Determining compensation details

Compensation is a broad topic, which I’ll write about more in my next post, but a quick overview on determining compensation details for your offers. There are two particularly important questions which should be detailed in your hiring role definitions: who calculates the initial offer, and what are the approval steps once an offer has been calculated?

The approach that I’ve found effective is:

Recruiter calculates the offer and shares it into a private chat channel with the hiring manager.
Offer approvers are added to the channel to approve decision to make an offer, the candidate’s leveling, and the offer details.
Offers following standard guidelines, e.g. at or below 1.0 compa-ratio, require approval from hiring manager’s manager, and those outside the guidelines require approval from both the hiring manager’s manager and the Engineering executive.
Any escalations or modifications to the offer occur within the same private chat, to ensure all relevant parties are aware.

A centralized approach where recruiters follow a structured process to calculate offers has many benefits. First, it facilitates training on the shared process, and retraining on that process as your compensation bands adjust, you experiment with offer strategy and so on. Second, it avoids less effective hiring managers leaning on compensation, such that those hired by worse hiring managers get outsized compensation packages. (Which is surprisingly common, although of course it’s almost always framed as the weak hiring manager pursuing exceptional candidates.) Finally, you can still design a process to break bands if you want to, but with a centralized mechanism to make it easier to both manage costs and drive consistency.

Some managers argue that this approach doesn’t give them enough flexibility to make compelling offers to the best candidates. That’s true, but I’ve consistently found that there’s always another way to close a candidate other than more compensation. Further, outsized compensation packages will always create ongoing problems in your annual compensation process, which will be designed to normalize compensation across individuals with similar performance ratings at a similar level. Your broader perspective as the Engineering executive is necessary to balance these incentives, whereas an individual hiring manager is almost always incentivized to hire even if it creates a long-term mess for the wider organization.

Managing hiring prioritization

The intersection of headcount planning and hiring is discussed in How to plan as an engineering executive, but is worth mentioning here as well. In practice there are two fundamental modes of prioritizing hires:

In rapidly growing companies, there is so much headcount that you’ll be constrained by recruiter assignment rather than headcount.
At slower growing companies, headcount is the more likely constraint.

In both cases, you’ll frequently have teams pushing for higher priority for their roles. I’m a believer in forcing leaders to solve within their constraints rather than frequently shifting those constraints, but my preference is just one of many ways to approach these tradeoffs. The most important thing to highlight is that both recruiter assignment and headcount are global constraints that you must control as the Engineering executive.

This control can either be something you do personally, or something you delegate to one individual to do on behalf of the wider organization, but they must be made centrally. Making these decisions centrally doesn’t mean that you have to spend a lot of time on it. The simplest way to sidestep this is to determine the headcount and recruiters for each Engineering sub-organization (roughly, each area corresponding to one of your direct reports) and then allow those sub-organization to optimize within their boundaries and allocations.

The biggest trap to avoid is prioritizing recruiters based on hiring need will often steer all recruiting capacity towards your least effective hiring managers. My learned belief is that slow hiring is almost always an execution issue by the hiring manager or the recruiter, and only infrequently the consequence of limited staffing. The exception is when you’ve opened too many concurrent roles for your current recruiter staffing, which is easy to diagnose by looking at the ratio of recruiters to roles (if you have more than three open roles per recruiter, something is very likely going wrong). If you really want to help, first consider spending time training the individuals involved rather than shifting headcount or recruiter staffing.

Training hiring managers

Earlier, I mentioned shadowing and reverse-shadowing as an effective mechanism to train interviewers. That is a crucial part of an effective hiring process, but there’s a second component of training that’s often ignored: training your hiring managers.

There’s a handful of particularly common hiring problems that are usually due to untrained or inexperienced hiring managers:

Demanding unrealistic, unicorn, candidates from the recruiting team, such that you never make offers
Allowing any concern from any interviewer to block a candidate from getting an offer, such that you never make offers
Pushing for non-standard compensation on every candidate rather than learning to close candidates within the standard compensation bands
Being indecisive on candidates, asking for additional interviews until the candidate withdraws from the process
Refusing to talk to candidates early in the process and then blaming recruiting team that senior candidates aren’t interested in finishing the process

If you identify one of these, then I do recommend running focused trainings for your hiring managers on the specific topic that’s coming up. These are all topics that I’ve devoted a session of my Engineering Managers Monthly meeting to, talking through examples of why it’s problematic, why it’s not a sign of strong hiring, explaining what reasonable pass-through rates look like for a healthy hiring loop, and recommending strategies for overcoming the issue.

Once you’ve done a training session, you and the recruiting team should point out the issue to individuals who are running into it, and hold them accountable for fixing it. Folks making these mistakes will often have conviction that they’re doing the right thing, but don’t get swayed by their conviction. Effective hiring processes hire candidates. Hiring managers are accountable for their hiring process. Any argument suggesting one of these is false is a flawed argument.

Hiring internally and within your network

When I worked at Yahoo!, our team needed another engineering manager. We didn’t run a hiring process, or even do interviews. Instead, our Director brought on a colleague he’d worked with before. That new manager soon decided he needed a tech lead on his team. We didn’t run a hiring process, do interviews, or consider candidates on the existing team. Instead, our new manager brought over one of his previous colleagues. A third previous colleague reached out to our Director, and without a single interview we’d soon hired a new Chief Architect who would ultimately never write or read a technical specification about our product, nor contribute a single line of code.

One of my teammates–one who had joined the team through the more traditional route of interviewing–described this pattern as the flying wedge, and it’s emblematic of the worst sort of network-hiring. Hiring exclusively from your network will convince your existing team that they and their networks aren’t wanted for important roles at your company.

A similar, somewhat common, scenario is one where your company exclusively fills important roles with external hires. Each individual external hire may make sense, but in aggregate the pattern of prioritizing external hires will encourage your team to seek career advancement elsewhere, draining your organization of context and continuity.

When it comes to internally or externally hiring and hiring within or without your network, the ideal path is always moderation. Hire some internally, some externally, some within and some without. Too much of any path will either isolate your culture from valuable opportunities to evolve, consolidate it onto the culture that worked at a former employer, or prevent it from coalescing to begin with.

In my experience, almost everyone agrees with the above statement, but quite a few don’t follow its advice. As I’ve dug into that, it’s generally because of a missing hiring skill:

Inability to hire outside their network (or within it)
Inability to fairly evaluate internal candidate (or external candidate)

Periodically look at the number of internal versus external hires for senior roles within your organization, and dig into areas where there are exclusively hires of one sort. If you find a lopsided pocket of your organization, talk with the relevant leader and push them to make one hire of the sort they’re currently ignoring. Even one will force them to acknowledge the skill gap, and start the process of fixing the imbalance.

If the person with a significant imbalance is you, then take it seriously! Don’t hide from it by justifying the unbalance with philosophical or intellectual rationales, and instead push yourself to make one hire of the other sort. Particularly for new executives, I often find there’s an underlying belief that they cannot close strong external candidates, and disproving that belief is an important part of your personal growth.

Building an Engineering brand

The details of building an engineering brand are discussed in Building personal and organizational prestige, which I’ll avoid repeating here in full. Instead, I’ll briefly repeat its conclusion regarding building Engineering brands in particular:

Some companies do a tremendous amount of investment into their Engineering brand, those companies will generally tell you that their branding efforts are a fundamental part of their success
However, most companies, including some very successful ones, do very little Engineering branding, and rarely find themselves stymied by its absence. This doesn’t mean you shouldn’t do any, but rather than you shouldn’t view it as a fundamental necessity
If you choose to invest into Engineering branding, a small investment can capture most of the upside. The biggest exception is for companies who sell to engineers in addition to hiring them, where you may prioritize a larger brand effort from a lead generation perspective

In general, if you’re already finding enough top-of-funnel candidates for your hiring process, don’t spend more time here unless you can connect that time to another business objective, or have internal folks who find this work energizing enough to take it on as a side project.

Should you introduce a hiring committee?

Many companies introduce centralized (Engineering-scoped) or semi-centralized (Product Engineering or Infrastructure Engineering-scoped) hiring committees as part of maintaining a consistent hiring process. I’ve seen this happen frequently enough in Silicon Valley companies that some executives have come to believe that hiring committees are a natural or ideal landing spot.

Hiring committees are a useful tool, but I’d caution against introducing them as the obvious solution. They’re useful, but come with their own problems.

I generally dislike committees as they introduce ambiguity in who should be held accountable for outcomes. In this case, they also mean that hiring decisions are made further from the particular team, which often degrades individual decisions. These committees are also vulnerable to misaligned members. I was once in a hiring committee where a new member joined who relied very heavily on the universities that candidates attended, even when we clarified to the member that we didn’t hire that way, they refused to change and our Engineering executive was unwilling to hold them accountable to our hiring practices.

On the positive side, they are also a great mechanism for training hiring managers’ judgment on what makes a good candidate. They also introduce more consistent hiring practices across an inconsistent organization, solving a similar problem as Amazon’s Bar Raiser program. Committees are slower than a responsive hiring manager, but faster than a disengaged or very busy hiring manager.

Remember that the system exists to support you

If you came up as a rules-minded leader, you can almost certainly think of examples where your executive responsible for designing the hiring process also ignored that process to accomplish an immediate goal. Personally, I was most annoyed by executives who steamrolled the process to hire former colleagues who performed poorly in our interview process. Each time I’d complain to colleagues, “Why did we build this comprehensive hiring process if we don’t even trust its decisions?”

As is often the case, as I switched into the role of the executive responsible for Engineering hiring, I began to appreciate why perfectly following the process was difficult. When I vowed to loyally follow the hiring bands, sometimes I’d find peer executives paying far outside the bands, implicitly penalizing hires in Engineering. When I endeavored to respect each negative hiring review, sometimes I’d encounter interviewers who refused to use the stated rubrics. When I hired for a brand new role, I’d sometimes find interviewers who interpreted the role’s requirements very differently, even if I pulled together materials explaining the role.

Each of those challenges can be solved over time with better training, but as an executive you rarely control the timeline you’re working in. Sometimes your problem is urgent today. In those scenarios, the question to answer is sometimes whether the company will be better off if you solve the underlying problem (e.g. missing a leader for a key role) or if you respect the process (e.g. don’t break the rules you created). You should try to solve your problem within the process you’ve designed, but don’t get so blinded by your process that you think the process is always more important than your problem at hand. Sometimes the process is clearly less important than the current problem.

That doesn’t mean you should always ignore your process. If your interview process indicates a candidate has gaps, there is usually a valuable signal when our hiring processes decline a candidate. Even if we’re confident the negative signals are wrong, it still undermines a hired individual when their new colleagues know they performed poorly in the hiring process but were nonetheless hired. There is a cost to defying your process, just as there is a cost to following it, and as an executive you need to make that tradeoff deliberately.

Summary

This post has covered your role as an executive in your organization’s hiring, the components you need to build for an effective hiring process, and provided concrete recommendations for navigating the many challenges that you’re likely to run into while operating the hiring process. There are an infinite number of questions to dig into, but this coverage will give you enough to get started, build a system that supports your goals, and start evolving it into something exceptionally useful.

Manage your priorities and energy.

Thu, 03 Aug 2023 06:00:00 -0600

Back when I was managing at Uber, I latched onto a thinking tool that I drilled into the teams I worked with: reach the right outcomes by prioritizing the company first, your team second, and yourself third. This “company, team, self” framework proved a helpful decision-making tool, and at the time I felt it almost always led to the correct decision. It also helped me articulate why I disagreed with some of my peers’ decisions, which violated this hierarchy by placing individual or team preferences over the company’s priorities.

As I’ve become a more experienced manager, I’ve stopped giving this advice. I still believe it’s conceptually good advice, and I continue to see managers who fail because they are missing this perspective. However, I’ve also seen some of the best leaders that I’ve worked with burn out by following this advice too loyally. A long-term career depends equally on being impactful and staying engaged.

In this post we’ll discuss:

How I previously used the “company, team, self” framework to prioritize
How energy management is positive-sum, and why that’s changed my model for prioritization
How I’ve moved to an “eventual quid pro quo” framework
Distinguishing between self-interested behavior and the appearance of self-interest
Why it’s important to remain flexible, even with the best framework

Let’s drill into the details.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer, and expands on ideas I earlier wrote about in Company, Team, Self.

“Company, Team, Self” framework

As I mentioned above, I used to rely on the “company, team, self” framework to check my decision making, particularly during my time at Uber. Uber Engineering had a strong Not Invented Here bias, and frequently invested in creating its own technology like the M3 metrics system or the Mezzanine sharded storage system. While many of these systems addressed very real scalability concerns, they also tended to get their proponents promoted, and over time it became challenging to distinguish whether any particular platform was principally anchored in business impact or career progression. In that environment, pushing the team to explicitly prioritize the company and team first was extremely valuable.

Some of the concrete scenarios where I found this framework useful were:

Deciding whether to prioritize an interim orchestration solution before a planned Mesos-based orchestration system was rolled out. Yes, we should build an interim automated solution to support Uber’s engineers while we waited for the replacement, even thought it was viewed as less promotable work than working on the delayed replacement
Fixing hiring loops where stressed out hiring managers wanted to hire candidates who didn’t meet our hiring bar. Yes, you do need to make a hire, but no, making this hire isn’t the right outcome for the company
Evaluating the relative priority of staffing requests within the organization. Yes, every team would benefit from additional staffing, but which would be most valuable to the wider organization?

The framework was particularly helpful within platform and infrastructure teams where our distance from end-users and revenue often obscured answers that might have been more obvious on user-facing teams with the loud signal of user feedback.

However, the more I followed the “company, team, self” approach,” the more I appreciated its biggest downside. The most valuable work at a company is rarely the most interesting nor—oddly enough–what the company itself particularly values. I frequently noticed cases where engineers did the best thing for the company, solving an urgent problem with little staffing, but weren’t recognized or promoted for their work. Those engineers would slowly become deenergized and frustrated. As this pattern got clearer, it became increasingly clear to me that always prioritizing the company’s needs was too literal a solution, and a durable approach would require balancing the company’s priorities with remaining personally energized for the long-haul.

Energy management is positive-sum

People are complex, and they get energy in complex ways. Some managers get energy from writing some software. That’s great, particularly if you avoid writing software with production dependencies. Some managers get energy from coaching others. That’s great. Some get energy from doing exploratory work. Others get energy from optimizing existing systems. That’s great, too. Some get energy from speaking at conferences. Great. Some get energy from cleaning up internal wiki’s. You get the idea: that’s great. All these things are great, not because managers should or shouldn’t program/speak at conferences/clean up wiki’s/etc, but because folks will accomplish more if you let them do some energizing work, even if some of that work itself isn’t very important.

Rigid adherence to any prioritization model, even one that’s conceptually correct like mine that prioritized the company and team first, will often lead to the right list of priorities but a team that’s got too little energy to make forward progress. It’s not only reasonable to violate perfectly correct priorities to energize yourself and your team, modestly violating priorities to energize your team in pursuit of a broader goal is an open leadership secret. Leadership is getting to the correct place quickly, it’s not necessarily about walking in the straightest line. Gleefully skipping down a haphazard path is often faster than purposeful trudging down the safest path.

There are, of course, rules to breaking the rules. The most important being that your energizing work needs to avoid creating problems for other teams. If your fun project is prototyping a throwaway service in a new programming language, then hmm, maybe that’s fine. But if you put it into production, then your energizing detour is going to be net negative on energy generation after other teams are pulled in to figure out how to support it.

Correctness and humans mix in complex ways. The most important lesson I’ve learned as I’ve become a better manager is that there is almost always a correct answer, but applying that answer to your specific situation will always be nuanced and messy. Further, the correct answer is often different if you’re taking a short-term or long-term perspective. Every specific decision is nuanced and complex, but you’ll be a better leader if your decision making modestly factors in your team’s energy than if it ignores it.

Eventual Quid Pro Quo

These days, my framework for personal prioritization merges the business-first perspective of “company, team, self” with the belief that becoming deenergized or disengaged is my biggest risk in any particular job. I think of this new framework as “eventual quid pro quo.”

The core framework is:

Generally, prioritize company and team priorities over my own
If I’m getting deenergized, artificially prioritize some energizing work. Increase the quantity until equilibrium is restored
If the long-term balance between energy and proper priorities can’t be balanced for more than a year, stop everything else and work on solving this (e.g. change your role or quit)

Arguably, this is a small tweak to my earlier framework, but it’s been very important for me, which is the key component of a successful framework. What makes it useful for me is that I am a very literal interpreter of frameworks, and this gives me space within the framework to maneuver and adapt. I can remain faithful to the framework without draining out my energy in the first few years in any given role.

Just because it’s the right framework for me, doesn’t mean it’ll be the right one for others. To develop an effective prioritization framework requires getting to know yourself (what work energizes you) and your priorities (career, financial, and otherwise). Until you understand those answers, no premade framework is going to produce the correct answers for you.

You’ve probably worked with someone who operated in a short-term quid pro quo mode. This person may have frequently asked what the benefit was to them to take on a given project. That approach does not work well in leadership roles. In particular, it adds friction at the wrong moment, when the requestor (typically your boss) is trying to solve the problem. Over time, short-term demands will often mean you’re considered last for valuable assignments. Decoupling timing of repayment lets you keep access to interesting work while also pushing to ensure you’re rewarded for that work in an appropriate currency. This isn’t perfect, as sometimes you won’t be rewarded for some work, but it’s generally better than the alternative if you are in, or pursuing, an executive role.

Mirrors of misalignment

One challenge with frameworks is they sometimes don’t tell the entire story. As I get further into my career, I’m acutely aware of moments when my actions are perceived as in conflict with some of the earlier mental models I used to evaluate work. For example, I’ve recently spent a fair amount of time threading a needle through Engineering to prioritize Large Language Models through our execution. If I were sitting in many of my previous roles, I would judge myself as prioritizing misaligned work. In my current role, I appreciate that there is an organizational risk budget around how and how much investment we can do to leverage potentially-but-not-necessarily transformative new techniques.

I think of this dynamic–when I prioritize work that my previous frameworks would have considered misaligned–as navigating the mirrors of alignment. There is always another layer of context that validates or invalidates any given piece of work. This is why operating from a place of curiosity is so powerful: folks who appear to be doing something nonsensical almost always have more information than you or are missing a key piece of information you have.

The place where I see these mirrors the most frequently is corporate planning and projects that bypass the corporate planning process. It’s very common for companies in dynamic markets to spend a month planning their next half, then immediately invalidate that plan by changing their priorities. This infuriated me for a long time, and my framework thinking labeled the offenders as bad planners. Now I see myself doing the same thing, and I’ve come to appreciate that many of the folks I silently accused of malpractice were balancing context that I had no idea existed.

As an executive, each time this happens is a reminder to share your context more widely, and remain curious. You’ll rarely be missing strategic context, but you’ll increasingly be missing the tactical context around how things actually work. Seniority creates just as many gaps in knowledge as it closes.

Orthogonal but not in opposition

As you think about expanding your framework to create space for energizing work, the most important guideline to remember is that while it’s okay to do a small amount of work that’s orthogonal to the company’s needs, you will always regret doing work that is opposed to those needs.

Working through some examples:

Many folks do more public speaking, perhaps at conferences during working hours, than their company strictly values. Doing one or two speaking gigs a year may not directly help your company, but it’s also hard to argue that it’s causing any negative effects. (Whereas if you did eight or ten speaking events in a year, then it’s easy to argue that you’re encouraging others in your organization to focus away from the work at hand.)
If you push to use a new technology on a project principally out of interest in learning that technology, then you’re creating significant execution risk and maintenance cost for your company. This might be energizing for you, but in most cases is likely in opposition to the company’s needs.
Doing some angel investing in unrelated business verticals won’t quickly teach you a tremendous amount of relevant information, but it will slowly broaden your horizons across the industry. Further, it won’t take up too much time to do it. (Doing a tremendous amount of angel investing, or running your own syndicate, certainly might cross a line into being opposed to your core work.)

Almost any outside activity is fine when done in moderation. Almost any decision to use work projects as an opportunity to develop yourself by using non-standard tools is a bad choice. If you’re not sure, ask a friend, but it’s probably a bad sign that you’re uncertain.

Remain Flexible

If you strongly disagree with the idea of factoring in engaging work that is imperfectly aligned with your company’s goals, I’d push you to remember that humans aren’t robots. That’s surely obvious advice, but I really struggled with that idea for a long time. It was very frustrating to me that the correct answer to most prioritization and architecture projects was obvious (e.g. why is it controversial that we should use boring technology?), but teams so frequently did something incorrect instead. What folks may not understand, is that for a certain type of person, strictly adhering to the correct path is very energizing. That kind of person, a person who I used to be most of the time (and still revert into today when particularly frustrated), doesn’t need to do sub-optimal energizing work, because doing the correct work is inherently energizing.

However, I have to admit that I’ve never really seen this personality succeed in senior roles, because those roles require working with so many different people, and only a relatively small subset are energized by adherence. I’ve worked with dozens of extremely talented engineers and managers who are professionally stymied by being energized by adherence and demotivated by the lack of adherence in those around them. It is a powerful value system, but it is not a widely appreciated one. It’s particularly rare to find executives who are motivated by adherence. In some brilliant moments you’ll get to work with folks with the same values, and those will be some of your most rewarding moments, but you have to learn to turn those values off when they’re interfering with forward progress. Blunting this edge has been one of my most valuable lessons over the last decade.

Summary

While there are many prioritization frameworks for teams, such as RICE, there aren’t nearly enough to help individuals think through their priorities. Now that you’ve finished this article, you have several examples of frameworks that have (and haven’t) worked for me, and are well-positioned to think of what the right framework might be for you. There’s no one solution, but you’re going to accomplish less in your career if you’re so focused on correctness that you lose track of keeping yourself energized.

Gelling your Engineering leadership team.

Tue, 11 Jul 2023 06:00:00 -0600

One of the first leadership books I read was Patrick Lencioni’s The Five Dysfunctions of a Team, which introduces the concept of your peers being your “first team” rather than your direct reports. This was a powerful idea for me, because it’s much harder to be a good teammate to your peers than to your direct reports. While your incentives are usually aligned with the team you manage, it’s very common for your incentives to be at odds with your peers’ incentives. The Head of Marketing may want you to prioritize work that your team believes won’t be impactful, or the Head of People may want you to be more selective in assigning top performance designations even when you believe your team has earned them. Those are difficult topics to agree on.

Even if it’s easier to align with your direct reports than your peers, it is nonetheless surprisingly difficult to create an engineering leadership team who are aligned with each other. Well-intentioned members will occasionally find themselves at odds with one another, and it’s your job as the engineering executive to establish clear values for the team, and to referee conduct that falls outside those values.

In this article, we’ll discuss:

Debugging the Engineering leadership team after stepping into a new role
Gelling your leadership into an effective team
What to expect from your direct reports in that leadership team
Diagnosing conflict within your team

By the end, you’ll have a framework for forming and operating an effective team.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Debugging and establishing the team

When you start a new executive role, one of your most important tasks is developing your point of view on your functional leadership team. This can be tricky because often by the time you are hired, the previous executive is long gone, and the engineering leadership team has been operating without guidance or feedback for some time. As such, early glimpses of your new leadership team are more likely to reveal them at their worst than their best.

Despite that, you need to answer a few key questions in your first couple months:

Are there members of the team who need to move on immediately? These are folks who either have significant behavioral issues or who have lost the confidence of their team and peers. Rather than underperforming, these are individuals actively causing damage.

For example, if a member is complaining widely about their role and the company, such that they cannot energize the team they lead, it may be time for them to move on.
Are there broken pairwise relationships within your leadership team or between members of your team and their key stakeholders? You’re looking for individuals who clearly cannot work together, or even communicate successfully with one another despite needing to collaborate.

For example, if the product and engineering leaders on a business line never interact directly in meetings, and contradict each other in private, you need to solve that quickly.
Does your current organizational structure bring the right leaders into your leadership team? Your biggest focus should be adding representation for teams like Data, Security, or Infrastructure where their absence prevents important decisions from moving forward quickly, and on elevating important teams from places where they are being neglected or mismanaged.

For example, if your company suffers from a significant lack of a security mindset, and your Security team is nestled deep within Infrastructure where it’s managed by someone without direct Security experience, you may want to pull Security into your leadership team.

As you spend time meeting with folks one-on-one, attending meetings, and watching the team operate, you should quickly form initial opinions on answering each of these questions. Once you have a first opinion, spend time trying to disprove it. You’ve been hired to determine the truth, not coalesce the most popular narrative, and you’ll never regret being suspicious of simple narratives.

Once you’re confident in your interpretation, it’s time to take action. Start by moving out the individuals who won’t be part of the team going forward. Each time you change members of your leadership team, you will have to gel that team afresh, so it’s important to get to a plausible leadership team quickly. This can feel uncomfortable, but letting an untenable situation linger doesn’t help anyone. Making personal changes quickly lets you shift time from dealing with interpersonal conflict to instead working through more durably valuable decisions around execution and strategy.

Sometimes the decision to be made won’t have a clear answer. For individuals who are struggling but you believe have a role to play on your leadership team, have a frank discussion. As a new leader, you have a small window to set clear expectations, which makes it easier to have those difficult discussions. I particularly want to discourage you from waiting to have these discussions because you’re afraid you might lose someone you can’t afford to lose. Most leaders have a strong desire to perform, and establishing clear expectations sends the message that you’re here to help them perform. That’s what good folks want! Anyone who leaves because you set expectations is already too frustrated or burned out for you to help.

Finally, once you’ve edited the members in the room, you have to decide whether to tweak your organizational structure to bring the right folks into the room. It’s not a goal to directly represent every team in your leadership team, but there should be someone who can represent the most important teams with reasonably high granularity. Particularly early on in your tenure, I recommend erring on an overly broad leadership team to ensure a wide set of perspectives are surfaced. You’ll have to balance that against the value of stability. If you’re just not sure, you can always experiment with bringing folks into your meetings on an interim basis.

Should you include engineers in Engineering leadership?

When it’s possible, it’s well worth your time to consider including one or two senior engineers to report to you, as opposed to only managing engineering managers. An engineering executive is responsible for blending the technical and managerial perspectives of engineering, and that’s most easily done when both perspectives are represented within your leadership team.

In some cases, this will be difficult to accomplish. You may have too many direct reports already, and feel it will be overwhelming to include more. In that case, work to establish a group of senior engineers that you interact with frequently to ensure you’re hearing their perspectives. This is frequently an architecture team or the senior-most engineer from each business unit.

Operating your leadership team

Once you’ve decided on your initial leadership team’s members, you have the larger task of turning those individuals into a team. This hinges on creating the values, structure, and relationships your team needs to be effective. Whereas your initial edits on the leadership team will be quick, gelling and operating this team will require ongoing effort throughout the entirety of your role.

Operating your team can be boiled down to four themes:

Define team values. How do you want this team to make decisions? How much should they focus on enforcing consistent policy versus solving escalations? Your team values should connect with your organizational values, but will likely focus more on behavioral tradeoffs in how you do work. The most important values to establish are those that explain navigating conflict within the team, e.g. how should the team navigate the tradeoff between delivering new product functionality and supporting the migration necessary to deprecate an unreliable component?

Some teams will write these team values down, and others will simply live them actively. The important thing is having and enforcing them, whether you’ll benefit from writing them down will depend largely on the sorts of people on the team.
Establish team structure. These are the mechanisms and rituals that the team relies on to operate Engineering. What meetings does this team rely on to broadcast context and resolve conflict? How does information flow from your company’s executive team down to this group? Do folks give a weekly update over chat?
Find space to interact as individuals. Many senior leadership teams become so transactional that they feel impersonal. A transactional collection of individuals can do good work, but a gelled group that knows each other is better positioned to work through the inevitable difficult moments. Your role as the team’s manager is to create some unstructured space to interact as informal, unstructured humans. That unstructured space is locally inefficient–it certainly won’t generate any action items–but relationships are the slack necessary for large groups of humans to work together successfully.
Referee detection from values. Some leadership teams struggle despite having clear values, structure and relationships. Usually this can be traced back to one or two members of the team violating the team’s values without consequence. Once someone defects, others will become skeptical whether following the values is necessary for them either. Your role as the executive is to hold members accountable to the values. Mistakes will certainly happen, but tolerating consistent violations is the same outcome as not having team values at all.

As is often the case in leadership, these steps are fairly simple, and the hard part is consistent implementation. There will be some initial setup here, and there will also be ongoing maintenance every single week–to some extent, every single decision–that you’re serving as an executive. Fortunately, you are not doing this alone, you’ll need the active participation from the team itself.

Expectations of team members

A central part of creating your team’s structure is setting expectations for how members participate with the team. Often individuals on their first leadership team will assume that the team exists to help them do their work, rather than recognize that they lead their organization on behalf of the business’ goals. Setting explicit expectations helps folks recognize if they have an inverted view of priorities.

It’s particularly useful to set expectations on:

Leading their team. Some companies focus internal leaders on bureaucratic execution, lauding pristine headcount spreadsheets and thoroughly documented quarterly plans. Other companies evaluate their internal leaders primarily on dynamically firefighting issues as they emerge. Another set of companies anchor heavily on visionary leadership rather than operational concerns. It’s essential that your leadership team understands however you and your company will judge them.
Communicating with peers. Some leaders are naturally relationship oriented, and will build a wide network within the company. Those folks will generally know what their peers are working on, and what they’re struggling with. Not all leaders act that way though. Another common archetype will be heads down optimizing their team’s operations, mostly unaware of what their peers are doing. If you don’t tell your team what you expect, they’ll default to their natural state, which probably won’t be what you want.
Staying aligned with peers. Being aware of their peers’ goals is a necessary precursor to resolving conflict, but is certainly not sufficient. Your team should understand your expectations around how they resolve tension between each other’s goals. Should they work to resolve the conflicts directly, and only escalate to you when that is unsuccessful? Should they escalate together, or is it more importantly to escalate quickly, even if it’s a bit messy?
Creating their own leadership team. Although the leadership team reporting to the functional executive is particularly important, each member will have their own leadership team as well. It is turtles all the way down. It’s not enough for you to communicate to your team, you need each leader to communicate with theirs, and so on: leaders exist everywhere in high-functioning organizations.
Learning to navigate you, their executive, effectively. One of the enduring controversies in the engineering management community is whether “personal READMEs,” short documents which describe how to work with you, are a sign of professional or self-absorption. Regardless of your opinion there, the reality is that your team will spend a significant amount of energy figuring out how to work with you, and you should do what you can to make it easy for them. Are you a stickler for process or very much the opposite? How much risk should they take? Which stakeholders should they prioritize? Let them know!

Each of these is a useful topic to discuss in your weekly team meeting or one-on-one sessions. Start early, engage with individuals who are finding aspects difficult to adopt, and review these topics periodically as membership of your leadership team shifts. Your team will eventually pick up your expectations even if you don’t state them explicitly, but it’s much faster to just tell them what you want from them.

Competition amongst peers

Sometimes you’ll pull together your leadership team effectively, it works well for a while, but a year later you’ll notice something’s gone wrong: collaboration has slowly faded into internal competition. There are a few different reasons why this can happen, but ultimately it means that members of your team believe they’ll be rewarded more for capturing capacity from each other than creating capacity for the organization overall.

The three most common causes of competition that I’ve seen are:

Perceived lack of opportunity
Misapplying poor habits learned in the context of shrinking or bureaucratic companies
Signal that you are failing to referee your team

The first of these three, a perceived lack of opportunity, has been particularly common in my experience. If your team believes they’ve exhausted their personal runway within your team, then they’ll look for ways to create opportunity for themselves, which often means competing with peers. It’s certainly true that only one person can take on any given stretch assignment.

When you’re managing less experienced folks earlier in your career, it makes sense to solve this problem for them by giving them specific opportunities for growth. For senior leaders, I recommend giving the problem back to them: the next step in their career is being an executive, and certainly no one is going to tell them how to advance their scope at that point. Particularly encourage them to look at opportunities to grow themselves and their career in ways that aren’t zero sum. With some creativity, there’s no reason for your team to compete over access to opportunity.

The next common challenge is team members who’ve learned bad habits in bureaucratic or shrinking companies. Those environments often teach the implicit leadership lesson that it’s more effective to capture existing capacity within the company than to create new capacity for the company. Leaders immersed in that lesson often view success as a zero-sum game. Your goal is to convince them that today’s capacity represents a small fraction of the future capacity. By instead focusing on the capacity that can be collaboratively created in the future, they’ll have significantly more opportunity.

Finally, if neither of those hypotheses fit particularly well, then it’s likely the issue here is you. Teams composed of ambitious individuals typically only follow rules when following those rules is the best path forward. If they’re defecting from your stated rules, then you’re not enforcing them effectively. Get back to the basics that we started with!

Summary

Working with a new team, particularly one that has been trying to lead itself without an engineering executive for some time, is both messy and necessary. This piece covered getting started debugging your new Engineering leadership team and operating that team once it’s gelled. Keep in mind that, even if you do a remarkable job with your team, you’ll be spending time on this from your first day until your last.

Building personal and organizational prestige

Tue, 04 Jul 2023 06:00:00 -0600

Most months I get at least one email from an engineering leader who believes they’d be a candidate for significantly more desirable roles if their personal brand were just better known. Similarly, when funding is readily available during periods of tech industry expansion, many companies believe they are principally constrained by their hiring velocity–if their engineering organization’s brand was just a bit better, they believe they’d be hiring much faster.

Writing a great deal online, and working at companies during periods when they invested heavily in their external engineering brand, I’ve seen the opportunities and limitations of personal and organizational brand building. From those experiences, I’ve come to believe that the value of brand building is overemphasized, but that the impact of enhancing prestige is underrated, and that there’s a relatively straightforward playbook to increase your personal and organizational prestige.

In this post, we’ll work through these topics in building eng org and personal prestige:

The distinctions between building prestige, building brand, and building an audience
Deciding whether it’s valuable to build your personal and engineering brands
The playbook to manufacture prestige with a small quantity of high-quality content
Pitfalls of measuring prestige, and what to measure instead

After reading, you’ll be equipped to decide whether to invest into these areas for yourself or your engineering organization, and a clear path forward.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Engineering brand versus. external communication

Sometimes you’ll hear an argument that personal brand is important for engineering executives because they play an important role in external communication, but in practice that’s fairly uncommon. Few companies lean heavily on their engineering executives for external communication. When this does happen, it’s generally done by a founding CTO, such as Honeycomb’s Charity Majors, who has been very successful in using her considerable reach to create visibility for the company. You’ll rarely see an external CTO hired to take on that sort of role, and there are even fewer examples of individuals succeeding as an external CTO hired into such a role.

After raising their Series A or Series B, most companies bring in specialists to lead external communication roles. If they’re trying to generate more marketing qualified leads, then they’ll likely want Marketing to lead that. If they’re trying to increase product sign ups, then they’ll likely want either Developer Relations or Growth teams to own that.

Engineering brand is a bit different, as it’s generally viewed as a priority because of its impact on Engineering hiring, rather than being a central business function. In such cases, efforts are usually steered from within Engineering itself. Things get a bit messier at companies that sell to developers, and these efforts might fall anywhere within Engineering or Marketing at such companies.

Brand versus prestige

Brand as a deliberately crafted, sustained narrative that is actively known about you. You don’t have to research Google engineering to have an opinion about Google engineering. In your career and as an engineering leader, you will likely be given the advice that it’s very important to build a brand.

If you participate frequently in social media, it’s easy to get sucked into its reality distortion field. When you spend a lot of time in a given online community, being well-known in that community feels equivalent to professional credibility. However, my experience is that few of the most successful folks I know are well-known online, and many of the most successful folks I know don’t create content online at all. Maybe they have an Instagram account, but it focuses on their family and non-professional interests.

Enough folks find this counter-intuitive that I’ll emphasize this theme a bit. The majority of successful executives I’ve worked with don’t write online. They won’t post on Twitter or Mastodon. They haven’t written a book. They don’t speak at conferences. They don’t have a YouTube channel. They don’t stream on Twitch. In your engineering leadership career, you will at times be immersed in the message that you need to be creating content to be successful, but there’s abundant evidence to the contrary. You absolutely don’t have to do this sort of thing.

Similarly, most Engineering organizations spend little time developing their external brand, and are not externally well-known. For every Meta Engineering blog or Netflix Engineering blog, you’ll find hundreds of engineering organizations with limited public visibility around their work. Many of those silent organizations are doing very interesting work, they just don’t spend much time talking about it publicly. You can, without a doubt, be a successful engineering organization without ever doing any external communication to build your brand.

Prestige is the passive-awareness counterpart to brand. Rather than being what someone actively knows about you, it’s what someone can easily discover about you if they look for it. Many interviewers won’t know anything about me, but a few minutes of research will find my writing, conference talks, and work history.

You can build prestige by attending a well-respected university, joining a well-known company, or giving a recorded conference. Companies can build prestige by focusing on a problem that’s immediately attractive to software engineers, finding an attractive way to approach that problem, or retaining prestigious employees. More engineers are interested in working on self-driving cars than on automating personal taxes. If your company does work on automating personal taxes, engineers would certainly be more interested in fully automating that process than streamlining back office processes for a team of accountants.

While many successful engineering leaders and engineering organizations don’t have much of a brand, most are prestigious in one way or another. Prestige is a universal lubricant. It opens the door to taking senior roles and recruiting senior candidates. It creates edges in your network graph that open doors across the industry.

Is building prestige worthwhile for you?

Here are some questions to consider when evaluating whether to invest more time into building prestige:

Are you able to start the interview process for jobs you’re interested in? (Not necessarily receive an offer, but start the process.)
Is an executive recruiter able to match you with interesting roles? (Particularly roles that are more complex or desirable than your current role.)
Are you able to hire senior candidates to work in your organization? (Particularly those with more applicable experience than you.)
Does your team seek you out for career advice and advice beyond the immediate scope of their current work? (Not just your direct reports, but more widely.)
Is your network expanding by default, allowing you to reach out further and to more senior individuals? (Prestige expands its reach by default, as those you already know go into more senior roles.)

If you answer “yes” to most of those questions, then I wouldn’t invest much additional energy here. On the other hand, if you answered no to many of those questions, if you didn’t attend a prestigious university, work at a prestigious company, or select a core business space that software engineers are interested in, then it’s worthwhile to learn how to manufacture prestige.

Manufacture prestige with infrequent, high-quality content

In my experience, engineers confronted with a new problem often leap to creating a system to solve that problem rather than addressing it directly. I’ve found this particularly true when engineers approach a problem domain they don’t yet understand well, including building prestige.

For example, when an organization decides to invest into its engineering brand, the initial plan will often focus on project execution. It’ll include a goal for publishing frequency, ensuring content is representationally accurate across different engineering sub-domains, and how to incentivize participants to contribute. If you follow the project plan carefully, you will technically have built an engineering brand, but my experience is that it’ll be both more work and less effective than a less systematic approach.

Prestige is an ambient, positive familiarity. This doesn’t require an organizational program or a strict content calendar, rather it depends on building awareness of a small amount of noteworthy accomplishments. The most effective approach I’ve seen is doing a small amount of writing or public speaking, and then ensuring that work is discoverable.

The steps to that approach are:

Identify a timeless topic where you have a meaningful perspective, and have an atypical perspective. You’re looking for topics where your writing can remain relevant for decades, and perspectives that demonstrate depth rather than chase controversy. Be mindful that atypical doesn’t mean controversial, it’s usually introducing an additional layer of detail to a low nuance debate.

For a few example topics, consider Productivity in the age of hypergrowth, where I argued that effective onboarding was the key constraint in hypergrowth companies rather than hiring. Or in _Migrations,_where I argued that the value of technical platforms should be foremost evaluated through migration cost rather than the capabilities of the technical platform. Both of these topics remain relevant today, and will hopefully remain relevant for much longer.
Pick a format that feels the most comfortable for you, typically this is either a blog post or a conference talk. The ideal format is something that you’re excited about, you can iterate on the content until it’s ready for release, it’s small enough that you can iterate quickly, and it generates a permanent digital artifact (e.g. a video or piece of writing).

I recommend avoiding books and podcasts in this step. Podcasts are hard to iterate on, as you generally get one take plus whatever you can fix in editing. Books are a difficult format to learn in, as iterating on the content can consume a great deal of time.
Once you’ve picked a format, create the content! Go into this process assuming that you will throw away two or three drafts. Get early feedback, and get that feedback from folks who are experienced in that format: everyone has an opinion on what good content looks like, but not all opinions are equally valuable.

Your content is done when readers can accurately identify your key insight and enjoy it enough to read or listen to its entirety. It’s a good sign if some readers disagree with you: anything interesting will generate some disagreement.
Develop an explicit distribution plan to share your content. The simplest effective distribution plan is coordinating with a few more visible friends to share your article online: ten people committing to share your article online around the same time is a fine distribution plan.
Make it easy for interested parties to discover what you’ve written in the future by collecting them on a personal website and your various presences online (e.g. LinkedIn).
Repeat this process two or three times over the next several years.
You’re done! Many more folks will have ambient, positive awareness of you, and whenever you interview for a role or show up as a hiring manager, it will be easy to discover this previous content that reflects positively upon you. If you enjoyed the process, you can do more, but you don’t need to spend more time on this unless you particularly enjoy it.

This approach works equally well for building your company’s engineering brand as it does for building your personal brand as an executive. In both cases, a small amount of positive, thoughtful content will go further than a larger volume of lesser content, and the short-term distribution benefits of engaging in controversy is at odds with your goal of building prestige.

If this advice feels counterintuitive–if it feels too easy–it’s likely because you’re applying advice for building a brand or an audience to the rather different topic of building prestige. To build a brand that you measure through an audience, consistency is valuable, and volume does matter. Prestige doesn’t need all that, just easily discoverable content that paints a positive connotation.

Does anyone follow this advice?

Viewing myself through this lens, I’ve written hundreds of posts, but probably only about four have generated significant prestige: Migrations, A forty-year career, Sizing engineering teams, and Productivity in the age of hypergrowth. I’d very likely be equally prestigious if I’d simply written those four without the surrounding six hundred.

Going beyond my writing, examples of others who’ve created significant prestige from a narrow slice of their visible work:

Charity Major’s Engineer/Manager Pendulum
Tanya Reilly’s Being Glue
Patrick McKenzie’s Salary Negotiation
Julia Grace’s Scaling yourself during hypergrowth
John Ousterhout’s A Philosophy of Software Design

Certainly some of these folks are written or spoken widely, but they could have accomplished their prestige-related goals without doing so.

Measuring prestige is a minefield

All corporate initiatives demand a metric to measure their outcomes, which brings us to the messy topic of measuring prestige. I recommend making a small, timeboxed time investment, tracking how often content comes up in hiring processes (both as a candidate and as a hirer), and avoiding spending any more time on measurement.

There are several measures I specifically recommend avoiding:

Pageviews are usually the easiest measure to instrument, but establish the wrong incentives. This is because it’s much easier to drive pageviews by being controversial than by being thoughtful, but controversy reduces prestige rather than enhancing it. Pageviews also incentivize selecting large audiences (“early-career software engineers”) over influential audiences (“technology executives”), whereas the influential audiences will become increasingly important to your career and hiring priorities as you get further into your career.
Social media followers are a good measure of reach. Reach is part of distribution, and distribution is an important part of building prestige, so this is a useful measure, but again suffers from the same issues of incentives as measuring pageviews. There are many ways to build social media reach that are at odds with building prestige, particularly anchoring on controversy, which makes this a poor measure.
If you’re selling your content as a book or course, then measuring sales is attractive. Unfortunately, once again the correlation with prestige is questionable. Particularly if this is your only content, it’s likely that selling it will reduce your reach.
Volume of writing is also tempting, but this will orient you towards building an audience rather than building prestige. There’s certainly nothing wrong with writing a lot, but it’s an inefficient route towards prestige.

If you build out a heavy investment into your brand, it’s unavoidable that you’ll end up measuring the above measures, but it’s very unlikely that this measurement will be useful to you. You can fight that by looking for better measurables, but I’ve found it’s much easier to address that friction by limiting yourself to a modest investment, and channeling the saved energy towards something more directly useful to your business or career.

Summary

In this piece, you learned how to build prestige for yourself as an executive, and your engineering organization as an employer. You also spent time on the distinction between building prestige and building an audience, and why it’s more efficient to prioritize prestige unless you’re selling directly to software engineers. You further spent time on why most measurements will steer you away from your goal, which should encourage you to timebox your investment rather than invest more heavily into measurement.

If you only take one idea away, it’s that you should be lightly skeptical of following advice and patterns from the companies and executives around you. Companies and individuals play many different games when they create and distribute content online, and the rules and values conflict across many of those games. Make sure you know what game you are playing, and why you’re playing it.

Playing with Streamlit and LLMs.

Mon, 19 Jun 2023 06:00:00 -0600

Recently I’ve been chatting with a number of companies who are building out internal LLM labs/tools for their teams to make it easy to test LLMs against their internal usecases. I wanted to take a couple hours to see how far I could get using Streamlit to build out a personal LLM lab for a few usecases of my own.

See code on lethain/llm-explorer.

Altogether, I was impressed with how usable Streamlit is, and was able to build two useful tools in this timeframe:

a text-based LLM notebook key, with prompt history
a spreadsheet-based LLM notebook

It would be pretty easy to port over the tooling to query embeddings from my blog, which I build in an earlier blog post, as well as to index and query against arbitrary pdfs following the ask-my-pdf demo.

These are just rough notes, if you’re not interested in Streamlit, consider just skimming for the screenshots below to get a sense of what you can build in a couple hours.

Installation

First, let’s create a new virtualenv to play around in:

mkdir llm-streamlit && cd llm-streamlit
python3 -m venv env
pip install streamlit

Then we can run the “Hello world” script to verify the install:

streamlit hello

This should bring you to a streamlit page.

Making our own Hello World

Next, we can follow these instructions to write a very simple Streamlib page:

import streamlit as st
import pandas as pd
df = pd.DataFrame({
'first column': [1, 2, 3, 4],
'second column': [10, 20, 30, 40]
})
df

Which we can then run via:

streamlit run main.py

Which turns into this website at localhost:8501.

We can also add a linechart and highlight the maximum value by replacing df with more explicit commands:

st.line_chart(chart_data)
st.dataframe(df.style.highlight_max(axis=0))

Which gives us a nice little linechart on top of our table.

Now, let’s experiment with creating a widget:

import streamlit as st
st.text_input("Your name", key="name")
name = st.session_state.name
st.write(f"This table is created by {name}:")

Which allows us to

Next, let’s add a second page so we can see how we’ll create different pages. We’ll keep our main.py, adding a call to st.sidebar.markdown to populate content for our sidebar:

import streamlit as st
import pandas as pd
st.markdown("# LLM Notebook")
st.sidebar.markdown("# LLM Notebook")
df = pd.DataFrame({
'first column': [1, 2, 3, 4],
'second column': [10, 20, 30, 40]
})
st.text_input("Your name", key="name")
name = st.session_state.name
st.write(f"This table is created by {name}:")
st.dataframe(df)

Then we’ll create a second page in pages/pdf.py that is a stub for later adding a tool:

import streamlit as st
st.markdown("# PDF Extraction")
st.sidebar.markdown("# PDF Extraction")

Now we’ve pulled together a very simple scaffold that we’ll extend with interacting with OpenAI’s LLM APIs.

Interfacing with LLMs

For this section, I’m leaning on Langchain_Quickstart from Streamlit.

Our first step is to install LangChain to simplify interacting with LLMs:

pip install openai langchain

Then we’ll want to figure out how to load our OpenAI API key. The standard example includes an input in the sidebar for the OpenAI API key, but I don’t want to have to paste my key in every time, so I’m reusing a key loading pattern that I borrowed from Simon Willison in my last experiments with OpenAI.

I’ve created a oai_utils.py and adding this function to it:

import os
def get_openai_api_key():
api_key = os.environ.get('OPENAI_API_KEY')
if not api_key:
key_file = os.path.join(os.path.expanduser("~"),
".openai-api-key.txt")
api_key = open(key_file).read().strip()
return api_key

You can then either create a file with your key at ~/.openai-api-key.txt or simply include your api key as an environment variable when you run Streamlit:

env OPENAI_API_KEY=ABC123 streamlit run main.py

Either way works fine. Now let’s update main.py to support querying OpenAI:

import streamlit as st
from langchain import OpenAI
from oai_utils import get_openai_api_key
openai_api_key = get_openai_api_key()
def generate_response(input_text):
llm = OpenAI(temperature=0.7, openai_api_key=openai_api_key)
st.info(llm(input_text))
st.markdown("# LLM Notebook")
st.sidebar.markdown("# LLM Notebook")
with st.form('my_form'):
text = st.text_area('Prompt:', '',
placeholder='How do I use Streamlit to query OpenAI?')
submitted = st.form_submit_button('Submit')
if submitted:
generate_response(text)

This code gives us a fully functional, albeit basic, setup for querying OpenAI.

Next we can start adding some functionality.

Selecting between OpenAI models

Right now we can’t select the model we want to query against, which is a bit annoying, so let’s use the st.selectbox widget to let us select between models.

This is the same as above, with tweaks to generate_response and the call to st.form:

def generate_response(input_text, model_name):
llm = OpenAI(temperature=0.7, openai_api_key=openai_api_key, model_name=model_name)
st.info(llm(input_text))
st.markdown("# LLM Notebook")
st.sidebar.markdown("# LLM Notebook")
with st.form('my_form'):
oai_model = st.selectbox(
'Which OpenAI model should we use?',
('gpt-3.5-turbo', 'gpt-4', 'ada', 'babbage', 'curie', 'davinci'),
)
text = st.text_area('Prompt:', '',
placeholder='How do I use Streamlit to query OpenAI?')
submitted = st.form_submit_button('Submit')
if submitted:
generate_response(text, oai_model)

Now we’re able to query using whichever model we want. Note that gpt-4 won’t necessarily work if you don’t have beta access to that model, but otherwise things should be fine. The list of OpenAI models is available here.

This is, arguably, a good enough interface for interacting with the OpenAI LLMs using your own API key, if you’re not interested in using the hosted chat interface for some reason, most likely because you have an Enterprise contract for your API key to provide clearer control over any data included in your API requests.

LLMs and Spreadsheet (aka Editable Dataframe)

Next, let’s experiment with creating a spreadsheet interface to LLMs. The goal is to allow you to create a spreadsheet, then run an LLM against each row individually. For example, you might have an email in each row, which you want to test a prompt against.

My first hope was to use st.data_editor (introduced in this blog post) to allow dynamically creating and editing spreadsheets. This almost works, as you can add rows, but it doesn’t let you add/remove columns. I’m certain you could figure out a way to make this work, but for simplicitiy I decided to allow you to upload a CSV to populate the table, and then let you add/remove rows after that initial upload, which we can do that using st.file_uploader.

I created pages/spreadsheet.py which is 50 lines of Python, which you can see on Github.

I tested this using a subset of the financials spreadsheet from my recent Planning blog post, and I think the iterface is pretty good here. I exported a Google Sheet into CSV, uploaded it, and then was able to use the st.data_editor to remove rows I didn’t want to see (slightly unintuitive, but select a row then use the delete button to get rid of it). From there, I was able to run a prompt against each row to extract data from the rows.

I will say, I think the specific data I selected was exceptionally bad for this kind of analysis, but you could imagine it performing much better against other types of data, for example generating an outreach email for folks in a spreadsheet based on some information about them.

Recording previous prompts

Returning to main.py, I wanted to see how hard it would be to store my previous prompts in the left sidebar, which lead to playing with session-state. I could not get session state to work at all for local development. It’s quite possible that this is a “Streamlit cloud”-only feature or some such. I was able to initialize a list of prompts, validate I’d updated that list, but it always reset to empty, even when I hadn’t closed the tab or even left the page, e.g. after submitting a new prompt to evaluate, it would always detect session state as empty.

So, I decided to just implement it using a local JSON blob, which you can see in get_prompts, render_prompt, and add_prompt on Github.

If you were building a more comprehensive tool, you’d probably want to fiddle around with st.experimental_user to create a per-user cache rather than a global cache. That is made to work with Streamlit’s cloud hosting, but I imagine if you boke around the streamlit-authenticator that you can figure out how to work with other authentication mechanisms as well.

Ending thoughts

Altogether, I’m certain this sort of “personal LLM tooling” will exist in a number of usable formats sometime over the next year or two, but I’m not sure if it’ll be better than what you can build yourself, particularly when it comes to customization. The one counter-point here is the ChatGPT plugins which are exceptionally interesting, and might well represent the sort of tool that is nearly impossible to build or interact with in your own local sandbox–we’ll see!

Equally interesting, I think, is seeing how these kinds of internal tools will play out for internal usage at companies building out their own investment into LLMs. Based on what I’m seeing in the industry, I’d expect most companies with a legitimate use for LLMs to build out something along these lines internally. I’m fairly confident that there’s a real usecase for self-hosted LLM tooling, which could be an interesting startup idea, but there’s a real risk in building a startup where there’s already a lot of startup saturation, I would almost certainly go in a different direction. (If you are building something along these lines, are looking for angel investment, and are trying to build tooling for LLMs rather than sell LLM-magic, I’d be interested in.)

Extract the kernel.

Sat, 27 May 2023 06:00:00 -0600

As I’ve served longer in an executive role, I’ve started to notice recurring communication challenges between executives and the folks they work with. The most frequent issue I see is when a literal communicator insists on engaging in the details with a less literal executive. I call the remedy, “extracting the kernel.”

For example, imagine a team is presenting about their upcoming timeline, and the CTO asks, “Can’t you just use ChatGPT to solve that instead of building a custom model?” From there, the conversation will often derail into a debate about ChatGPT versus building a custom model, but the CTO’s point is almost always not about ChatGPT. Instead, their point is that the timeline feels too slow. If the team anchors on responding to the specific suggestion, they’ll miss the more important discussion entirely. Even if they convince the CTO that a custom model is the better choice, the CTO will be annoyed that their real concern about the schedule wasn’t addressed.

If you’re that team presenting to the CTO, you could rightly argue that executives ought to be better communicators. That’s certainly true, but the reality is that executives are human. You’ll make much more progress by focusing on improving how you communicate with them than by blaming them for their deficiencies.

I recommend that teams receiving executive feedback try to “extract the kernel.” When you get a question from an executive, focus on understanding the insight or perspective within the question. Then confirm that insight with the executive explicitly. Going back to the example above, after the CTO recommends using ChatGPT, you could confirm back by saying, “To make sure I’m understanding, the most important feedback here is that we should figure out how to accelerate our timeline?” Now you can ensure that the CTO’s feedback is addressed rather than getting caught up on the incidental details in their question.

Slides for Measuring an engineering organization.

Sat, 20 May 2023 06:00:00 -0600

Last week, I gave a 30 minute talk to a group of CTOs and VP Engineerings in San Francisco about measuring engineering organizations. This talk was essentially this blog post, and here are the slides.

A few topics worth highlighting:

Measurement educates you, and your audience, about the area being measured. Even flawed measures can be very effective educators. Don’t get caught up on not measuring things because they have some flaws, let the audience learn about those flaws
Instrumentation is costly to implement. Let specific problems guide you to instrumentation, don’t instrument widely without a clear goal
Many new companies cargo-cult Amazon’s approach to metrics, but that doesn’t necessarily work
If teams don’t trust each other before you have metrics, they probably won’t trust each other must afterwards. Metrics are implicitly objective, but there are many judgment choices within selection and measurement

Unfortunately, I don’t have a recording of the talk itself, although I imagine I could record it relatively easily once my time opens up a bit more. Which won’t be in the near future! Quite busy with the new role, capturing the new ideas I’m learning in my new role, and finishing up my next book.

Good hypergrowth/curator manager.

Sun, 30 Apr 2023 06:00:00 -0600

In 2016, I wrote Productivity in the age of hypergrowth to discuss the challenges of engineering management during periods of hypergrowth. Managers in such periods spend much of their time on hiring and onboarding, with the remainder devoted to organizational structure and high-level strategy. Their technical expertise is important, but it’s demonstrated indirectly in the quality of their strategy, structure, and hiring.

In 2023, our universe has shifted. There’s little hiring happening, and most companies are eliminating roles to meet investor and market pressure to operate in an environment where fundraising new cash is significantly more expensive. The role of engineering management has changed as well. Hiring and onboarding are now secondary components of our work, strategy and structure are elevated, and there’s a demand for us to employ our technical expertise more directly.

My experience is that this shift is real, has been relatively subtle, and hasn’t really been directly acknowledged in any discussion I’ve had over the past couple years. Consequently, I wanted to write up a short comparison between the “hypergrowth manager” and “curator manager” archetypes, in the spirit of Ben Horowitz’s Good Product Manager/Bad Product Manager. This list reads a bit skeptical of hypergrowth management, but that’s to better articulate the distinction: both are valid approaches to different situations.

Good curator/hypergrowth managers:

Curators don’t spend time arguing internally for headcount, because they know the headcount ins’t coming; they create more impact by picking the right work and ensuring that work’s effective implementation. Hypergrowers know that headcount growth will solve most current problems in the long-run
Curators view every project as essential, ensuring their team’s work is successful short-term, and that it ladders up into a larger strategy over time that accelerates the team’s impact. Hypergrowers have so many bets that they focus on a portfolio approach where only some, not all, bets need to gain traction
Curators are in the data to align their team with the real opportunity, and are stubborn about starting work whose premise doesn’t connect with their understanding of the data. Hypergrowers know that the underlying shape of the data is changing rapidly and their mental model may easily be out of date since last month
Curators treat hiring and onboarding as one-off projects, and know that the hire has to fit into their specific team. Hypergrowers treat hiring and onboarding as a program, and know that the memberships of each team will change significantly over the next year
Curators lead teams of real, specific individuals, and work to find task-individual fit. Hypergrowers lead teams of rapidly shifting individuals where tasks and individuals shift too often to optimize task-individual fit
Curators know that growth won’t fix their organizational missteps, and engage with issues directly (e.g. if a team is too small to handle an on-call rotation, they merge teams). Hypergrowers know that missteps will be solved indirectly by organizational growth without requiring direct resolution
Curators know that team morale is a precious resource, and invest explicitly into maintaining morale. Hypergrowers know that the rapid expansion of the company’s valuation will paper over most morale issues
Curators find ways to grow their career that avoid artificial competition with colleagues. Hypergrowers often make the mistake of viewing career growth through the size of their team
Both know that poor quality and technical debt will slow forward progress, even though time is constrained for different reasons
Both prefer properly sized teams of 6-8 engineers

Shifting elevation, a few thoughts about Directors and VPs operating in these modes, which I’ll lump together using the term “executives” for ease of use:

Curator executives are judged by their execution against a rotating spotlight of emergencies. Hypergrowth executives are judged by the outcomes of their diversified portfolio of bets
Curator executives know that their ultimate impact is derived from systemic changes to culture and durable investments, but that they’ll be judged almost entirely on managing emergencies: real success comes from successfully managing both dimensions. Hypergrowth executives know that there will be emergencies constantly, that they’ll burn out quickly if they personally address each one, and focus instead on managing their portfolio of bets: success comes from maintaining altitude as emergencies try to pull you in
Curator executives work with their teams to fully resolve fires. Hypergrowth executives mitigate fires until they can be handed off to newly hired managers for resolution
Curator executives generally solve problems directly. Hypergrowth executives generally work via programs. Both prefer to work the policy rather than solve via exceptions
Curator executives anchor on reality as perceived by their engineers. Hypergrowth executives anchor on reality as perceived by their engineering managers
Curator executives are deep in the margin profile and revenue plan. Hypergrowth executives maximize revenue growth without deteriorating margin much
Curator executives design organizations that are steady-state durable. Hypergrowth executives design organizations that minimize the impact of frequent expansions

As an ending thought, if you have a suggestion for a better term than “curator”, let me know. The first term that came to mind for me was “steward manager” but there’s an implication of passiveness that misses the mark. Replacing “manager” with “leader” is unhelpful because it misses the reality that these folks are still managers and fundamentally doing management work.

The Engineering Executive's Primer.

Thu, 27 Apr 2023 06:00:00 -0600

See on O’Reilly’s website for The Engineering Executive’s Primer.

In 2019, I worked with Stripe Press to publish my first book, An Elegant Puzzle, which captured many of the lessons I’d learned as an engineering manager in fast growing Silicon Valley companies. In 2021, I decided to learn the entire process of publishing myself, self publishing my second book, Staff Engineer, which synthesized the stories of a number of individuals into a framework to attaining and operating in Staff-plus engineering roles. I wrote both books because I kept working with very talented individuals who were nonetheless lost in their roles.

As I worked in my first engineering executive role at Calm, it became hard to ignore the drumbeat of recurring questions. How do I work with my CEO who doesn’t have an engineering background? Harder still, how do I work with my CEO who does have an engineering background and thinks every project should take at most four hours? How do I measure engineering capability? How do I report to my board about engineering’s impact and progress? How do I run planning? How do I balance engineering resources across multiple business units?

After three years in role, and dozens of attempts at answering each of these questions, I felt like it was time to pull together those answers into a book, which has become O’Reilly’s The Engineering Executive’s Primer. The book is in early release now; I anticipate finishing writing in 2023, and hope to have the printed book available by June, 2024.

As I did for my previous two books, I am writing the book “in public”, and you can see early drafts of each chapter using my blog’s “executive” tag. There is a lot under that tag already, as I’ve been writing full time for the last 6-8 weeks, probably 150-200 pages. The final drafts in the book will be different, having gone through extensive editing and technical review, but you can get most of the book’s core ideas from these posts as long as you don’t mind the occasional typo or confusing wording.

A few folks have asked about Infrastructure Engineering: I am still planning to write that book, but it’s now second on the list. Primer felt at the top of my fingers, like I could just sit down and write it, so I decided to focus here for the time being before I started to forget everything I’ve learned over the past few years. I do fully intend to comeback to write Infrastructure afterwards, but that’s looking like a 2025 or 2026 release at this point.

You can see the Early Edition on OReilly.com, and subscribe to the Irrational Exuberance newsletter for draft chapters (along with my other writing).

Balancing your CEO, peers, and Engineering.

Thu, 20 Apr 2023 06:00:00 -0600

There are so many stories of hiring a new executive who comes in and wreaks havoc. I’ve seen engineering leaders start with a giant, doomed migration, marketing leaders who accelerate expenses until they necessitate a round of layoffs, and a number of executives fired in their first month. When people tell these stories, it’s almost always framed as a failure of the individual executives, but they happen so frequently that I believe there’s an underlying structural challenge in addition to individual missteps. Fortunately, this structural trap that snags many executives can be avoided by acknowledging its difficulties and navigating them with a deliberate approach.

When a new Engineering executive is hired, they are usually hired by a CEO who believes that the Engineering function is underperforming. When you talk to members of Engineering, they will have a different perspective, potentially that the CEO keeps changing direction too frequently. When you talk to peer executives and to the Board of Directors, there will be a third and a fourth narrative about what’s happening. Where new executives fail is that they think of these as opposing perspectives, when in reality they are all incomplete but valid slices of a complex situation. Successful executives debug these mismatched concerns until they merge together into a single cohesive perspective. Ineffective executives anchor on one or two perspectives and dismiss the rest.

In this piece, we’ll discuss these topics on building effective relationships that overcome this core structural challenge of being a new executive:

Understanding whether you’re supported, tolerated or resented
Navigating the implicit power dynamics
Bridging narratives across the CEO, Board, peers, and your function
Avoiding anchoring on previous experience
Fostering an alignment habit
Focusing on a small number of changes at a given time
Embracing transient conflict and preventing persistent conflict
Surviving a panicking peer executive

After working through these topics, you’ll have a clear roadmap for debugging the mixed messages inherent to operating as an executive, and an approach for building durably healthy relationships that support you through your entire tenure.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Are you supported, tolerated or resented?

It’s obvious to say, but the most effective executives are proactively supported by their peers, the CEO, and their functional team. This is not the common scenario where members of the executive team operate in isolation within their functional spheres, and it’s very different from environments where some functions maintain a long-running animosity towards one another.

There’s no status website to check your relationship with other parts of your company, but it’s reasonably straightforward to figure out:

Supported is when others proactively go out of their way to make your efforts successful. For example, if you push for a monolith to services rewrite, they’d come to you with concrete concerns for why this might not work out, and ideas for how to adjust your approach.
Tolerated is when others are indifferent to your work. For example, they hear about your services rewrite, and pull any related work tickets into their sprints, but don’t make an effort to flag structural inefficiencies in your approach.
Resented is when others view your requests as a distraction from their actual work. For example, they’ll advise their teams to ignore your migration tickets until you escalate loudly.

If you’re already supported by most executives and functions you work with, then you’re in an excellent place and can stop reading now. If you’re not sure, or suspect you’re tolerated or resented by some functions, then read on to unpack the most frequent causes of damaged relationships and how to resolve them.

Navigating the implicit power dynamics

Even if you’re an executive who prioritizes supporting your team, it’s very hard not to come into your new job anchored on your CEO’s evaluation of how Engineering is performing. This is natural because you’ve just spent the entire interview process speaking with the CEO, and the CEO is your boss: if you don’t impress them, you’ll get fired. However, CEOs only hire new executives when there is a problem to solve, which almost always means they’ve been telling you all about Engineering’s problems, but they’re probably not telling you about everything going well, and their diagnosis of the problem may have some gaps.

You should listen closely to the CEO, the Board of Directors, and your executive peers’ perspectives, but don’t close your mind to other new perspectives. Just because Engineering is having problems doesn’t mean Engineering is the problem, and you’ll want to dig in with the team directly before switching from listening to problem solving. To be fair, Engineering is almost always part of the problem, but they are rarely the entirety.

Executives who underestimate the impact of these power dynamics are prone to treating the top-level symptoms as if they are the underlying causes, which leads to superficial and often misguided solutions. When you look at all the executives who quickly mandate widespread migrations or initiate reorganizations, underneath the decision is almost always this sweet siren’s song of power dynamics that misleads them to dismiss the input of the people doing the actual work.

While I certainly don’t recommend ignoring your CEO–they are indeed your boss–you’ll serve them more effectively by listening widely before you solve their problem. Moving with urgency to address symptoms rarely gets you closer to solving the problem that your CEO truly cares about.

Bridging narratives

The most talented executives I’ve worked with have a remarkable skill to weave four or five seemingly incompatible perspectives into a unified understanding. For example, Product might feel Engineering is slipping on delivery. Engineering might feel that Design keeps changing requirements. Design might feel that Product is making late but reasonable change requests, but that Engineering is intolerant of their change requests. Sales might believe that Engineering is missing all their committed dates because they’re not customer-oriented. The simplest explanation here is that Engineering is struggling to execute effectively, but there’s undoubtedly a more nuanced diagnosis that goes beyond casting blame, and that nuance will make it much easier to identify an effective solution.

When you run into a complex problem, slow down to bring many different perspectives into consideration before you push to solve the problem. When you get good at this, it doesn’t take much more time, and rather speeds up problem solving by skipping the contentious debate around assigning blame.

Executives who practice this skill of bridging narratives are acting as company-first problem solvers rather than operating from a place of loyalty to the function they lead. This leadership creates a culture–and expectation–of cross-functional partnership, and is the most effective way that I’ve personally found to build supportive relationships across functions and executive peers.

Don’t anchor on previous experience

Many new executives accidentally alienate their peers and functions by assuming their new company will work the same way that their old company worked. This is particularly common when senior leaders at large companies transition into smaller startups. Those leaders are conditioned to an unnatural degree of support and resources, which simply don’t exist in smaller companies. Simply by running their familiar executive playbook they can easily alienate their new colleagues.

Avoid this by watching how others solve problems in your company and then asking them why they solved them that way. Some executives think it’s faster to learn by running into walls, but I’ve found that to be ineffective when those so-called walls are in fact your coworkers. No one appreciates being run into, and it creates a lasting impression that you’re focused on yourself rather than operating as a member of a collaborative executive team.

Fostering an alignment habit

Many executives assume feedback will come to them if they make mistakes. This is eventually true, but you’ll get the feedback much later than useful. You should instead build a habit of inviting feedback and making sure you receive it well.

This might be reaching out to participants in an executive meeting to ask if there’s anything you could have done better, or should not have done at all. Much of the time you won’t get any meaningful feedback, but occasionally you will. When you do get feedback, even when it isn’t particularly helpful, say thank you and ask a follow up question to better understand the feedback. When possible, ask narrow questions like, “Did I ramble a bit when I shared my perspective about reducing bonuses?” which are much easier to answer than open-ended requests for feedback. Most importantly, strive to reward feedback by actually improving on the things that are mentioned.

You don’t need to do this all the time with everyone you work with, but make a deliberate effort to occasionally ask everyone you work with closely. You should always be asking for occasional feedback in order to ensure folks have an opening to share.

Focusing on a small number of changes

Each time you come to colleagues with an idea, they refer back to their mental list of the last few ideas you brought them. Did those ideas go well, and did they end up feeling good about them? Are those ideas still ongoing without clear resolution, did they end inconclusively, or did they end up being viewed as a waste of time?

As long as your previous ideas have been effective, then folks will generally be glad to support your next batch. If they haven’t gone well, then they won’t. Consequently, your best bet to retain the support of your team and peers is to focus on delivering a small number of changes with meaningful impact. Don’t be the executive who pushes a number of technology migrations before finishing the first one. Don’t be the executive who claims to rework the planning process without ever using that new process.

Pick a few things that you’re confident will work, and deliver them. This is obvious advice, but it’s often ignored, and ignoring it will quickly get you placed into the tolerated or resented buckets.

Having conflict is fine, unresolved conflict is not

Many executives aim to eliminate conflict in their working relationships, which I believe is subtly the wrong goal. Real companies experience rapid changes as they grow and the markets they operate in shift. If you’re experiencing zero conflict day to day, then it’s very likely that you are ignoring conflict rather than resolving it.

Conflict isn’t inherently negative: experiencing new kinds of conflict is an important sign of growth. The sort of conflict that you want to avoid is unresolved, recurring conflict. An example of healthy conflict is an executive team who disagree on implementing a return to office policy. This is a complex topic with a number of different perspectives, and you want to surface all those perspectives enroute to deciding on your company’s approach. An example of unhealthy conflict is an executive team who strongly disagree on the size of relative investment between Research & Development (R&D) and Sales & Marketing (S&M), and who continue arguing without alignment each time they revise the financial plan.

When you detect an unresolved conflict, spend time understanding the different points of view and why they aren’t able to agree. Much like the challenge of finding a unified narrative that spans CEO and functional perspectives, there is usually a unified view that acknowledges seemingly opposing perspectives. If you can find it, you can often untangle the conflict. If not, it may be time for your executive team to establish a clear escalation mechanism to explicitly resolve points of conflict.

Finally, keep an open perspective on whether you are the person preventing conflict from resolving. Sometimes I’ve gotten so attached to my own perspective that I couldn’t let things go, even when it wasn’t particularly important, which always ended up causing more harm than it was worth. Even if you disagree, it’s still worth letting most stuff go to keep the executive team healthy.

Surviving peer panic

Sometimes things go awry even if you build effective relationships within the company. The most frequent version of this is when one of your peers starts to get negative feedback from the CEO, and that peer struggles to incorporate that feedback effectively. This is particularly messy to deal with as a newly hired executive, who is trying to build a relationship with a peer who’s more focused on short-term survival than building a long-term relationship with you.

Some examples of this happening:

Engineering executive is held accountable for slow delivery, and switches to blaming Product for frequently changing feature roadmap
Sales executive is held accountable for missing quota, and switches to blaming slow Product and Engineering delivery
Marketing executive is held accountable for high customer acquisition costs, and switches to blaming Engineering delivery on a marketing website

When this happens, try to find empathy for the individual who is struggling to address the concern, and then sit down with them to try to solve it together. Your peers will sometimes struggle, and sometimes they will point blame towards you, but ultimately succeeding as an executive team is a group activity. Even if an executive is performing poorly today, it’s usually less disruptive for you and the company to improve that executive’s execution than to find and hire someone new. Even if you aren’t ultimately able to help this particular peer succeed, you’ll come away with a clearer understanding of how to support their replacement.

At their worst, these situations get pretty unhealthy, particularly if the CEO has written off an executive but doesn’t have a concrete plan for exiting the executive from the business. The longer this goes on, the more uncomfortable it will get. At that point, push your CEO to make the change, and insert yourself as a buffer between your team and the panicking executive. Until your CEO cleans up the mess, you can at least minimize the disruption on your team.

Summary

At this point, you have a clear framework for becoming, and remaining, an executive supported by your peers, your CEO and your team. Continue to invest in your relationships. Continue to find the overarching perspective that merges the seemingly incompatible views around you. If you’re ever uncertain what to do, search for the approach that maximizes your impact at the company over the next three years rather than the next three months, and do that.

Grab bag of random thoughts.

Mon, 17 Apr 2023 07:00:00 -0600

A bit over a week from now, I’ll be joining a company to start a new role, and I wanted to ramble a bit to braindump the numerous loose threads in my head as I transitioned from Calm to the past month of full-time writing, and then into this new role. This isn’t really a job announcement post per se, as I won’t share any details about the job itself until I’ve officially started. Instead, this is a snapshot of what’s top of mind for me, particularly driven by the dozens of discussions with friends and colleagues as I thought about what’s next for me over the past few months.

Job search

My last day at Calm was in early March, and I was planning to take three to six months off before starting my next job search. My initial goal was to specifically avoid talking to anyone about open roles until I was ready to start, because there is a certain momentum to executive searches that is hard to avoid: if you’re an eligible candidate who interviews reasonably well, once you start talking to VCs and recruiters, you’re going to end up placed somewhere. If you fight against that momentum for too long, you’ll eventually annoy the folks trying to place you, turning supporters into neutral parties at best.

If you disagree that there’s some risk to resisting executive search momentum, a quick aside for you. At this point in my career, I am selected as a backchannel reference for a number of folks. If you’ve worked with me, and you interview for an executive role at companies backed by a certain handful of venture firms or staffed with executives whose network overlaps with mine, it’s fairly likely I will get a call about you. I try extremely hard to center the positive for everyone I’ve worked with, even in the rare case that I didn’t love working together with them, but it highlights something important: even if you’re trying to spare me from performing too many references on your behalf, if you run a long executive search and we’re worked together, then I am getting a lot of pings. It’s not that I’m special, this is how the executive recruiting ecosystem works. If you run a long search–even if you don’t personally pull many people in–you will tire out your network. The VCs and recruiters will get tired too.

Back to my own search: Talking to friends, a recurring theme was the lack of exceptionally good executive openings in 2023 relative to searches in prior years, especially relative to the 2020-2021 era. There were still many open executives roles, but many were in deeply challenged businesses or very early businesses (e.g. Series A). There were still some executive roles in thriving businesses, but there simply weren’t very many of them. This made me rethink my planned three to six months break before talking with recruiters.

I ended up deciding to talk with companies about roles that I felt confident I would accept, barring uncovering major flags in the interview process, and where I could accept an offer without experiencing FOMO about the other companies I didn’t speak with. In other words, set a high bar,, and let it take as long as it takes to find an opportunity. This worked a bit faster than anticipated, and I’m quite excited about the role and company I’m starting next Monday. I’m confident that I wouldn’t have found a better opportunity for me, even if I’d spent the next six months talking with companies, and equally confident I would hae regretted saying no.

Leaving Calm & how I thought about my next role

When I was thinking through my decision to leave Stripe (the role I left to join Calm), I wrote A forty-year career, which describes each role as a mix of profit, people, prestige, learning and pace. This framework continues to resonate with me. The nuance I’d add to it, as I’ve gotten better at managing my own energy, is that pace is often more of a ratio between energizing and draining work rather than an absolute speed.

My time at Calm was very rewarding, and the hardest part was leaving the leaders, peers and team that I got to work. What solidified my decision to leave was my belief that my rate of contribution to the business and my rate of learning were both tapering off. If I could go back in time to 2020 and pick any available job, I’d pick Calm again, because I have learned so much over the past three years, but it also felt like the right moment for me to move on to something new.

As I thought about what I would do next, finding an opportunity with a significant rate of both contribution and learning was the foremost criteria, and I believe that four years from now I’ll have learned just as much as I learned over the past four. This isn’t a precise science, but if I keep learning enough to write a new book every four years, that will be good evidence that my learning at work remains at the right pace.

Sabbatical

As mentioned above, I left Calm planning to take a three to six month sabbatical, aiming to finish writing my next book. I’d say that I’m about 60% done writing the initial draft, and will slow down a bit as I start my new job. I remain optimistic that I’ll finish it over the next four months or so, which was the timeline I established with O’Reilly, admittedly under the assumption that I wouldn’t be returning to work quite so early.

The book’s first chapters will be up for early release very soon, at which point I’ll write more about that project. It just feels a wee bit premature to write about something I can’t link to. (For the record, this upcoming book is not going to be Infrastructure Engineer, which has fallen a bit down my priority list. My hope at this point is to pick it back up in late 2024, I don’t think I’ll see much progress on it until then.)

A number of people have asked me for sabbatical advice, and I’ve established absolutely no credibility in terms of doing the sabbatical I intended to, but I’ll still share what was important to me when thinking about the sabbatical and then implementing it:

Particularly in 2023, you should have a financial position to support a six month job search. That is in addition to the resources to pay for the sabbatical itself. The job market out there is just very strange right now. I’m seeing some folks find new jobs very quickly, and others struggle to find new jobs for months.
Figure out whether others in your family will or won’t have time to do it with you, and adjust for that. My wife was continuing to work, so this meant I had more time to myself, but also that I needed to keep to my existing child care and family schedules. Life is long and complex, don’t try to make others take time when you’re taking time, let people live!
Give yourself a few weeks, maybe two, to do absolutely nothing productive. Don’t have a schedule, don’t have meetings, don’t take calls, don’t catch up with friends. Just relax. Maybe go somewhere else.
Don’t get anxious and start scheduling lots of meetings. You can easily spend all of your time taking meetings. You could just keep working and get paid to take meetings if that’s what you want.
Have a clear set of goals to focus on. I kept a strict writing schedule during my time, which helped me feel productive. I occasionally “cheated” on the schedule sometimes to take care of my son, work on a fun project, meet up with a friend or whatever, but I mostly kept to the schedule. It was not a brutal schedule by any means. It was peaceful, but focused.

I also had a running goal, to get back up to an 8-mile weekly “long run”, which I hit last week as the deadline started to get uncomfortably close. As someone who’d fallen into three mile maintenance runs for the past decade, it was a good to remember that running further is mostly about not stopping when it gets uncomfortable.
Remember that rest is about resting. There have been days when I didn’t get stuff done, or even get stuff started, and I just gave myself permission to rest. When working, I would have pushed through and gotten it all done, but my goal for this time was essentially to heal and recuperate from the last ~15 years of working full-time, so I didn’t ever push. If it felt like I needed to push productivity harder, I just relaxed the constraints instead. I wanted to be ready to utilize my higher gears when I returned to work, rather than to exhaust them during my break.

If I could do it again, I think I’d do it the same way! Life comes at you fast, and rarely according to plan, but I don’t think I could have planned it more with better results.

Who’s successful anyway?

Recently when I chatted with a friend about our careers, we got onto what I think is an interesting topic, which is the inscrutability of success. There are people in the industry who appear extremely successful, but who are not. There’s two different dimensions to consider here.

First, there are individuals who you’d assume are financially very successful but who are working out of necessity despite working very high prestige roles (e.g. head of engineering at a trendy company for years, but still working out of necessity 25-30 years into their career).
Second, there are individuals who are widely considered very successful by folks without detailed context of their work, but who are considered very unsuccessful by those with concrete awareness of their work (e.g. well-known on conference circuit but considered a poor performer in role by former colleagues).

As I’ve gotten deeper into the industry and a bit better networked, it’s less common that I’m shocked by someone’s performance–usually I know someone who knows someone who worked with them–but I remain constantly surprised at the industry’s inconsistent financial outcomes. I don’t have any concrete lesson to share, but I think it’s interesting to note that I’m still frequently surprised at how little financial success is extended to some nominally extremely successful people in the industry. Conversely, there are also many, many folks who are financially successful, without a clear correlation between that success and their various contributions. Downturns make this phenomenon even more extreme, with many excellent folks failing to “cash out” of their work, largely due to timing constraints out of their control.

Culturally, there’s a strong pull towards using financial success as a sort of moral compass, but it’s my experience that luck plays far too high a role. Even if you think you’re appropriately discounting the role of luck, I suspect you are still underestimating its impact.

Executives without much range

There are a number of executives out there who are very good at some things, but lack the flexibility to operate in varied environments. Sometimes this is because they are stubborn, and have a specific working style that they insist on following independently of the company. Another major contributor, in my experience, is executives who lack experience working in middle management.

Middle management is, of course, something that people often view as fake work, but it’s the critical work of translating an executive’s stated plan into a series of real plans that the company can actually implement. Executives who don’t understand this are doomed to create systems and processes that impede organizational execution, often screwing things up while claiming to improve organizational execution. Based on my experience, I don’t think you can be an excellent executive at a scaled (or scaling) company without middle management experience.

There are lots of details here, but the two biggest ones I’d mention are: I find that executives without middle management experience often rely on trust rather than inspection, and then generally a lack of understanding of how to design useful processes (usually viewing process as a one-off thing rather than an ongoing evolution).

LLMs

I had fun playing around with the OpenAI API this past week. I summarized my view of this technology shift as, “LLMs are showing significant promise at mediocre solutions to very general problems.” This captures my perspective pretty fully, but there’s one other nuance I’ll throw in for the moment: chat hasn’t yet proven itself as a durable paradigm for serious interactions. It’s a good paradigm for broad discussions that can tolerate some loss of meaning, e.g. initial communication about intent to purchase a SaaS product or customer support for a user who’s ordered two specific items, but not a good interface for areas where meaning is fundamental, e.g. here is how you should write a legal document. This is similar to the issues Alexa and Siri as meaningful product lines, when they’re largely stuck playing music, setting timers, and checking the weather.

The issues with chat absolutely don’t mean that LLMs won’t find a sweet spot, I’m just a bit skeptical that chat’s the sweet spot. Most valuable applications are trying to do valuable work where correctness matters, and I don’t see this technology as likely to upend those sorts of interactions. I’m not in the rooms making these decisions, but my personal hypothesis is that chat is a very smart use case where you can show significant value while minimizing costs (e.g. imagine the price of evaluating LLM responses for each of your ten million users every night as opposed to only incurring those costs when a human directly asks for one).

That said, it’s clear that LLMs are going to absolutely break the current paradigms of writing, editing and evaluating written work. Publishers are already getting overwhelmed with generated submissions, and it will be instructive to see where that ends up. I’m personally a bit concerned about it, not because I think it’ll impact me in a major way, but because I think it has the potential to make it much more difficult for new writers to distinguish themselves. That said, it’s exciting for the world to keep changing, and it was never going to statically remain the same to prioritize my comfort over technology’s drive to innovate.

That’s all folks

OK, I’ve gotten my random thoughts out of my head for now. I’ve been publishing on pretty much a weekly cadence for the past few months, and suspect my cadence will slip a bit as I ramp up in my new role, but hopefully not too much!

Interviewing engineering executives.

Mon, 17 Apr 2023 06:00:00 -0600

Earlier I wrote about getting hired as an Engineering executive, and it’s perhaps even more important to discuss the opposite question: how should you interview and evaluate Engineering executives? As an Engineering executive, you may not directly run one of these searches, but you’ll likely be asked for advice about how to run them, and may be asked to design the process to hire your successor.

The key topics I want to explore are:

Avoiding the unicorn search
How interviewing executives goes wrong
Structuring your evaluation process
Focusing on four areas to evaluate engineering executives

These topics will prepare you to conduct an engineering executive search that culminates in hiring a leader that can support your company today, and will help you avoid bogging the executive team down in a multi-month process that is ambivalent about potential candidates.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Avoiding the unicorn search

While most companies struggle to evaluate incoming executives effectively, there’s a second category that are effective at evaluating executives but nonetheless fail to hire effectively because they want a rare intersection of skills. For example, I once saw an engineering executive search that wanted someone with experience leading a large Engineering function, with deep go-to-market and Product experience, deep domain exposure to a narrow infrastructure engineering domain, cultural alignment with consensus-based decision making, and a sufficiently strong motor to skip consensus-making to accelerate company processes.

There is always some candidate who fits any mold you define, but hiring them gets very challenging. Identifying and hiring them is even harder once you acknowledge the breadth of the error bars inherent to this process. Worse, you usually can’t go back to reactivate a candidate after you’ve passed on them, so even if you later realize an earlier candidate was excellent, that prospect will have already passed you by. If you run a narrow search for too long, by the time you open the search up, you may have already rejected your best potential candidates.

My biggest advice for avoiding the unicorn search is to get the search’s sponsor, generally the CEO for an Engineering executive, to spend time before the formal search talking to seasoned Engineering executives to assess the profile. These don’t need to be folks you could hire, and your goal isn’t really to hire them, rather it’s to listen to their feedback on the profile you want to hire and what could make your opportunity sufficiently compelling that a qualified candidate would accept it. This will take a few weeks, but will save you months of time in the long run.

How interviewing executives goes wrong

Typically, the hiring loop for a software engineering role starts out messy and is slowly refined into an effective hiring process as you make more successful hires. This post-hire calibration process is particularly important to reduce the number of false positives and false negatives in your interviewers feedback. Executive searches only make one hire, while evaluating for a very broad role, which makes these loops even harder to calibrate.

These loops are further made challenging because there’s rarely someone wholly qualified to assess the potential engineering executives, who are being hired to be the most senior technical leader at the company, but despite that gap you’ll have many folks with strong opinions about who should be hired. Combining these messy incentives and challenges, most companies bouncing between two interview formats:

Vibes and backchannel: hiring is heavily weighted on a small number of discussions along with backchannel references who provide input on candidates previous work. These processes generate very little direct signal, which means that internal colleagues often don’t feel bought into the hires. Even the candidates themselves may not feel particularly evaluated either, which may cause them to decline the offer.
Broadchurch: hiring incorporates a wide range of internal interviewers. This might include an interview with the CEO, Product, Design, People, Finance, Marketing, Sales, along with four or five folks from Engineering. Introducing this many interviewers, many who will be unpracticed at interviewing for this role and may rely on an entirely informal interview, will generate numerous false negatives, often anchoring evaluation on the perspectives of folks without clear evaluation criteria and limited exposure to the role you’re hiring.

My experience is that neither of these are particularly effective at evaluating candidates, with the former accepting too many candidates and the latter rejecting candidates randomly. Further, these experiences leave the candidates themselves skeptical of the company’s decision making.

Structure for evaluating executives

Fortunately, it’s straightforward to design a reasonable process that’s comparable to most engineering executive evaluations and avoids some of the common missteps. That’s not to say that this process is perfect, but rather than most obvious changes introduce at least as many problems as they solve. I recommend starting with:

Recruiter screening: generally executive searches are run through an executive recruiter, an executive recruiting firm, or a VC recruiting firm. They should lightly filter for candidates’ interest in the role and their plausibility for getting an offer. My experience is that executive recruiters are excellent at this filtering as long as you listen to them!
CEO chat(s): make sure the CEO and candidate could work together well, with a focus on the candidate’s understanding of the opportunity and the core challenges. Don’t lean out of the challenges: the best candidates know the challenges exist and will be skeptical if you try to avoid or downplay them. Have as many of these as necessary to build conviction that the candidate is plausible and engaged.
2-3 interviews with executive peers: have two to three executive peers interview. These interviews should explicitly cover the topics discussed in the next section, and should have a documented rubric for assessing candidates. A written rubric will particularly reduce the risk of false negatives and false positives, which unstructured executive interviews often introduce.
30 minute presentation with a 30 minute Q&A: a short presentation given by the candidate that’s focused on their understanding of what they’d need to do in their new role is an excellent way to assess whether the candidate is listening throughout the process, and whether they have the executive acumen to operate within your organization. Avoid the temptation to expand attendees, and instead reuse the executives and CEO who have already met the candidate.

Introducing more attendees will randomize evaluation rather than improve evaluation. For example, bringing in the Chief Marketing Officer (CMO) for the first time will often cause the CMO to observe that the presentation missed several key marketing needs, which is somewhat expected if the candidate hasn’t met anyone yet from Marketing. That’s a randomizing signal. If the CMO is indeed a key stakeholder then they should be one of the executive peers included in the previous step rather than added to the presentation.
Perform rigorous backchannel references: find three to four individuals who have worked directly with the individual in question for an extended period of time. I’m generally ambivalent about backchannels, but in the case of executives I believe the risk of hiring a poor executive is high enough that it’s an essential step. Candidate supplied references are not very high signal at this level, because the candidate will prepare their reference with talking points, including how to answer questions around gaps.
2-3 interviews with members of Engineering: assuming the other steps have gone well, then end with several interviews with engineers and engineering managers. Your primary goal here is to build commitment to the candidate from within the Engineering team, but you also want to listen for any major concerns from Engineering. Your interviewers should be running a structured interview with explicit areas of evaluation. It’s not ideal, but acceptable, to get some lukewarm signals at this stage, as long as you don’t get any major concerns. It’s very rare to hire any new manager whose team doesn’t have some concerns.

At this point in the process, either go to offer or decide not to extend an offer. Resist the temptation to hedge. Delaying sends candidates a bad message, and there’s rarely additional information out there that will change your mind for the better.

Four areas of evaluation

There are an unlimited number of skills to assess executives on, and ultimately there are more skills than you can viably assess. I recommend drilling in on these areas:

Executive skills: are they an effective listener and communicator? Do they have the fundamental skills expected of an executive at this level, such as operating to a financial plan, supporting a single or multi-business unit organization, running a hiring or performance process, etc? You’ll get a signal on this from the presentation, sessions with peer executives, and from backchannel references.
Role and company specific skills: every executive role you’re hiring for is aimed to solve a handful of specific problems at your company, and you should assess on those dimensions. In some cases this is improving partnership between Sales and Engineering, in other cases it’s improving Engineering velocity, and in others it’s partnering more effectively with peer executives. Identify whatever these are, and ensure that you explicitly cover them in either the CEO or peer executive sessions.
Engineering functional expertise: depending on how you’ve scoped your engineering executive role, you’re going to want some sort of functional expertise. In some companies this is deep on running a scaled organization, guiding new product development, partnering between Engineering and commercial functions, technical architecture, or even infrastructure. Whatever it is for the role you’re hiring for, you should ensure that either the engineering interviews or the peer executive interviews cover these points.
Historical performance and behavior: use your backchannel references to get an accurate understanding of the candidate’s true performance and behavior over time. There are effective executives who leave a trail of angry peers behind them. Similarly, there are very ineffective but beloved executives who remain far too long at companies that they serve poorly. You can assess self-awareness by asking about these directly, but you can only assess actual performance by talking to folks who were there. Good executives can spin even the worst performance into something positive, which means you simply cannot rely on them to self-evaluate.

You’ll note that I’ve not provided a checklist of skills to evaluate against. This is deliberate, because the role of a strong engineering executive is exceptionally broad, and reducing it to a list of skills will distract you from evaluating what is particularly valuable to your search. Most current CTOs and VPs of Engineering out there are the wrong fit for your role at your company. Evaluate on the specifics, not the universal.

Summary

You now know how to avoid the unicorn search, avoid bringing too many interviewers into the process, and how to evaluate the particulars of what you need rather than anchoring too heavily on vibes. Even with all of that in mind, these are still difficult searches. Don’t get discouraged if it takes you five or six candidates before you find someone you’re excited about, this is a natural part of learning how to hire a new executive role. Conversely, do get worried if you’re not excited after talking to ten-plus candidates; that probably means your search is going a bit off the rails.

Poking around OpenAI.

Wed, 12 Apr 2023 15:00:00 -0600

I haven’t spent much time playing around with the latest LLMs, and decided to spend some time doing so. I was particularly curious about the usecase of using embeddings to supplement user prompts with additional, relevant data (e.g. supply the current status of their recent tickets into the prompt where they might inquire about progress on said tickets). This usecase is interesting because it’s very attainable for existing companies and products to take advantage of, and I imagine it’s roughly how e.g. Stripe’s GPT4 integration with their documentation works.

To play around with that, I created a script that converts all of my writing into embeddings, tokenizes the user-supplied prompt to identify relevant sections of my content to inject into an expanded prompt, and sent that expanded prompt to OpenAI AI’s API.

You can see the code on Github, and read my notes on this project below.

References

This exploration is inspired by the recent work by Eugene Yan and Simon Willison. I owe particular thanks to Eugene Yan for his suggestions to improve the quality of the responses.

The code I’m sharing below is scrapped together from a number of sources:

I found none of the examples quite worked as documented, but ultimately I was able to get them working with some poking around, relearning Pandas, and so on.

Project

My project was to make the OpenAI API answer questions with awareness of all of my personal writing from this blog, StaffEng and Infrastructure Engineering. Specifically this means creating embeddings from Hugo blog posts in Markdown to use with OpenAI.

You can read the code on Github. I’ve done absolutely nothing to make it easy to read, but it is a complete example, and you could use it with your own writing by changing Line 112 to point at your blog’s content directories. (Oh, and changing the prompts on Line 260.

You can see a screenshot of what this looks like below.

This project is pretty neat, in the sense that it works. It did take me a bit longer than expected, probably about three hours to get it working given some interruptions, mostly because the documentation’s examples were all subtly broken or didn’t actually connect together into working code. After it was working, I inevitably spent a few more hours fiddling around as well. My repo is terrible code, but is a full working code if anyone else had similar issues getting the question answering using embeddings stuff working!

The other comment on this project is that I don’t really view this as a particularly effective solution to the problem I wanted to solve, as it’s performing a fairly basic k-means algorithm to match tokenized versions of my blog posts against the query, and then injecting the best matches into the GPT query as context. Going into this, I expected, I dunno, something more sophisticated than this. It’s a very reasonable solution, and a cost efficient solution because it avoids any model (re)training, but feels a bit more basic than I imagined.

Also worth noting, the total cost to developing this app and running it a few dozen times: $0.50.

Thoughts

This was a fun project, in part because it was a detour away from what I’ve spent most of my time on the last few months, which is writing my next book. Writing and editing a book is very valuable work, but it lacks the freeform joy of hacking around a small project with zero users. Without overthinking or overstructuring things too much, here are some bullet points thoughts about this project and expansion of AI in the industry at large:

As someone who’s been working in the industry for a while now, it’s easy to get jaded about new things. My first reaction to the recent AI hype is very similar to my first reaction to the crypto hype: we’ve seen hype before, and initial hype is rarely correlated with long-term impact on the industry or on society. In other words, I wasn’t convinced.
Conversely, I think part of long-term engineering leadership is remaining open to new things. The industry has radically changed from twenty years ago, with mobile development as the most obvious proof point. Most things won’t change the industry much, but some things will completely transform it, and we owe cautious interest to these potentially transformational projects.
My personal bet is that the new AI wave is moderately transformative but not massively so. Expanding on my thinking a bit, LLMs are showing significant promise at mediocre solutions to very general problems. A very common, often unstated, Silicon Valley model is to hire engineers, pretend the engineers are solving a problem, hire a huge number of non-engineers to actually solve the problem “until the technology automates it”, grow the business rapidly, and hope automation solves the margins in some later year. LLM adoption should be a valuable tool in improving margins in this kind of business, which in theory should enable new businesses to be created by improving the potential margin. However, we’ve been in a decade of zero-interest-rate policy which has meant that current-year margins haven’t mattered much to folks, which implies that most of these ideas that should be enabled by improved margins should have already been attempted in the preceeding margin-agnostic decade. This means that LLMs will make those businesses better, but the businesses themselves should have already been tried, and many of them have failed ultimately due to market size preventing required returns moreso than margin of operating their large internal teams to mask over missing margin-enhancing technology.
If you ignore the margin-enhancement opporunties represented by LLMs, which I’ve argued shouldn’t generate new business ideas but improve existing business ideas already tried over the last decade, then it’s interesting to ponder what the sweet spot is for these tools. My take is that they’re very good at supporting domain experts, where the potential damaged caused by inaccuracies is constrained, e.g. Github Copilot is a very plausible way to empower a proficient programmer, and a very risky way to train a novice in a setting where the code has access to sensitive resources or data. However, to the extent that we’re pushing experts from authors to editors, I’m not sure that’s an actual speed improvement for our current generation of experts, who already have mastery in authorship and (often) a lesser skill in editing. Maybe there is a new generation of experts who are exceptional editors first, and authors second, which these tools will foster. If that’s true, then likely the current generation of leaders is unable to assess these tools appropriately, but… I think that most folks make this argument about most new technologies, and it’s only true sometimes. (Again, crypto is a clear example of something that has not overtaken existing technologies in the real world with significant regulatory overhead.)

Anyway, it was a fun project, and I have a much better intuitive sense of what’s possible in this space after spending some time here, which was my goal. I’ll remain very curious to see what comes together here as the timeline progresses.

How to plan as an engineering executive.

Mon, 10 Apr 2023 06:00:00 -0600

Some years back, I interviewed a senior leader for an engineering role, and asked them a question about planning. I enjoyed their response, “Ah yes, the ‘P’ word, planning.” That answer captured an oft heard perspective that planning is some sort of business curse word. Even when it goes well, planning is an objectively difficult task. When it goes poorly, the business loses months or years of potential progress. Despite those challenges, guiding your company’s plans towards the right work is uniquely impactful executive work, and is an area where effective executives can greatly distinguish themselves.

In this essay, we’ll walk through:

Approaching planning as an infinite process rather than a finite one
Discussing the default planning process at most companies
Decomposing planning into three discrete components: financial plan, functional portfolio allocation, and roadmap
Setting the company’s annual financial plan
Defining Engineering’s functional portfolio allocation
Agreeing on the 3 to 12 month roadmap
Establishing a shared timeline across these three processes
Exploring planning’s frequent failure modes

By the end, you’ll be prepared to operate effectively within an existing planning process, balance functional and cross-functional demands, and diagnose how your current planning process may be holding Engineering and your company back.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Planning is an infinite game

James Carse’s Finite and Infinite Games proposes that you can view most things in life from two different perspectives. The first is seeing life as a series of finite games, with clear rules, a way to win, and many ways to lose. The second is seeing life as an infinite game, whose rules are evolved over time by its players, and where the goal is continuing to play. I’ve found this simple distinction life changing in general, and particularly applicable to planning.

Although planning is often presented as a finite, rules heavy process, I believe that planning is actually an infinite, ongoing game with dynamic rules. Before my realization, I would rigidly follow each planning rule, and then get frustrated when the resulting plan wasn’t very good. Worse, the plans I spent so much time on would routinely get thrown away a month or two after the process finished. It was only when I was able to look past the stated rules that I was able to become an effective planner.

The default planning process

By the time companies reach a couple hundred employees, they mostly reach the same documented planning process:

Every year, the executive team agrees on an annual financial plan, including a headcount plan
Every quarter (or half) the executive team will create a planning artifact that describes the plan for the upcoming quarter, e.g. Objectives and Key Results (OKRs) for each team, target business outcomes, or a list of projects to deliver
Teams are responsible for managing their own execution against the quarter or half plan, using the process of their choice, e.g. Scrum, Agile
There may be a monthly execution review to track progress against company goals, e.g. a Business Review

This process isn’t entirely uniform across companies, things do vary a bit in the details. There are company-specific details about when the process happens, what document you need to deliver to who, how budgeting works and so on. What will be particularly different across companies is how planning actually works. While the executive team will nominally follow the planning pattern above, there is always another layer of emergent rules based on their behavior. For example, some companies have a stated financial planning process where the executive team creates a shared annual financial plan, but in practice individual executives privately advocate to the CEO to increase their function’s headcount. If the CEO rewards those private escalations, then that becomes the real process, and the jointly created financial plan will become a polite fiction.

Even in companies where the executive team works together in good faith, new information will emerge that invalidates the current plan: \

A country where you do business might pass an unexpected privacy regulation
A financial partner might go out of business
You might settle an accessibility lawsuit with a discrete timeline to make improvements

From the finite game perspective, updating your plan to address these new concerns is a failure, but from the perspective of operating a well-run business, you obviously want to operate against a plan that addresses today’s reality rather than last month’s.

Planning’s three discrete components

Without shared goals for the planning process, planning often becomes a bloated, overloaded system that poorly solves many problems. You will encounter planning processes that spend the majority of their time on whether you’ve completed security reviews, ensured you’re staffing cross-functional projects, or verified that headcount planning is aligned with recruiting capacity. Those are all valuable, but they’re solvable outside of the planning process, and distract from where planning can be most valuable.

Your planning process will be effective to the extent that it can do three things well:

Set your company’s resource allocation across functions, as documented in an annual financial plan. This plan sets targets for revenue and expenses, broken down by function and business line. From this you can answer questions like the relative investment into Research & Development (R&D) versus Sales & Marketing (S&M) or General & Administrative (G&A), as well as between the company’s products or business units.
Refresh your Engineering strategy’s resource allocation, with a particular focus on Engineering’s functional portfolio allocation between functional priorities and business priorities. There’s some nuance here because there’s little consistency in what companies consider in Engineering’s functional scope, e.g. Security might or might not be included.
Partner with your closest cross-functional partners, particularly Product and Sales (or whatever the leading Go-To-Market function is within your company), to establish a high-level quarter or half roadmap. This is the cross-functional agreement on the scope and timing of work to be done.

Planning in these three distinct phases reduces the number of dependencies in each step. Fewer dependencies lead to a simpler, more focused process. Although you might assume the opposite, I’ve found that artificially unbundling planning this way increases decision quality. For example, the financial plan focuses on revenue, expenses and headcount, while holding the roadmap and functional portfolio allocation constant. This means that you can focus on getting the financial plan reasonably correct without getting in a debate about the specific product releases. Once you start down the path of juggling the financial plan’s details with the product releases and the mix of functional and cross-functional priorities, things get more complex but are rarely more accurate.

These constraints create better decisions both because constraints drive innovation, and because removing constraints drives overly simplistic magical thinking such as spreadsheets that show tripling Engineering this quarter will triple Product velocity. Planning with sequenced constraints feels unnatural at first, but it’s worth the initial discomfort.

I’m not arguing that you should never change your financial plan or never change your roadmap outside of this sequence. If you complete your planning roadmap and still believe that a headcount adjustment is necessary, it’s reasonable to have that discussion. However, I am a strong believer in sequencing these discussions rather than having them in tandem to allow more focused, rigorous decisions.

Establishing your financial plan

In your non-executive career as an engineer at a private company, you may never see your company’s financial plan. If you do see it, you may only get a brief flash without the time to dig in and understand what the numbers mean. That makes it easy to get surprised in executive roles when you finally realize that your company’s financial plan is the foundation of all company planning.

The below table shows an example financial plan, with targets for every business line’s revenue, and each function’s expenses within that business line. These are dense documents that contain a great deal of data.

For example, this chart shows the Consumer business line is growing 31% year-over-year, with only $133m in expenses on $625m of revenue. Conversely, the Enterprise business line is growing much faster at 95% year-over-year growth, but has $96m in expenses for only $52m of revenue. With only those two rows, you can have a very interesting set of discussions about the two business lines.

Your executive team will need to pull together an updated financial plan every year, which comes down to generating three specific documents:

Your profit and loss statement showing revenue and costs, broken down by business line and function.
Your budget showing more detailed expenses by function, including vendors, contractors, and headcount.
Your headcount plan showing the specific roles in the organization, with an emphasis on the new roles to be hired.

With these three documents, there are enough constraints to begin the rest of your planning process. That’s not to say that these documents are easy to pull together, they can be rather difficult.

The good news, from an Engineering perspective, is that you’re a stakeholder in creating the financial plan rather than directly responsible for it. Every Finance leader will have their own variation on the details, but expect the executive team and business line leaders to iterate on a series of proposals until you finish with something that no one’s quite happy with but seems plausible.

This process is significantly different depending on your company’s stage (e.g. Series B versus Series G), and whether you’re public or private. Look to peer companies for their processes rather than emulating much larger companies you’ve worked at previously. But keep in mind that these plans are inherently low accuracy because they depend on projecting financial outcomes without having a defined product roadmap or knowing what chaos the year may bring. Push a bit on the details, but don’t exhaust yourself.

Reasoning about Engineering’s role in the financial plan

Switching from the executive team perspective to the Engineering executive perspective, the problem gets a bit simpler. Engineering is rarely directly accountable for revenue (although almost always indirectly accountable to either Product or Sales for that revenue), so your primary contribution to the financial plan is managing Engineering’s expenses.

I recommend segmenting Engineering expenses by both business line and three specific buckets:

Headcount expenses within Engineering
Production operating costs (most cloud costs, vendor costs related to production, etc)
Development costs (vendor and hosting costs related to CI/CD, running test, development environments, etc).

Within those segments, you should spend minimal time on business lines where revenue is accelerating faster than expenses (the business’ quality is already improving), and a great deal of time on all other business lines.

For any business line where expenses accelerate faster than revenue, you and the entire executive team should have a clear, documented hypothesis for when and how you’ll reverse the setup. It’s not always a problem, e.g. it’s expected for new business lines to spend a while in that phase, but it’s essential that the overall executive team is marching to the same orders on each given business line.

Why should financial planning be an annual process?

Your company’s financial plan is the foundational constraint for every function. Most companies adjust their plan over the course of the year, which is appropriate, but I’d argue that you’ll be more effective as an executive team and as an Engineering function if you operate as if the plan is fixed.

There are three core reasons for this:

Adjusting your financial plan too frequently makes it impossible to grade execution, because your targets will keep moving.
Making significant adjustments to your financial plan is a planning intensive activity that requires a great deal of time across many functions. As such, changing the plan creates a great deal of churn, and often requires the reworking of other planning components.
Like all good constraints, if you make the plan durable, then it will focus teams on executing effectively within it. If you make it flexible, then teams will instead focus on moving the constraint (e.g. asking for more headcount). The former is much preferable to the latter.

Certainly, if you must, then you must, but it’s much easier to run an effective company with a stable financial plan.

Attributing costs to business units

Early companies have a single business, which makes things simple. All Engineering costs are tied directly to running that one business. As companies grow, they’ll eventually expand to multiple lines of business, where things get a bit trickier.

Even the simplest attributions get messy as you dig in. For example, consider a company like Figma which has one core product, but likely segments its business into two business units: Enterprise and everything else. The core product is the same in both cases, but many Enterprise features aren’t valuable to non-Enterprise buyers. In that case, it’s easy to attribute Enterprise-focused Product engineers to the Enterprise business unit, but less clear how you should attribute Product engineers building the core product. Attribute according to revenue? Attribute all costs to the non-Enterprise segment and show artificially good margins in Enterprise? Do something else?

The answer to all of these questions is always, “It depends.” My experience is that there’s relatively little precision on this topic. Focus on finding a defensible methodology that Finance is comfortable with, and try to avoid reopening the dispute too frequently.

Why can financial planning be so contentious?

Financial planning is not inherently contentious in well-run management teams. However, when executives are focused on their function rather than the executive team’s shared success, allocating expenses is a zero sum game. Combined with the tendency for struggling executives to point at insufficient budget as the rationale behind their underperformance, it’s easy to end up with either broken executive relationships or out of control spending.

If your financial planning process is contentious, I’d recommend pushing a bit on the topic with your CEO. It’s most likely a sign of an executive team struggling to partner, or a particular struggling executive, rather than an issue that you can solve directly.

Should Engineering headcount growth limit company headcount growth?

In general, I do believe that most companies should constrain overall headcount growth on their growth rate for Engineering growth. This creates accountability to operate an efficient company, even when it’s painful to do so. It also avoids the difficult scenario where other organizations scale more rapidly than Engineering, and end up starving Engineering’s bandwidth to make progress on the company’s highest priorities.

There are, of course, always exceptions in the details. Uber did a good job of rapidly scaling city-specific operational teams with flexible tools that empowered the city-teams without letting them overload Engineering with requests. Uber may well have lost significant market share during their chaotic stretch of hypergrowth if they had constrained operational growth on Engineering headcount.

Informing organizational structure

Whenever you request additional headcount, you should have a documented organizational design to incorporate that headcount. The easiest way to do this is using some rough organizational math:

Divide your total headcount into teams of eight. Each of these should have a manager and a mission
Group those teams into clusters of four to six. Each cluster should have an experienced manager and a focus area (e.g. Consumer Products, Enterprise Products, or Infrastructure)
Continue recursively grouping until you get down to five to seven groups, which will be your direct reports. In a company of ~40 engineers, you only need to form one tier of groups, but at company of ~200 engineers will require multiple tiers

Although this will feel a bit superficial, I wouldn’t recommend thinking about this in more detail during the financial planning process. There’s a final phase of real organizational design that has to account for the strengths and experiences of the individuals at hand, but that degree of specificity isn’t helpful during the financial planning process.

Aligning hiring plan and recruiting bandwidth

The last financial planning topic I’ll mention is that it’s common for executive teams and leaders at the next layer to spend time arguing about headcount that’s unlikely to get filled. I find it extremely helpful to compare historical recruiting capacity against the current hiring plan. If it’s unlikely that you’ll be able to make the planned hires based on the historical numbers, then I wouldn’t spend too much time arguing about it.

This is particularly common in hyper growth companies, where the executive team is not overly focused on R&D expenses, and may greenlight most headcount requests. In practice that scenario leads to teams within R&D acquiring headcount by capturing recruiter bandwidth (e.g. getting specific recruiters assigned to their team’s hiring pipeline) rather than through headcount approval. If you do find yourself in this situation, my recommendation is to align yourself with recruiting by being a strong hiring partner: recruiters are graded on their hiring outcomes, and everyone likes being successful.

Determining your functional portfolio allocation

One of the three core components of your Engineering strategy is how you allocate Engineering resources against your current priorities. This functional portfolio allocation applies to your entire Engineering budget across vendors, contractors, and full-time headcount. This allocation directly impacts planning.

The fundamental question to answer in your functional allocation is how much Engineering capacity do you want to focus on stakeholder requests versus investing into internal priorities each month over the next year? This is just a series of percentages, e.g. 63% of Engineering time in June will be focused on cross-functional projects, 75% in July, and 60% in August. However, deciding on these numbers can be difficult, and they have a significant impact on the company roadmapping.

The following diagram shows one potential answer to that question, showing a fixed Infrastructure investment, expanding Developer Experience investment, and a temporary inwards shift in Product Engineering.

Despite being a simple percentage, determining the correct percentage allocation is inevitably a bit trickier. The approach I recommend is:

Once a year or so, review your full set of Engineering investments, their impact, and the potential investments that you haven’t made yet. Build the list of potential impacts from broad sources: your developer productivity survey, explicit brainstorming, and so on.
Update this list on a real-time basis as work completes and new ideas are suggested.
As the list is updated, revise your target steady-state allocation to functional priorities. Concretely, this is your investment into Platform and Infrastructure Engineering teams who exclusively work on functional priorities.
Aim to solve all functional priorities within your steady-state allocation, but be willing to discuss whether teams that generally do cross-functional work, e.g. Product Engineering, should temporarily assist for particularly high impact or context-dependent projects.
Invest leadership time, including your own, into spot fixing the large allocations which are not obviously returning much impact. These are the places you can learn the most about your organization, and also make the most meaningful adjustments.

If you’re debating how much energy to invest into this process, remember that it doesn’t need to be perfect, it needs to be useful. If your cross-functional partners are happy and engineers are happy, then you can get away with something very lightweight.

Why do we need a functional portfolio allocation?

If setting the company’s budget and roadmapping is a joint exercise, it’s worth asking why setting the functional portfolio allocation isn’t also done by the executive team as a whole. The simple answer is that functional planning is best done by the responsible executive and team. Functional allocation relies so deeply on function-specific expertise that doing it in a larger, cross-functional group leads to worse outcomes.

That’s not to say that functional leaders should allocate in isolation. I would strongly recommend doing it in partnership with your Engineering leadership team and other senior members of Engineering as described in Writing an engineering strategy, and then verifying your proposal with peer executives. However, including peer executives too deeply is unlikely to help them or you: something has gone very wrong if you’re trying to prioritize your CFO’s opinion about monorepos over your Staff Engineers’ perspectives.

Over time, address the root of this problem by slimming your allocation to functional priorities by converting them into cross-functional priorities. Things like compliance, security, reliability and so-on should really be fundamental company work, not invisible Engineering functional work. Bringing Engineering work out of functional allocation is much more effective than trying to bring non-Engineering executives in.

Keep the allocation fairly steady

The most common allocation challenge for executives is adjusting their allocation too frequently. You should prefer continuity and narrow changes over continually pursuing a theoretically ideal allocation. Changing your allocation feels like progress, but each change is fairly disruptive, so there’s a significant penalty to making frequent changes. As such, I recommend a strong bias towards the resource allocation status quo, even when your current allocation is slightly suboptimal.

Another reason to make infrequent allocation changes is that teams can become overly focused on competing for allocation, which results in a zero-sum competition with their peers. That’s a bad place to be culturally, and distracts from the more valuable opportunity of creative problem solving, which has potentially infinite returns without cross-team competition.

Be mindful of allocation granularity

There’s an inherent tradeoff between allocating at large (e.g. Infrastructure and Product) and small (e.g. Frontend Platform team and North America Growth team) granularities. The larger the granularity that you allocate, the more you empower your team to lead their teams. For example, if you allocate to Infrastructure Engineering, then Infrastructure’s leader is empowered to make shifts within Infrastructure. If you make explicit allocations to teams within Infrastructure, then Infrastructure’s leader would be obligated to work with you on changing their own allocations.

Don’t over index on early results

The most frequent concrete error I see in Engineering portfolio allocation is overestimating impact, or underestimating cost, based on early results. A few illustrative examples:

Two engineers work on improving build performance, and speed up build times by 50% in two months. They argue that this shows you should theoretically invest 50% of Engineering capacity into developer productivity, and if not 50%, you should at least triple the size of the team. It’s easy to invest more here, but they may (or may not) have already captured the early returns, meaning future results will be much lower,
Two engineers are working to migrate all frontend code into a shared frontend monorepo. After three months of work, you have a proof of concept, four annoyed frontend engineering teams, and no metrics showing an improvement. It’s easy to cancel this project, but it may (or may not) be about to consolidate these gains, meaning future results will be much higher,

To prevent these mistakes, I recommend establishing a fixed investment into projects until there is at least one inflection in their impact curve. For example, keep the two engineers working on build performance until you see their impact shift downwards, and only then estimate the correct long-term level of investment. If this feels too slow, then spend some time designing projects to show inflection earlier.

Agreeing on the roadmap

There are many areas where I’ve found a definitive solution that I strongly believe in. Roadmapping is not one of them. You can get there with a four-column spreadsheet (e.g. project, opportunity, implementation cost, and the owning team). There are innumerable other ways to create a roadmap that can work as well, and it’s rarely the format that causes roadmapping to fail. I recommend using whatever is already being proposed rather than spending time arguing about the format.

Instead, roadmapping fails because of

coupling roadmapping with changes in budget or functional portfolio allocation
tension between Product, Engineering, and their stakeholders
roadmapping a mix of concrete and unscoped projects
disempowering teams by planning too deeply

We’ve already spent time on the coupling problem, so let’s dig a bit into the others.

Disconnected planners

Effective roadmapping requires Product, Engineering and Design work together collaboratively. While one function often leads planning, the other two should be active partners. Whenever that isn’t the case, you will generate a roadmap that seems reasonable but fails to account for the real underlying topography of the planned work.

Similarly, I’ve often seen roadmaps where Product, Engineering and Design are closely aligned with each other, but the proposed work doesn’t account for stakeholder requests (generally from Sales or Marketing). In these cases, a properly scoped roadmap will come together, but it won’t generally be a set of work that optimally solves the company’s challenges.

You can quickly determine if this is happening by checking with folks in other functions to see if they generally agree with the proposed plans. If not, pull the different functional planners into a room and hash it out. The frequent mistake I see here is executives trying to solve this through a series of one-on-one discussions. Debugging the situation one-on-one works well, but resolving it requires an open, group discussion. I’ve had particularly good success with structured group discussions where each person has one to two minutes to share their perspective, without interruption, before the group starts the discussion.

Roadmapping unscoped work

In Kellan Elliot-McCrea’s excellent post on How to plan?, he makes an important observation that planning processes often proactively invite new ideas,

The implicit or explicit ask of Planning is often, “Tell us about all your exciting new ideas. We need your creativity to achieve our goals. Swing for the fences! Throw big rocks!” This is a mistake.

The challenge is that new ideas are unscoped and unproven, and you end up comparing the raw potential of an unscoped idea against the proven potential of an existing idea. How you discount the unproven idea is usually a reflection of your executive team’s psychology rather than its actual potential, which makes planning feel capricious. It also means, in the case that new ideas are quickly disproven, that your planning process will generate short-lived plans.

Two effective techniques for avoiding this scenario are:

Agree on an allocation between scoped and unscoped initiatives, and then prioritize like-against-like within those allocations. For example, you might say that 20% of product engineering time should go against validating new projects. Then you can debate which unvalidated projects should staff that 20%, followed by a separate discussion around investing the 80% of time in validated projects
Have a long-running allocation towards validating projects, say 10% of product engineering time, which provides enough validation that you can reasonably prioritize those projects against existing projects

Even if you’re not able to formally adopt either of these approaches, you can often informally steer the discussion towards separately discussing scoped and unscoped projects.

Roadmapping in too much detail

The final roadmapping challenge I’ll note is that if you get too specific, you can accidentally disempower the team responsible for the actual work. Melissa Perri captures this idea eloquently in Escaping the Build Trap, which describes how roadmapping focused too narrowly on project to do rather than your desired outcomes can constrain teams’ thinking.

Sometimes new leaders will take this concern a bit too literally, and argue that executives should not be allowed to discuss specific projects, because it strips their team’s autonomy. That’s a bad outcome for everyone, because it turns the executive team into a pure resource allocation decider, which is a bit too abstract a way to run a business until it gets exceptionally large. (As much as teams hate an executive team that micromanages, they’ll hate an executive who only does resource allocation even more.)

Instead, the goal should be to emphasize the target outcome first at a high level of confidence, and discuss the specific project second with a lower degree of confidence. As long as they remain aligned with the target outcome, the team should feel empowered to change the project, but digging into the specific project will surface a much richer discussion around execution than any abstract discussion about goals.

Timeline for planning processes

Mapping the above planning process to a calendar usually looks like:

Annual budget should be prepared at the end of the prior year
Functional planning should occur on a rolling basis throughout the year
Quarterly planning should occur in the several weeks preceding each quarter. If you’re in halves instead of quarters, you’d prepare in the weeks preceding each half

This typical implementation is shown in the following figure.

The particular details here will reasonably vary across companies, and I have not found the details to matter too much one way or another. The biggest thing I recommend remaining vigilant against is something that Claire Hughes Johnson would often remark about when we began planning at Stripe: planning efforts always expand to fill the allocated time. If you set aside one week for planning, it will take one week. If you schedule one month for planning, it will take one month. Because planning is an infinite game, it’s best to avoid spending too much time on any given finite step.

Running a shadow planning process

At some point in time, you may find yourself in a planning process run by someone who is unwilling to independently solve for the budget, functional priorities and roadmaps. Similarly, you may find yourself working with a planning process that insists on solving every company problem, despite the fact that it’s clear that it’s solving all those problems poorly.

As an executive, you should work with the owner to improve the planning process, but you may also need to run a shadow planning process within your function. This is straightforward to do for both the budget and the functional resource allocation. For example, you can usually establish a conservative annual headcount plan for Engineering and run Engineering towards that headcount plan, even if other functions jockey for more headcount over the course of the year.

On the other hand, it’s pretty well impossible to run a shadow roadmapping process, even if the roadmapping process is difficult to work with. I’ve never seen a shadow roadmapping process work well except in the case of teams with exclusively Engineering customers such as Platform Engineering, Developer Productivity, or Infrastructure Engineering.

Ways that planning processes fail

Even talented executive teams often run planning processes that go awry. The causes are a mix of incompatible goals, subtle defections from team goals to individual goals, and a long list of small, tactical errors with outsized consequences. To help you in diagnosing your planning challenges, I’ve pulled together the most frequent planning challenges I’ve encountered as well as the symptoms that you can use to identify which challenge is impacting your planning process.

Planning as ticking checkboxes

Signs that you’ve delegated planning to someone who is process oriented rather than outcome oriented, or someone who is unable to push back on others demanding process oriented changes:

Planning is a ritual rather than part of doing work: teams put together dozens of planning artifacts to support the planning process, and the executives celebrate the depth of work necessary. After planning ends, the plans are rarely if ever referenced again until the next planning cycle begins. This incentives individuals to focus on optically valuable work rather than genuinely valuable work.
Planning is focused on format rather than quality: executives feel obligated to provide input on plans, to show that they value the planning process. If they don’t have valuable feedback, they will still find feedback, often related to correctly following the planning documents’ format or planning process. This distracts from assessing the quality of the planning decisions themselves

An effective planning process must have an empowered executive sponsor who can keep it focused on its core goals, as well as a planning implementor who is at least as passionate about generating useful plans as running a clear process.

Planning as inefficient resource allocator

Signs that your executive team is optimizing for their function rather than working together to optimize for the company’s overall outcomes:

Planning creates a budget then ignores it: many executive teams will establish a resource allocation across functions in their budgeting process, but will then make headcount requests that violate that allocation. This leads to requests that are assessed by secondary factors, often individual’s interests or demands, rather than by their alignment with company goals
Planning rewards least efficient organizations: often leaders will sandbag their organization’s capabilities, arguing that they need significantly more headcount to even continue operating at their current level. This culminates in a business where most resources are directed towards your least efficient leaders and organizations.

A variant of this pattern is one where executives imply they can’t be accountable for their function’s output unless their full headcount request is met. This, combined with consistently high headcount requests, results in a toxic cycle of executives missing their obligations and claiming they can’t be held accountable
Planning treats headcount as a universal cure: in headcount oriented executive teams, it’s easy for planning to become focused on rationalizing headcount requests rather than deciding on the most important work to be done. It’s clear this is happening when prioritization and planning discussions assume velocity is fixed and that work must happen, and instead anchor on headcount. This penalizes creative leaders who understand how their function works whose knowledge of how the function operations makes them appear less strategic than headcount oriented peers

An executive team can create social pressure on its members to optimize for the greater good, but ultimately only the CEO can hold executives accountable for their behavior. It is valuable to politely raise these issues, but only the CEO can address them. Pushing hard to resolve them will only backfire.

Planning as rewarding shiny projects

Signs that your planning process is more focused on energizing your executive team than solving the inherent resource allocation and functional alignment problem at hand:

**Planning is anchored on work the executive team finds most interesting:**certain projects will always be more energizing for any given executive team to discuss. For example, most executive teams are excited to discuss sales numbers or new product development, but fewer are thrilled to discuss compliance. If planning processes are driven by work that executives find fun, then they will frequently lead to poor planning outcomes
Planning only accounts for cross-functional requests: many planning processes are focused on meeting cross-functional commitments rather than planning the entirety of work to be done. It’s reasonable for roadmapping to only deeply consider cross-functional requests, but it’s essential that the functional allocation is being planned as well, such that the executive team is able to discuss critical work that happens to be considered a single function’s responsibility (e.g. some companies view customer satisfaction, security, compliance, and stability as the responsibility of a single function)

Creating impactful planning outcomes depends on solving the business and functional problems at hand. Energizing your executive team is important, but distracting planning towards this goal comes with an unreasonably high cost.

Planning as diminishing ownership

Signs that your executive team is approaching planning in a way that is reducing autonomy and ownership within your broader team:

Planning is narrowly focused on project prioritization: an effective planning process serves as guidance for teams within the company to accomplish their work. Executive teams sometimes focus too narrowly on prioritizing specific projects rather than on the necessary outcomes or areas of investment. This often results in the teams with the most context, those implementing the project themselves, being unable to tune their approach to be more effective. This leads to disengaged teams and executives who are frustrated by the lack of urgency in the company’s execution
Planning generates new projects: in some planning processes, it’s the only time that some executives think about a given area. For example, your Chief Finance Officer may not spend much time thinking about the Customer Success organization’s roadmap outside of planning. This is not inherently a problem, but sometimes those distant executives will use planning as an opportunity to suggest significant new, unscoped work. This will almost always result in a delayed or failed planning process, as it leads to attempting to prioritize well-scoped work against unscoped work, which is very challenging

Executives should prioritize coaching the broader team through planning. At times you will identify a leader who is not able to plan in their area, and at times you may need to make top-down changes to the plan, but these should not be exceptions rather than the routine.

Summary

Working through this post, you’ve dealt with the executive team’s annual budgeting process, maintaining Engineering’s target functional allocation, and combining the two to establish a concrete roadmap. You’ve also worked through a number of smaller topics like whether Engineering growth should be the limiter on your company’s overall growth, considering the granularity within your functional allocation, and the tradeoffs of running a shadow planning process.

As you take these ideas into your next planning process, my biggest hope is that you remember the first section: planning is an infinite game, and there’s always a bit of mess at every stage. The key thing is to continue improving bit by bit!

Who runs Engineering processes?

Mon, 03 Apr 2023 06:00:00 -0600

Uber ran a tech spec review process called the DUCK Review. “DUCK” didn’t stand for anything–it was created as a deliberate non-acronym–but was otherwise a fairly typical review process. When I first joined, we’d review one or two specs each week. The volume of requested reviews kept growing, and six months later there was a one to two week delay between requesting a review and receiving feedback. A year after that the process was disbanded due to lack of bandwidth to process all the incoming specifications. This was an instructive experience for me, because the DUCK Review produced very valuable feedback, but ultimately still failed because its operational costs were higher than we were willing to pay for that feedback.

Early in my management career, I believed most processes failed because they insufficiently addressed the problem at hand. I thought that if a process’ results were poor, it was because the process needed more nuance and sophistication. What I’ve learned since, mostly through designing thoughtful processes that nonetheless failed, is that good process is a deliberate tradeoff between quality and overhead. It’s common to see a code review process that runs quickly but provides low quality feedback, for example requiring code reviews be completed within two working hours of a pull request being made. It’s equally common to see very involved, high-quality process that eventually folks ignore because it’s too slow, including the earlier story of the DUCK Review.

To figure out the appropriate degree of process sophistication for your team, you have to start by figuring out who’s going to run the processes, and reason backwards from that answer to the degree of process sophistication your organization is capable of sustaining. To help you think through that question, I’ll work through:

What is the typical progression for companies across patterns?
What are the pros and cons of these patterns?
How do you operate the baseline pattern successfully?
How do you deal with the budgeting realities?
How should you navigate the trend cycle?

When you’ve finished, you’ll have a clear perspective on who should be running process within your organization as it exists today, when you should consider switching patterns, and some tips on avoiding the most frequent mistakes made in staffing process.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Typical pattern progression

There are five frequent patterns I see organizations use to run their Engineering-scoped and company-scoped processes. Whatever pattern you’re using today, it’s likely you used a different one three years ago, and if you’re growing quickly then it’s likely you will use a third pattern three years from now. There’s no right pattern, just the most appropriate pattern for your current constraints. (There is one wrong pattern, the business unit pattern rarely goes particularly well.)

Early Startup

Brand new companies all use what I call the Early Startup pattern: either a founder or the functional executive does something for their function, or it doesn’t happen. My first year as Calm’s CTO, there were no shared People processes, and I ran Engineering’s performance review process out of a spreadsheet. It needed to get done, and I was in the position to do it. This is how much very small companies work, typically through their first 30-50 hires, at which point they move onto something slightly more structured.

Baseline

The next phase is the Baseline pattern, which is first adopted around ~50 hires, and is the final pattern for most companies. Company-scoped processes (e.g. company planning, performance management, and budgeting) are run by a central People (aka Human Resources) function, and engineering-scoped processes (e.g. Tech Spec Review, Incident Review, and developer productivity surveys) are run by engineers and engineering managers. There are no specialized roles to support engineering processes, and limited specialization of the company’s processes to support engineering.

Specialized Engineering Roles

As companies grow beyond 200 engineers, Engineering-scoped processes often start to take enough time that they benefit from full-time support. This often happens first for either an Incident Review or Tech Spec Review, but depending on your company’s particulars, any process might be the first one to become sufficiently expensive to run. I call this the Specialized Engineering Roles pattern, because most companies address this problem by spinning up a new function, usually either Technical Program Management or Engineering Operations.

Company Embedded Roles

At many technology companies, Engineering is the largest function, and consequently will reach certain points of scale earlier than other parts of the business. Ensuring that processes work properly despite all these extended requirements often leads to Engineering splitting away from the company’s default mechanisms for hiring, performance management, and so on, such that there are dedicated recruiters and members of the People team working with engineering. I call this the Company Embeddes Roles pattern.

In addition to pure size, Engineering has several other factors that create a particularly complex set of requirements, and support the switch to this pattern:

Dual-sided career ladder across engineers and engineering managers, whereas other functions typically have only one
Often includes one or more specialized roles along the lines of Developer Relations, Technical Program Management, Quality Engineering, or Security Engineering
Engineers and engineering managers tend to be slower and more expensive to hire and retain than most other functions
Long ramp time from starting to becoming highly productive, which places a higher value on retention

Business Unit Local

Finally, there is the Business Unit Local pattern, where engineering reports into each business unit’s leadership, and there is no longer any one leader who is responsible for the entirety of engineering. This typically results in engineering moving back to a standardized process run by centralized functions or embracing inconsistency as each business unit’s engineering organization finds its own path forward.

Patterns’ pros and cons

Now that we’ve introduced the typical sequence that companies go through these process management patterns, I want to go a bit deeper into the pros and cons of the patterns themselves:

Early Startup: when you start a new company, it’s just one or more founders. Early hires will almost always focus on building a small product to validate the market or run the business day to day, and generally won’t include anyone focused purely on process. This makes sense, because processes are about coordinating large groups, and the company is only a handful of people at this point. Any process that you want to run needs to be run by the founders, or managers hired by those founders.

Companies usually exit this pattern because there is impactful process they want to run, often initially this is performance management, but they feel they’re too busy to run it with the current team.

Pros: low cost, low overhead, few downsides at small scale beacuse most process is focused on supporting large teams

Cons: often valuable stuff doesn’t happen because everyone is too busy, sometimes quality of process is very low due to limited time or experience running that process
Baseline: Engineering-scoped processes are run by engineers or engineering managers, and company-scoped processes are run by a centralized function, usually in People or Human Resources. Most companies stay in the baseline pattern for an extended period of time, with perhaps one small exception for a dedicated recruiter for engineering hires. It typically works well through 100+ engineers.

One of the reasons this works significantly better than the early startup pattern is that identifies a clear team responsible for selecting vendors (e.g. Greenhouse for hiring needs, Lattice for performance management). These tools are not perfect, but they’re significantly better than a spreadsheet, which is the default tool when seven managers are independently trying to run a decentralized process.

Pros: some modest specialization to allow Engineering to focus on engineering rather than foundational company process, complexity of HR tasks tend to outstrip engineering manager depth by this phase (e.g. managing visas), unified systems allow executive team to inspect across functions rather than just in their own function

Cons: outcomes are highly dependent on quality of these centralized functions, changing company-scoped processes becomes challenging
Specialized Engineering Roles: increase efficiency in Engineering-scoped processes by hiring dedicated, specialized roles to fulfill that work. Typically this is hiring either an Engineering Operations or Technical Program Management team. The first areas of investment are usually organizational planning, incident management, and engineering onboarding.

Pros: specialized roles generally introduce significant efficiency, offering a better skill set for the task to be done at a lower salary point. Introducing these roles is usually energizing for the individuals no longer doing the work. Specialists who are able to focus on a given process often make significant improvements in that process

Cons: While it is more efficient, it’s usually more expensive as these roles introduce additional headcount. It’s a lot of work to effectively support specialized roles like engineering operations or technical program management. Specialized roles often freeze a company in a given way of working, whereas engineers are incentivized to eliminate a process, the specialist is incentivized to improve it
Company Embedded Roles: Engineering’s needs for company-scoped processes reach a sufficient scale or sufficiently diverge from standard operating procedure such that support functions like People and Recruiting hire individuals specifically to support engineering. This usually happens one function at a time, often starting with Recruiting, followed by People and Finance.

While it will often happen first for Engineering, this pattern will repeat for other sizable organizations in the company, for example you’ll frequently see a setup where Engineering and Sales have dedicated embeds, and the rest of the organization work with a centralized team. Within a sufficiently large company, every function will have dedicated support.

Pros: empowers Engineering to customize process and approach. Builds stronger relationships across functions

Cons: expensive to operate. Quality is heavily dependent on the embedded individuals
Business Unit Local: each business unit in a company has its own, separate engineering function. This split typically occurs when an organization becomes large enough to have multiple meaningful business units, and there’s a perception that standardization across engineering in those business units is unnecessary or unnecessarily expensive. For example, one business unit might require HITRUST compliance, whereas the others do not, which might lead to significantly different engineering tradeoffs.

Sometimes this pattern is adopted but one business unit is so much larger than the others, that it becomes the defacto engineering hub for the wider company, including leading cross-organizational developer tooling and infrastructure investments.

Pros: aligns engineering with business priorities within each business unit

Con: engineering process and strategy often gets stuck wherever it was when the split first happened. Future changes require consensus across many engineering leaders. Investments across engineering become difficult to prioritize

All of these patterns have flaws, but are worth considering. That said, if you asked me to pick one without any additional context, I would always recommend the baseline pattern. It’s generally good enough, and the easiest to manage.

Operating the Baseline pattern

What I’ve called the Baseline is the most common approach to running processes, with a centralized People organization running company-scoped processes, and a mix of engineers and engineering managers running the Engineering-scoped processes.

It’s common for a handful of reasons:

It’s the default scenario: simply don’t hire any specialized roles or embed support directly with engineering and you’re at the baseline
It’s relatively simple: you don’t have to learn to hire or manage anything new, and it doesn’t require treating engineering differently from other functions
It appears cheaper: when you propose moving to another pattern, it’s usually framed as a way to free up existing capacity to focus on higher value work, but that requires hiring additional roles to support the pattern, such as Technical Program Managers. That makes it more expensive from a budgeting perspective, even if it’s meaningfully more effective, and most budgeting discussions are cost-driven rather than effectiveness-driven

There’s a fourth reason it’s common: it works pretty well! Every company I’ve worked at, including Uber and Stripe, got very far before leaving this pattern, well past a hundred engineers. I personally believe that many companies switch to other patterns too early, often because they’re copying the playbook from their previous, much larger, employer.

If you’re tempted to switch away from Baseline and are below a hundred engineers, there are a few things I’d recommend focusing on first to try to make the current pattern work for you:

Rotate the work. Calm has had great success with six month rotations, where individuals serve in one capacity (e.g. running Incident Review) for six months before swapping off. We increase continuity by having two people serve together in each capacity with overlapping but offset periods of service
Make the process interesting for the people running it. If someone is leading your Incident Review, also give them exposure to the company’s wider planning process as you factor in reliability projects. If someone is running your Tech Spec Review, pull them into your annual revision of your engineering strategy. You should go beyond service, folks should feel that these are unique learning opportunities to work with senior leadership
Ensure your promotion rubrics, or your promotions if you don’t have a rubric yet, value this kind of work. You can even experiment with the rubric strongly preferring that engineers serve a term in one of your processes before moving into Staff-plus or managerial roles. People are smart and drift towards the work that you value
Make sure to call out the individuals doing the work behind your processes, particularly in All-Hands and Q&A meetings

There are no guarantees, but my lived experience is that making these relatively minor tweaks will allow the Baseline pattern to function well, and keep working much longer than you might expect.

Dealing with budgeting realities

While there are operational challenges to moving across patterns, most of those challenges are only obvious in hindsight. Instead, the biggest impediment for moving from Early Startup to Baseline or from Baseline to Specialized Engineering Roles will be the budget implications. Even if you can show that three Technical Program Managers would significantly improve Engineering impact–which likely is true but you likely can’t prove with a spreadsheet–most budgeting processes are anchored on relative dollars rather than absolute impact.

Digging in, the problems at hand are:

Most companies that successfully shift up the patterns’ cost curve do so when they are rapidly growing. Such companies are comfortable with rapidly growing headcount costs, and are generally tolerant of adding roles to support that growth. In all other circumstances, companies generally aren’t tolerant of adding new roles and structures, even if they would likely increase efficiency. In the latter scenarios, you’re much more likely to hear folks talk about “doing more with less” than “designing our organization to operate efficiently,” and it’s very hard to make efficiency-based arguments with an executive team in a cost reduction mindset
Many companies operate to a portfolio allocation target across departments, e.g. 20% on General & Administrative (G&A), 30% to Sales & Marketing (S&M), and 50% to Research & Development (R&D). Most pattern shifts will eat into the G&A budget, increasing G&A costs and preventing other planned G&A hiring. This generally means that many of your peer executives will be resistant to the proposal
A surprising number of budgeting processes operate in headcount rather than budget, which makes it hard to intuitively express ideas like “replacing two high cost Staff engineers with three mid-level Technical Program Managers.” Headcount-anchored budgeting processes portray the later as more expensive than the former, even though it’s not. It’s possible to push through this confusion, but it imposes an ongoing communication cost

Each of these concerns can be overcome with planning, patience, and conviction, but you have to decide yourself whether it makes sense to do so. My general advice is to stick with Baseline, and avoid moving up the pattern cost curve unless revenue or headcount is growing significantly.

Navigating the trend cycle

There are numerous process trends, for example the industry’s era of Agile obsession: I once worked with someone who’s job title was Agiletect. Similarly, there are also trends in who works on processes, and whether evolving processes is highly impactful, essential work or boring, dull, glue work that won’t get you promoted. In my own career, I’ve seen a major shift from “technical management” to “people-centric management,” and an equally abrupt counter-reaction from “people-centric management” to “management is overhead.”

If I bought into each of these trends, then I’d inflict a constant churn on the teams I lead, and we’d never get particularly good at any given way of working. That’s not to say that trends are meaningless, there’s always something real behind a given trend. In industry downturns, there will be a trend that reduces cash flow. In industry upturns, there will be a trend that takes advantage of copious investor cash. The shift towards people-centric management was in part driven by a generational shift in United States employees, with Millennials becoming a larger part of the workforce. There’s something to learn from these trends, but it’s a mistake to take them at face value: what has been working thus far hasn’t become irrelevant overnight.

My advice is to design your organization around one specific set of these foundational beliefs, and stick solid to those beliefs for at least three to four years before changing them again. Change can be restorative, but if you change too quickly then you’ll never be able to learn what is or isn’t working well enough to inform future changes.

Onboarding peer executives.

Mon, 27 Mar 2023 06:00:00 -0600

While many companies build out an elaborate Engineering onboarding program, the process for onboarding new executives tends to be an ad-hoc, chaotic affair. There usually is an executive onboarding process, but it’s used too infrequently to ever get excellent. Part of the problem is similar to that of an executive job search: every executive and role is unique, and it’s hard to create a repeatable program to handle one-of-a-kind onboardings.

The other part of the problem is that the executive’s manager, the CEO, usually hires a new executive to solve a specific problem. Once the executive starts, the CEO will often turn to focus on their next biggest problem. This happens so frequently that the two most common frustrations I hear from executives about onboarding their peers are “Isn’t this the CEO’s job, why should I worry about it?” followed shortly thereafter by “Why didn’t I worry about it more? I knew leaving it to the CEO was a mistake.”

Even when the CEO is engaged, most new executives want to impress their CEO. You’ll often find new executives struggling a bit with onboarding, while simultaneously reassuring the CEO that they’re doing fine. Your job as a peer executive is to help the new executive succeed, even if no one tells you it’s your job.

I’ll break down onboarding peer executives into five topics:

Why executive onboarding matters
How onboarding executives differs from onboarding engineers
How to share your mental framework
Defining your roles in respect to one another
Investing in time working together on an ongoing basis

Peer onboarding, like a surprising number of good executive practices, is straightforward, and easy to do well with some time and attention. By the end of this piece, you’ll have a clear structure for onboarding a new peer executive.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Why this matters

It’s unfortunately true that you can be perceived as a high achieving executive without accomplishing much, but you cannot genuinely be a high performing executive unless you, your peers, and your CEO are doing excellent work. You can create whatever narrative you want about your extraordinary engineering organization who’s excellence was masked by the weak product or sales teams, but you’ll only be exceptional if every part of the executive team functions well.

Your best tools for forming an effective executive team are working with a team-oriented CEO and being deliberate in the executives you hire. Once you’ve hired someone, your best remaining tool is effectively onboarding them. Without a good onboarding process, even strong executives will make missteps, sometimes unrecoverable ones. I’ve seen Engineering executives start by attempting a full rewrite of the company’s technology stack, others significantly accelerate spend until the company itself is at risk, and still more that replace the existing team with their friends. All of these are much easier to prevent than to recover from.

Onboarding executives vs onboarding engineers

As an engineer, you’ll often be responsible for onboarding peers onto your team. In those cases, you’re often implicitly more senior than the peer being onboarded, but you’re both knowledgeable in the shared field of software engineering. Onboarding executive peers is different. You’re truly peers, without any implicit hierarchy between functions, and you often don’t share the context of having worked in a common profession because they might be the new executive for Finance, Sales, or anyother function outside Engineering. Further, engineers (and most other functional contributors) go deep and narrow, whereas executives need to start by going shallow and broad.

Teasing out the distinction between onboarding engineers and executives:

When onboarding a peer software engineer, your goal is to help them understand your current process and one specific onboarding project well enough to implement that one project. After two or three such projects, the new engineer will be relatively ramped up.

The best indicator in engineering onboarding is completing initial projects quickly at a high quality bar, which is typically most dependent on their ability to navigate a large, lightly documented codebase for the first time. Bad signs are making very slow progress on projects, as well as arguing about existing processes before ramping up on them
When onboarding a peer executive, your goal is to help them understand the company’s landscape (business, team, project, and process), with a particular focus on the most critical, current issues. There is so much to absorb, that they will get a very broad overview, and then go deep as required by the immediate fires they need to put out.

Good executives will immediately focus on the highest priority areas, check their plan with nearby peers before beginning implementation, and bring a positive energy (even if things are an absolute mess). Bad signs are inaction on key issues, changing areas that are already working well before addressing higher priorities, or immediately falling back onto previous companies’ approaches that fit poorly in their new circumstances

George Lakoff’s Don’t Think of an Elephant talks about the power of framing a debate. You can often establish someone’s mental framework with a short phrase like, “Our product changes our users lives.” That frame is challenging to shake, even with data that suggests your users don’t use the product at all. Helping new executives build an accurate mental framework of the business is one of the most powerful acts you can take, and it can only occur in the that executive’s first few weeks on the job.

If you’re onboarding a new peer executive, I recommend covering all of these topics in the first two weeks, even if it means canceling some of your standing meetings to make it possible:

Where can the new executive find real data to inform themselves, rather than relying on existing narratives? The best executives will listen to you, but won’t fully believe anything until they’re able to find data to substantiate your perspective. That’s not because they don’t trust you, but because any seasoned executive has been burned by trusting someone who fervently believed something that ultimately wasn’t true.

Examples: Company dashboards. Board reports. Tableau. Looker. Google Analytics
What are the top two to three problems that they should immediately spend time fixing? (This is traditionally the first question to answer, but I think setting the mental frame is even more important!)

Examples: CEO is very upset with pace of execution. Friction between Product, Engineering and Legal is preventing forward progress on new initiatives. Personal conflict between two senior leaders has frayed cross-functional relationships
What is your advice to them regarding additional budget and headcount requests? The classic new executive move is to privately request more budget from the CEO, who agrees to keep the brand new executive happy, which often creates a cascade of cross-functional hiring that can significantly impact the company’s cash flow. If you don’t want them to immediately request new headcount, you should have an open discussion with them about the consequences.

Examples: Although I suspect the CEO would approve more headcount if you ask today, if you look at Product hiring relative to revenue, we’ve gotten significantly less effective each year over the last three years. I recommend spending time debugging why we’re getting less efficient before further increasing the size of the Product organization
What are areas where many companies struggle, but are currently going well? Specifically you want to steer them away from running their “default playbook” where things are already working.

Examples: Product and Engineering are already planning together effectively. CEO is very pleased with the release cadence (perhaps they are instead frustrated that cadence isn’t translating into revenue). Collaborative headcount process is already reaching reasonable cross-functional equilibrium
What is your honest but optimistic read on their new team? Give them a detailed, transparent look into the members of their team, while keeping in mind that any historical gaps in performance may have been connected to the reason the new executive was hired.

Examples: Two years ago, Design was an active part of planning product work. Then the Head of Design moved from an embedded model to a centralized studio, and the designers generaly lost context on user and business goals. In particular, the impact I’ve seen on the Design Leads has been quite high, going from some of the highest impact folks I’ve worked with to feeling a bit disengaged
Who do they need to spend time with to understand the current state and the company’s implicit power structure? Especially the longest tenured employees who uniquely hold parts of the business in their heads, and individuals who have significant influence over the executive team that wouldn’t be obvious from the reporting hierarchy.

Examples: Head of Quality Assurance or Customer Experience. Long tenured Product Manager. The third founder who isn’t in a managerial role but remains very influential
What is going to surprise them? Particularly things that are done in a unique way at the company, but also quarterly or annual processes whose schedule they might be unaware of.

Examples: Business reviews. Quarterly or annual planning. Written documents for every meeting
What are the key company processes, how long have they been in place, and how are they working? How should they propose changes to any of these processes if they feel it’s necessary?

Examples: Planning. Budget and headcount. Performance management. Cultural surveys

There are, without a doubt, peer executives who will only listen to guidance from the CEO. I’ve been responsible for onboarding multiple executives who fully ignored my advice, and immediately advocated for a full rewrite of our product or technology stack. I felt pretty bad about not influencing them more successfully, but nonetheless glad I tried my best to do so. Although they’re harder to count, I’m confident I’ve prevented many such nightmares by proactively sharing my mental frame with executives before they leap into action.

Define your roles

Effective Engineering organizations start with strong relationships across Product, Engineering and Design, and those relationships are greatly influenced by how well their executive’s work together. Many executives start their new role assuming that it’s more-or-less their old role but in a new company. This leads to implicit assumptions about who does what that may be at odds with how the company currently works, which results in conflict around role definition. You can’t prevent all tension between you and other executives about your respective roles, but you can avoid accidental friction by discussing the areas of intentional disagreement.

The three questions you should work together on are:

What are your respective roles? Both of you should independently write down your expectations of the other role, then share those expectations with one another. If you find something that you disagree on, that’s great! Either it’s an accidental disagreement that is quick to resolve, or it’s a genuine disagreement that the two of you will need to spend time working through. In either case, you get to skip the step where you’re surprised to learn about it
How do you handle public conflict? Part of being an effective executive team is appearing aligned with each other, even when you’re not, and that depends on having a clear communication strategy. This is easiest when you agree on how you handle public conflict, usually trying to politely resolve in the current meeting if it’s easy, and but agreeing to take it to a private meeting if you’re far enough apart that you can’t resolve without exposing the conflict to the wider team
What is the escalation process for when you disagree? It’s inevitable that you will disagree with each other, sometimes on very important topics. I have certainly disagreed with my executive peers on budget allocation, roadmap, acquisitions, and any other topics. Most executives stumble through agreeing on an implicit escalation process for resolving conflicts between each other, but you can also just have an explicit conversation about it. When people complain about politics, often it’s a subtly disagreement about an implicit escalation contract: you can just make it explicit instead

These jobs never stop changing, the relationship between you and your peer that will be the only constant in navigating those changes. Spending time upfront will take a bit of time, but save a tremendous amount of time longer term, and start your relationship off on the right foot.

Trust comes with time

Once you’ve agreed on an initial definition of your roles and how you’ll work together, the final step of onboarding is to schedule ongoing time working together. Your initial agreement won’t actually be quite right, and you’ll only figure out the issues by working through difficult issues together. Executive calendars fill up so quickly that, unless you book recurring time in advance, you’ll only work on very painful issues that escalate past your routine calendar, which are the sort of topics that strain rather than build an early relationship.

The key pieces I’d recommend setting up are:

Spend some time knowing them as a human. Schedule some lunches or dinners to create a few non-transactional experiences together with this new coworker. Executive roles often pull you towards treating each other transactionally, but it doesn’t have to be that way. Try to make this happen a few times a year
Hold a weekly 1:1, structured around a shared 1:1 document, that lasts an hour. Repeated exposure has a magic to it, and an hour is enough time to dig deep into two or three meaningful problems.

If a weekly hour feels like too much, it may be a sign that either your functions don’t work together much, or that your company is at a stage where executives are leading in the weeds, which many early stage companies are. Try the weekly hour a few times, but absolutely move to less time if this doesn’t feel productive
Identify the meetings where your organizations partner together to resolve prioritization and work through any conflict. These might be weekly metrics reviews, monthly operational reviews, quarterly planning, or something else. Once you identify them, make sure both of you attend at least a few sessions together and afterwards discuss any conflict or disagreement to be better aligned in the future

Finally, trust is about doing what you say you will, which takes time to build. You can absolutely immediately enjoy working with someone, but you can only trust them after having them make commitments and seeing them live up to those commitments. This can’t be rushed.

How much progress is possible?

If an engineer is unconstructively argumentative or does particularly poorly in their first three months, you would generally manage them out of the company. Struggling executive hires are often evident within three months, but will often take a year or longer to leave a company. Mediocre but very nice executives last much longer, often remaining indefinitely.

As a peer, your only bet is to partner in good faith, trying to help new executives succeed. Even the best executives need support, and you can significantly change your company’s trajectory–and your experience working within it–by helping them onboarding effectively. Even if a new executive isn’t working out, it’s the CEO’s job to address that, not yours, and pushing too hard will only reflect poorly on you without solving the problem.

Deciding to leave your (executive) job.

Mon, 20 Mar 2023 06:00:00 -0600

If two friendly executives meet for dinner, it’s likely they start by exchanging just how messed up things are at work. Initiatives are behind, layoffs are happening everywhere, the team is in disarray. Then they’ll laugh, and switch topics. Sometimes one of the executives can’t navigate the switch, and will keep ranting throughout their meal. Having problems is part of being an executive, but when you’re that second executive who can’t turn off the frustration, it’s time to start thinking about leaving.

Departing an executive job is much messier than leaving an individual contributor role, and will significantly impact the team and company around you. It’s also potentially impactful to your resume; I frequently talk with executives who hate their job, but don’t want to leave because they’re worried it looks bad. “If I just make it to two years, it’ll look great.”

The optics are real, and I understand why people get caught up in them. I’ve certainly gotten caught up in my own optics at times. That said, the deeper I get into my own career, the more convinced I become that we should think about departing roles in the context of managing our own energy.

We’ll walk through:

Succession planning before a transition
Making the decision to leave
How to think about short executive stints
Whether to line up another role before leaving
Telling the CEO
Negotiating the exit package
Transitioning out and actually leaving

Finally, we’ll end with a discussion about handling the hardest aspect of all, indecision. By the end, you’ll have a framework for making the decision to leave, and a structure to coordinate your departure.

This is an unedited chapter from O’Reilly’s The Engineering Executive’s Primer.

Succession planning before a transition

In a literal sense, deciding to leave your job is the first step in departing. However, in a smooth departure, the first step happens years earlier: building the team to support your eventual transition out of the business. Even if you intend to remain at your current company forever, life comes at you fast. Certainly, I never imagined I would have a stroke in my mid-thirties, with an infant at home, but then I did. I was fortunate to lead a collaborative team who worked together well in my absence, but that wasn’t an accident, it was the byproduct of ongoing succession planning.

This planning doesn’t need to be anything heavy:

In performance reviews, think about what your direct reports are missing to thrive in your role, and give them at least one of those areas to focus on in each review
As they make those improvements, talk to the CEO about the growth you’re seeing in your team. For the one or two with the best chance of eventually succeeding into your role, facilitate them building a relationship with the CEO
Every quarter, run an audit of the recurring meetings you attend. For each, can you delegate it to someone on your team? If it’s not something you can delegate, partner with them to improve on whatever is missing before they could be effective in that forum.

While it’s hard to displace yourself from executive-only or board meetings, almost everything else is possible. For example, I’ve seen a number of companies where the engineering leadership team alternates running their weekly team meeting rather than the engineering executive
Go on at least one long vacation each year (aim for two weeks, if you can), and explicitly delegate your roles across your team rather than slowing things down. Avoid chiming in on email and chat. Even if mistakes are made, they’ll be learning mistakes

Ultimately, who replaces you won’t be your decision, it’s the CEOs, but doing a small amount of ongoing prep work will make an internal transition possible. Without this proactive work, it likely won’t be seriously considered, which is a shame: so many companies would do well to consider the talent they already have.

Deciding to leave

There’s a certain romance to abruptly quitting a job. I’ve seen two people legitimately rage quit their jobs. They went into a meeting, got upset, and quit immediately. In one, where the individual reported to me, they threw a piece of paper at my face, where they’d signed their name beneath a brief message, “I quit, effectively immediately.” In both cases, I unironically appreciated their momentary clarity with the situation. They were done.

There’s a lot less romance in quitting an executive job. There’s the dream that originally convinced you to join, still recognizable but frayed. There’s dozens of little frustrations, miscommunications, and disagreements. There’s years of preparation ahead of time to facilitate a smooth transition. There’s also a huge reservoir of things that have gone well: people you love working with, teams that you’ve helped build, and company successes you were part of. For many, being an executive is an important pillar of their identity, and that’s hard to walk away from.

When talking to executives grappling with this intersection of identity and frustration, there are four questions I ask them to help with making the decision:

Has your rate of learning significantly decreased? Even frustrating situations can be extremely valuable to you long term, as long as you’re learning. It’s when the rate of learning starts to slow that you should get particularly concerned
Are you consistently de-energized by your work? It’s normal to have bad days. It’s normal to have bad weeks. You will even have bad months. However, if things are consistently bad, then it’s time to consider change. Don’t trust your gut here, instead keep a journal of your daily energy level for a quarter. If you can’t find a trend of good days, it’s worth considering moving on
Can you authentically close candidates to join your team? Every executive I’ve known can flip into selling mode. They’ll give a warm, balanced, but optimistic view of why you should join their company. Even if you’re upset with your role, you should still be able to turn the sell on. If it starts to feel dishonest to switch into selling mode, that’s a sign that you’re losing your ability to effectively perform your role
Would it be more damaging to leave in six months than today? Sometimes executives inadvertently create a values oasis, where their organization’s values differ significantly from the wider company’s values. That oasis will feel comfortable for their team, but their team will acclimatize to the oasis’ environment such that they can’t operate effectively if the executive leaves. Your goal is to create an organization that’s successful beyond your tenure with the company. If you believe your organization is drifting away from the company’s culture, and you’re unwilling to steer it back into alignment, then you should consider if you’re putting your team’s careers as risk by remaining with the company

One question that I don’t recommend anchoring on is how much money you believe you’ll lose by leaving. There are two reasons for that. First, you may be able to negotiate an exit agreement that minimizes your downsides. Second, people are extraordinarily bad at determining the value of their own compensation package. Many tech executives are sitting on equity packages potentially worth between zero and tens of millions of dollars, and the actual value is simply not knowable. You can’t make a good decision based on that.

Finally, I would caution against making ultimatums, taking a last stand on something, or whatever. Being an executive is an ongoing negotiation with the CEO and management team on how the company functions. If you’re not willing, or comfortable, to continue negotiating with the current group, it’s time to move on. Even if you leave, you’ve still been part of the organization and are partially responsible for the steps they’ve made, even the ones you disagree with. Being an executive is, ultimately, a compromising position, where you’re working inside to effect change, rather than a role where you can ever wipe your hands clean.

Am I changing too often?

I often talk to executives who have decided to leave except for “one little issue.” While I believe that these issues are often deliberate impediments raised to avoid making the decision, there’s one that comes up frequently enough to address explicitly: should executives stay in their role to avoid a short stint on their resume?

If something goes very wrong, very early, then it’s always fine to leave and omit it from your resume. You’ve uncovered gross financial fraud in the two months? Yeah, it’s fine to quit, just scrub it off your resume. Similarly, once you’ve hit three to four years, there’s no meaningful penalty. It’s the intermediate interval, roughly six months to two years, where folks often get tripped up about whether leaving will hinder their ability to find another role.

Because the people making hiring decisions about engineering executives generally are not engineers, the evaluation tends to rely a fair amount on optics, and your search will place meaningful value on your previous tenures. The intersection between my experience, peer experiences, and discussions with executive recruiters is roughly:

If it’s less than three months, just delete it from your resume and move on
If it’s more than two years, you’ll be able to find another role as long as some of your previous roles have been three-plus years
As long as there’s a strong narrative, any duration is long. For example, if your company is acquired by a larger company, the narrative is that you weren’t excited being in the larger environment
If a company reaches out to you, there’s no tenure penalty. Say, you’ve been in a role for six months, but then you get an email about another role. Because they reached out to you, the short tenure would be because you’re very excited about them, and not a reflection on you. (Admittedly, it’s a bad look to do this multiple time in a row.)

If you don’t fall into any of those scenarios, there probably will be a moderate penalty on your next search, with potential employers asking, “But will they stay if we hire them?” I certainly agree that this is unfair in some cases, but success comes from navigating reality’s many warts. In such cases, you may need to prioritize finding a next role where you can succeed for four to five years, even if it’s more of a lateral move than an upward one.

Leave with or without your next role?

Tied into the question of tenure, the easiest way to remove tenure risk from your search is to get your next role before leaving your current role. You’ll always have more leverage negotiating from an existing role, and if your primary goal is advancing your career, then I would recommend finding your next role before departing your current role. Similarly, executive searches tend to run slowly. You need to be comfortable supporting yourself and family for six-plus months to consider leaving your role without another role lined up.

That said, I personally believe the greatest risk to your executive career is running out of energy, so taking a bit of rest might be more valuable than the short-term numbers suggest. What if a three month rest gives you the energy to continue this work for another ten years? Then the obvious decision gets a bit less obvious.

Finally, leaving without a role lined up makes it much easier to support your outgoing transition, which often opens up the topic of exit packages, which I’ll discuss further down.

Telling the CEO

After you’ve made a decision to depart, your next step is telling your CEO. It’s important to go into that conversation with a firm grip on your goals. Many CEOs will try to change your mind, but it’s important to hold to your plan. If you are open to changing your mind, then you should be having a career discussion with the CEO, not a departure discussion. There’s a natural momentum here: once you start the departure discussion, you’re going to leave. Even if you and the CEO leave the conversation believing you’re going to stay, you will leave. It’ll just take a few months longer than expected.

Trust is the underpinning of your relationship with the CEO, and fraying that trust will make whatever else is happening just a bit worse, confirming whatever other factors contributed to your initial decision to move on. This is equally true if you try to leverage your departure as an ultimatum to push some change through. It might work in the short term, but long-term you’re definitely on the way out.

The most important points to land in this discussion are:

Departure timeline: what are your preferred and inflexible timelines for leaving?
What you intend to do next: taking on a new role, taking time to recuperate, or whatnot
Why you’re departing: assuming you’ve been clear about issues up to this point, there usually isn’t something new to say. You’ve already said your piece, and they’ve acted upon it to whatever extent they’re willing. As such, I’d think of this as preparing the messaging for the board, the wider management team, and so on
Recommended transition plan: how do you recommend the CEO handle this transition? Who, if anyone, internally should step into your role, and so on. Keep in mind that once you start this discussion, you are now making recommendations, rather than leading the function

Generally, the CEO isn’t going to make any decisions in this meeting, just hear you out, say they’re going to think this through, and schedule a followup to discuss details after they’ve been able to figure out how they want to handle your departure.

Negotiating the exit package

Depending on the discussion you’ve had with the CEO, you may have a lot of leverage for negotiating an exit package, or you may not have any leverage at all. If you’ve lined up a new role and are giving two to four weeks notice, then you’re very unlikely to negotiate an exit package beyond that you negotiated in your initial contract.

On the other hand, if you’re willing to time your exit to the company’s preference, and you have a good relationship with the CEO, then there’s a great deal of leeway in the negotiations. If you agree on a three month departure, then you might be able to stay on payroll for a month or two after your last day. Or you might be able to continue vesting and receiving medical coverage, but not salary, until your next relevant vesting cliff. They might be willing to approve a secondary stock transaction, even if they typically don’t approve such transactions.

None of this stuff is guaranteed, it depends entirely on how much you’re willing to facilitate the exit, how the company values your past contribution, whether they like you, and your CEO’s preferences. It’s worth discussing a bit, particularly if you go in with clarity about the one thing that really matters to you (e.g. approving a secondary stock transaction, staying on payroll, etc).

Transition out and actually leave

While there’s a great deal you can do before you decide to leave, once you’ve communicated your decision to leave, you won’t be able to change much. It can be tempting to believe you’ll finally get the chance to address a long-standing issue, but you’re a lame duck. Even if your idea is a good one, you’ve lost credibility to propose it because you won’t be around to deal with the consequences.

The most common mistake I see outgoing executives make is trying too hard to help. At this point, you’re a consultant, not an executive. The best you can do in a transition is get out of the way, and support your CEO, your team, and the company in the plan that they choose. Once you give notice, it’s no longer your engineering team, and you’ll have to navigate the abrupt shift from leader to follower. Sometimes this means implementing plans you don’t particularly agree with. Even if you disagree with the plans, saying you disagree will only destabilize the team’s transition. Do your best to support them, and then move on.

Sometimes you’ll see folks stay very involved after their departure. They might keep one-on-one meetings, serve as an explicit mentor to team members they managed, or attend social work functions. I understand the intent, but I strongly recommend against these sorts of prolonged departures. They are usually motivated by a sense of guilt rather than a clear plan to help, and undermine the person who replaces you.

Revisiting the decision

There are many micro-decisions within the broader choice to leave a job, and one place I see folks get stuck is revisiting the decision repeatedly, sometimes on a daily basis. They have a particularly bad meeting, and then recite, “Yup, I’m out of this place.” They tell a few external friends that they’re done. Two days later, earnings are up, and they get excited again. They’re too embarrassed to tell their friends they’ve changed their mind (likely for the Nth time), so they wave the question off when they meet up next, “Oh that, yeah. Just a bad day.”

Some people exist in that cycle indefinitely. Earlier in my career, we hired a new manager who I reported to, and that manager told me–in our very first one-on-one on our first day–that he’d made a mistake, hated the company, and should quit. Most of our one-on-ones focused on my manager telling me how stupid the company was, and how he thought he should quit or get the CTO fired. It wasn’t a particularly inspiring situation for me.

Getting caught in the decision is the least helpful thing you can do. You’ll spend a tremendous amount of your energy debating, but generally without a clear answer. If the answer was clear, you wouldn’t be stuck going in a circle. If you, for whatever reason, can’t make a decision right now, then I highly recommend establishing a decision date in the future. This could be a literal date, “the end of next month,” or after a particular set of circumstances, “once we see the impact of our latest product features.”

Once you reach that moment, then see if you’re in a place where you can make a decision. If so, then make it! If not, pick another date, and wait until then to reconsider. It’s not that the decision gets any easier, but it helps you spend less time spinning on a decision that you are clearly not ready to make.