Irrational Exuberance

How to get more headcount.

Sat, 16 Nov 2024 07:00:00 -0700

One of the recurring challenges that teams face is getting headcount to support their initiatives. A similar problem is the idea that a team can’t get a favored project into their roadmap. In both cases, teams often create a story about how clueless executives don’t understand why their work is important.

I understand why dumb executives are such an appealing explanation to problems: it fits perfectly into the Karpman drama triangle by making executives the villian and the team the victim, but I generally find that these sorts of misalignments are the result of basic communication challenges rather than something more exciting.

When there’s significant misalignment between a team and an executive, my experience is that it often manifests in discussion about a particular project, but it’s often rooted in a much broader topic rather than whatever is currently being discussed. Because the disagreement is about the larger topic, there’s no way to resolve it while discussing the narrow project at hand, and teams struggle to make progress because they’re arguing on the wrong layer of context.

To solve disagreement, the general hierarchy of alignment is:

Do we agree on the problem to be solved?

e.g. We’re having too many incidents and it’s impacting user perception and developer productivity.
Do we agree on the general approach to solving that problem?

e.g. We’re increaing end-to-end coverage and rejecting PRs that reduce coverage.
What evidence do we have that the team is executing well today?

e.g. Here are metrics on: how many user-impacting incidents we’ve had over time, how we’ve increased end-to-end coverage over time, and developer survey feedback on incidents from the last three quarterly internal surveys.
Alignment on the particular topic at hand: headcount, roadmapping, prioritization of a specific project, and so on.

e.g. To speed up further work on this, we’re requesting two more engineers.

If you are misaligned on any of the first three topics, addressing the fourth project is folly. For example, if the executive believes that your team is not executing your current project well, then they won’t believe that giving you more headcount is useful, because you’re already screwing up. To convince them to approve a headcount request, you need to first find evidence that your team is doing good work today.

Similarly, if the executive doesn’t agree with you on your problem or general approach, your headcount request is dead in the water. This is one of the reasons that I see bottoms-up “team mission” initiatives fail so frequently. Teams define their mission, and then tell executives what they’ve decided to focus on, but that’s a general misunderstanding of why teams exist: teams exist to solve a company need, not to solve the problem that the team itself wants to solve. When teams lean on their self-selected problem or approach to defend to an executive why they won’t do something, they dig in on a foundational misalignment that prevents addressing more nuanced discussions like project prioritization.

The solution here is obvious, always make sure you agree on the problem and general solution, and provide evidence the team is working well. These can be an appendix of a document or appendix slides, and should take little to no time to prepare as the first two are core decisions for your team, and the later is a set of metrics or plans that you should already be maintaining as part of operating your team.

If you refuse to engage on the first three topics, and skip to the fourth topic directly in aligning with an executive, then you are generally falling back to relying on social dynamics and executives’ general view of your prior work–what some might call politics–rather than having a joint problem solving session together. If you’re complaining about politics, and not taking the time to answer the first three, then perhaps you are inadvertantly contributing to the political environment that you dislike.

As always when discussing challenges communicating with executives, it’s true that executives should get better at explaining where they’re confused or struggling with the rationale. However, it’s a lot more useful to simply get better at this yourself than to spend time bemoaning how executives could, at a universal group, improve their communication. I know a lot of people who improved their company or moved into more senior roles by improving their own communication. I know zero folks who did either of those by complaining that executives are bad communicators, although they certainly weren’t wrong about it!

Navigating Private Equity ownership.

Mon, 11 Nov 2024 06:00:00 -0700

In 2020, you could credibly argue that ZIRP explains the world, but that’s an impossible argument to make in 2024 when zero-interest rate policy is only a fond memory. Instead, we’re seeing a number of companies designed for rapid expansion learning to adapt to a world that expects immediate free cash flow rather than accepting the sweet promise of discounted future cash flow.

This chapter wants to tackle that problem head-on, taking the role of an engineering organization attempting to navigate new ownership by a private equity group. It’s an increasingly frequent scenario: after many years of learning to operate under the direction of its original founders, and the brief excitement of going public, now there’s a short runway to change operating models. Let’s call this company Fungible Ecommerce Company. It’s a platform for supporting online commerce, and this is their Engineering Leadership team’s attempt to think through their options while waiting for new ownership to provide concrete guideposts.

This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts.

Reading this document

To apply this strategy, start at the top with Policy. To understand the thinking behind this strategy, read sections in reserve order, starting with Explore, then Diagnose and so on. Relative to the default structure, this document has been refactored in two ways to improve readability: first, Operation has been folded into Policy; second, Refine has been embedded in Diagnose.

More detail on this structure in Making a readable Engineering Strategy document.

Policy

Our policy for managing our new ownership structure is:

We believe our new ownership will provide a specific target for Research and Development (R&D) operating expenses during the upcoming financial year planning. We will revise these policies again once we have explicit targets, and will delay planning around reductions until we have those numbers to avoid running two overlapping processes.

That said, looking at our R&D investment relative to comparably growing peer set, we believe that we’ll get pressure to moderately reduce our spend. We aim to accomplish that reduction through a series of policies and one-off infrastructure projects, without requiring a major reduction in headcount spend.
We will move to an “N-1” backfill policy, where departures are backfilled with a less senior level. We will also institute a strict maximum of one Principal Engineer per business unit, with any exceptions approved in writing by the CTO–this applies for both promotions and external hires. These policies are effective immediately, and is based on our model of engineering-org seniority-mix.

We commit to this policy reducing headcount costs by approximately 5% YoY every year for the forseeable future.
We evaluated a number of potential changes to our geographical hiring strategy, but we believe that staffing engineers with cross-functional partners (Product, Marketing, Sales, and so on) is a priority. We have not been able to reach an agreement cross-functionally, and as such we are not changing our geographical hiring strategy at this time.

If we can agree on a policy here, we could accomplish 10-20% reduction in cost over 2-3 years, but the details matter a great deal, so we cannot commit to a specific outcome until we get more cross-functional alignment.
Our infrastructure spend has grown significantly more slowly than revenue for the past two years, meaning that we’ve successfully implement our infrastructure spend strategy of growing infrastructure costs more slowly than revenue. We will continue our current infrastructure efficiency strategy, and believe they are relatively few high impact efficiency opportunities at this point.

We commit to growing infrastructure spend at no more than 5% YoY, significantly lower than our projected revenue increase of 25% YoY.
There are two narrow infrastructure spend opportunities, both related to the integration of prior acquisitions into our shared infrastructure and away from one-off approaches. We will prioritize the post-acquisition integration work next quarter, with the goal of fully standarding all infrastructure across the company into the stack maintained by our centralized Infrastructure Engineering team.

We commit to a one-time reduction in infrastructure of 3% YoY.
We believe there are significant opportunities to reduce R&D maintenance investments, but we don’t have conviction about which particular efforts we should prioritize. We will kickoff a working group to identify the features with the highest support load.

Diagnose

We’ve diagnosed Fungible Ecommerce Company’s current state as:

Fungible Ecommerce Company’s revenue has grown 20-25% YoY for the past two years, and our target for next year is 25% YoY revenue growth. While this is not a guarantee, we grew slower than 25% last year, it’s a defensible goal that we have a good chance of achieving.
Our Engineering headcount costs have grown by 15% YoY this year, and 18% YoY the prior year. Headcount grew 7% and 9% respectively, with the difference between headcount and headcount costs explained by salary band adjustments (4%), a focus on hiring senior roles (3%), and increased hiring in higher cost geographic regions (1%).
Based on general practice, it seems likely that our new Private Equity ownership will expect us to reduce R&D headcount costs through a reduction. However, we don’t have any concrete details to make a structured decision on this, and our approach would vary significantly depending on the size of the reduction.
Infrastructure engineering spend (including vendors) has grown by 4-5% YoY for the past three years. We made a significant push on reducing costs three years ago, and have grown significantly slower than revenue since then.

There are few remaining opportunities to significantly reduce infrastructure costs, but we’ve made several acquisitions since our prior infrastructure consolidation, that represent significant potential savings: roughly one-time 1.5% YoY reductions for each of two largest opportunities.
A significant portion of our current R&D spend goes into maintaining our existing functionality, particularly functionality related to earier geo-expansion efforts that only apply narrows to some small markets. We suspect there’s an opportunity to reduce maintenance overhead here.

However, we lack believable metrics on both (1) time spent maintaining the software and (2) time that would be saved by these cleanup efforts. As a result, it’s hard to pitch projects of this sort as revenue saving with much conviction.

Explore

Financial markets evaluate companies in comparison to their peers. This is most obvious in public markets, where there’s significant information transparency about business performance, and sufficient liquidity to allow markets to revalue companies in something approaching real-time. While private equity firms generally take controlling interest of private businesses, or with the intent of taking the business private if it happens to be public, they value businesses in the same way.

In this exploration, we’re going to dig into two particular questions. First, we’re going to dig into a dataset on the performance of public technology companies, and then second we’re going to look into the concrete example of Zendesk, who were taken private in 2022 after being bought by two private equity firms.

Comparable companies

Exploring the benchmarking question first, most investors evaluate engineering within the context of the overall Research & Development (R&D) investment. They generally judge that spend by constructing a scatterplot of R&D spend versus year-over-year revenue growth for a cohort of similar companies. Perfectly similar companies don’t exist, so this cohort is generally constructed from companies in similar industries, with similar revenue, and operating in the same regions.

We have reached out to our investors to see if they can provide the internal datasets they use for this analysis, but in the mean-time we’ve developed a directionally useful dataset using the 2023 R&D Investment Scoreboard, with some rough cutting of the data to remove outliers. (If we repeat this process, we will use the SEC’s EDGAR database to pull a more specifically helpful dataset, but this has been a useful starting point.)

This isn’t a perfect dataset, we prefer revenue growth over rgowth in operating profit, but it’s the best option within the dataset that we were able to quickly pull down. Nonetheless, there’s a clear strong performer quadrant in top-left that we can plot ourselves into to understand our general performance, which is discussed further in the diagnosis section above.

Zendesk

The second topic of exploration we dug into is understanding the general sequence of steps taken by private equity ownership after taking ownership of a company. For an example with available public documentation, we focused on the purchase of Zendesk in 2022.

To start, we pulled Zendesk’s final 10-Q before going private.

Taking those values, we can reformat them into a chart focusing on the year-over-year changes in the 6 months period ending in 2022 versus the same period in 2021.

The changes are a bit concerning. Sales and Marketing costs have grown more slowly than revenue, which is positive, but Research and Development (R&D) expenses have grown about 50% faster than revenue, and General and Administration (G&A) charges have grown more than twice as quickly as revenue.

From those growth rates, we would assume that the new ownership might push to aggressively reduce spend in those two areas, which is indeed what history suggests happend, with a November, 2022 reduction, followed some months later by a May, 2023 reduction. It’s hard to get precise data here, but it’s our impression that these reductions focused on areas where expenses were growing quickly, with particular focus on G&A functions.

Using systems modeling to refine strategy.

Mon, 04 Nov 2024 07:00:00 -0700

While I was probably late to learn the concept of strategy testing, I might have learned about systems modeling too early in my career, stumbling on Donella Meadows’ Thinking in Systems: A Primer before I began my career in software. Over the years, I’ve discovered a number of ways to miuse systems modeling, but it remains the most effective, flexible tool I’ve found to debugging complex problems.

In this chapter, we’ll work through:

when systems model is a useful technique, and when it’s better to rely on other refinement techniques like Wardley mapping or strategy testing instead
a two minute primer on the basics of systems modeling, along with resources for those looking for a deeper exploration of the foundational topics
a discussion on systems modeling tooling, why there’s no perfect systems modeling tool out there, and how I recommend picking the tool that you build proficiency with
the steps to build a systems model for a problem you’re engaging with
how to document your learnings from a systems model to maximize the chance that others will pay attention to it rather than ignoring it due to the unfamiliarity or complexity of the tooling
what systems modeling can’t do, even if you really want to believe it can

After working through this chapter’s overview of systems modeling, you can see the approaches implemented in a number of system models created to refine the strategies throughout this book. The theory of systems modeling is certainly interesting, but hopefully seeing real models in support of concrete engineering strategies will be even more useful.

When is systems modeling useful?

Although refinement is an important step of developing any strategy, some refinement techniques work better for any given strategy. Systems modeling is extremely useful in three distinct scenarios:

When you’re unsure where leverage points might be in a complex system, modeling allows you to cheaply test which levers might be meaningful. For example, modeling onboarding drivers in a ride-sharing app showed that improving onboarding was less important than reengaging departed drivers.
When you have significant data to compare against, which allows you to focus in on the places where the real data and your model are in tensions. For example, I was able to model the impact of hiring on Uber’s engineering productivity, and then compare that with internal data.
When stakeholder disagreements are based in their unstated intuitions, models can turns those intuitions into something structured that can be debated more effectively.

In all three categories, modeling makes it possible iterate your thinking much faster than running a live process or technology experiment with your team. I sometimes hear concerns that modeling slows things down, but this is just an issue of familiarity. The more you practice, modeling can be faster than asking for advice fro industry peers. The actual models I’ve developed for this book took less than an hour. (With one notable exception: modeling Large Language Models (LLMs) impacts on developer experience, which took much longer because I deliberately used an impractical tool to reveal the importance of good tooling).

Additionally, systems modeling will often expose counter-intuitive dimensions to the problem you’re working on. For example, the model I mentioned above on LLMs’ impact on developer experience suggests that effective LLMs might cause us to spend more time writing and testing code (but less fixing issues discovered post-production). This is a bit unexpected, as you might imagine they’d reduce testing time, but reducing testing time is only valuable to the extent that issues identified in production remains–at worst–constant; if issues found in production increases, then reduced testing time does not contribute to increased productivity.

Modeling without praxis, creates unsubstantiated conviction. However, in combination with praxis, I’ve encountered few other techniques that can similar accelerate learning.

That doesn’t mean that it’s always the ideal refinement technique. If you already have conviction on the general approach, and want to refine the narrow details, then strategy testing is a better option. If you’re trying to understand the evolution of a wider ecosystem, then you may prefer Wardley mapping.

Two minute primer

If you want an exceptional introduction to systems thinking, there’s no better place to go than Donella Meadows’ Thinking in Systems. If you want a worse, but shorter, introduction, I wrote a short Introduction to systems thinking available online and in An Elegant Puzzle.

If you want something even shorter, then here’s the briefest that I can manage.

Accumulations are called stocks. For example, each of the boxes (Requests, Server, etc) in the above diagram is a stock. Changes to stocks are called flows. Every arrow (OK, Error in server, etc) between stocks in the diagram is a a flow.

Systems modeling is the practice of using various configurations of stocks and flows to understand circumstances that might otherwise have surprising behavior or are too slow to understand from measurement.

For example, we can use the above model to explore the tradeoffs between a load balancer that does and does not cap throughput to a load-sensitive service behind it.

Without a model, you might get into a philosophical debate about how rediculous it is that the downstream server is load-sensitive. With the model, it’s immediately obvious that it’s worthwhile protecting it, even if it certainly is concerning that it is so sensitive. This is what models do: they create a cheap way to understand reality when fully understanding reality is cumbersome.

More systems thinking resources

Thinking in Systems: A Primer by Donella Meadows

Business Dynamics: Systems Thinking and Modeling for a Complex World by John D. Sterman

An Introduction to Systems Thinking by Barry Richmond

Tooling

For an idea that’s quite intuitive, the tools of systems modeling are a real obstacle to wider adoption. Perhaps a downstream consequence of many early, popular systems modeling tools being quite expensive, the tooling ecosystems for systems modeling has remained fragmented for some time. There also appears to be a mix of complex requirements, patent consolidation, and percieved small market size that’s discouraged a modern solutions from consolidating the tooling market.

Earlier, I mentioned that system modeling is extremely quick, but that many folks find it a slow, laborous process. Part of that is an issue of practice, but I suspect that the quality of modeling tooling is at least a big a part of the challenge. In the LLMs impact on developer experience model, I go about the steps of building the model in an increasingly messy spreadsheet. This was slow, challenging, and extremely brittle. Even after finishing the model, I couldn’t extend it effectively to test new ideas, and I inadvertently introduced a number of bugs into the implementation.

Going in the opposite direction, I explored using a handful of tools, such as Sagemodeler or InsightMaker, which seemed like a potentially simpler toolchains than the one I typically rely on. There are so many of these introductary toolchains for systems modeling, but I generally find that they’re either constrained in their capabilities, have a fairly high learning curve, or make it difficult to share your model with others.

In the end, I wound up back at the toolchain that I use, which happens to be one that I wrote some years ago,lethain/systems. This is far from a perfect toolchain, but I think it’s a relatively effective mechanism for demonstrating systems modeling for a few reasons:

quick to create models and iterate on those models
easy to share those models with others for inspection and their own exploration
relatively low surface area for bugs in your models
free, open-source, self-hosted toolchain that integrates well with Jupyter ecosystem for diagramming, modeling and so on

You should absolutely pick any tool that feels right to you, and practice with it until you feel confident quickly modeling scenarios. Afterwards, I wouldn’t recommend spending too much time thinking about tools at all: the most important thing is to build models and learn from them quickly, and almost any tool will be sufficient to that goal with some deliberate practice.

How to model

Learning to system model takes some practice, so we’ll approach the details of learning to model from two directions. First, by documenting a general structure for approaching modeling, and second by providing breadcrumbs to the models developed in this book for deeper exploration of particular modeling ideas.

The structure to systems modeling that I find effective is:

Sketch the stocks and flows on paper or a diagramming application (e.g. Excalidraw, Figma, Whimsical, etc). Use whatever you’re comfortable with.
Reason about how you would expect a potential change to shift the flows through the diagram. Which flows do you expect to go up, and which down, and how would that movement help you evaluate whether your strategy is working?
Model the stocks and flows in your spreadsheet tool of choice. Start by modeling the flows from left to right (e.g. the happy path flows). Once you have that fully working, then start modeling the right to left flows (e.g. the exception path flows).

See the Modeling impact of LLMs on Developer Experience model for a deep dive into the particulars of creating a model.
Exercise the model by experiment with a number of different starting values and determining which rates really impact the model’s values. This is essentially performing sensitivity analysis
Document the work done in the above sections into a standalone writeup. You can then link to that writeup from strategies that benefit from a given model’s insights. You might link to any section of your strategy, depending on what topic the particular model explores. I recommend decoupling models from specific strategies, as generally the details of any given model are a distraction from understanding a strategy, and it’s best to avoid that distraction unless a reader is surprised by the conclusion, in which case, the link out supports drilling into the details.

As always, this is the sequence of steps that I’d encourage you to follow, and the sequence that I generally follow, but you should adapt them to solve the particular problems at hand. Over time, my experience is that most of these steps–excluding documentation–turn into a single iterative process, and that I document everything after several iterations.

Breadcrumbs for deeper exploration

Now that we’ve covered the overarching approach to system modeling, here are the breadcrumbs to specific models that go deeper on particular elements:

Modeling driver onboarding explores how the driver lifecycle at Theoretical Ride Sharing might be improved with LLMs, and introduces using the lethain/systems library for modeling
Modeling impact of LLMs on Developer Experience looks at how LLMs might impact developer experience at Theoretical Ride Sharing, and is demonstrates (the downsides of) modeling with a spreadsheet
Modeling engineering backfill strategy studies the financial consequences of various policies for how we backfill departed engineers in an engineering organization, and introduces further lethain/systems features

Beyond these models, you can find other systems models that I’ve written on my blog’s systems-thinking category, and there are numerous, great examples in the materials references in the systems modeling primer section above.

How to document a model

Much like documenting strategy is challenging, communicating with models in a professional setting is challenging. The core problems is that there are many distinct groups of model readers. Some will lack familiarity with the tooling you use to develop models. Others will try to refine, or invalidate, your model by digging into the details.

I navigate those mismatches by focusing first on the audience who is least likely to dig into the model. I still want to keep all the details handy, ideally in the rawest form possible to allow others to manipulate the model themselves, but it’s very much my second goal when documenting a model.

From experience, I recommended this order (it’s also the order used in the models in this book, so you’ll see it in practice a number of times):

start with learning section, with charts showing what model has taught you
sketch and explaing the stocks and flows
reason about what the sketch itself teaches you
explain how you developed the model, with an emphasis on any particularly complex portions
exercise the model by testing how changing the flows and stocks leads to different outcomes

If you remember nothing else, your document should reflect the reality that most people don’t care how you built the model, and just want the insights. Give them the insights early, and assume no one will trust your model nearly as much as you do. Models are an input into the strategy, never a reliable sole backer for a strategy.

What systems modeling isn’t

Although I find systems modeling a uniquely powerful way to accelerate learning, I’ve also encountered many practioners who believe that their models are reality rather than reflecting reality. Over time, I’ve developed a short list of cautions to help would-be modelers avoid overcommitting to their model’s insights:

When your model and reality conflict, reality is always right. At Stripe, we developed a model to guide our reliability strategy. The model was intuitively quite good, but its real-world results were mixed. Attachment to our early model distracted us (too much time on collecting and classifying data) and we were slow to engage with the most important problems (maximizing impact of scarce mitigation bandwidth, and growing mitigation bandwidth). We’d have been more impactful if we engaged directly with what reality was teaching us rather than looking for reasons to disregard reality’s lessons.
Models are immutable, but reality isn’t. I once joined an organization investing tremendous energy into hiring but nonetheless struggling to hire. Their intuitive model pushed them to spend years investing into top of funnel optimization, and later steered them to improving the closing process. What they weren’t able to detect was that misalignment in interviewer expectations was the largest hurdle in hiring.
Every model omits information; some omit critical information. The service migration at Uber is a great example: modeling clarified that we had to adopt a more aggressive approach to our service migration in order to succeed. Subsequently, we did succeed at the migration, but the model didn’t study the consequences of completing the migration, which were a very challenging development environment. The model captured everything my team cared about, as the team responsible for running the migration, but did nothing to evaluate whether the migration was a good idea overall.

In each of those situations, two things are true: the model was extremely valuable, and the model subtly led us astray. We would have been led astray even without a model, so the key thing to remember isn’t that models are inherently misleading, instead the risk is being overly confident about your model. A powerful tool to use in tandem with judgment, not a replacement.

Summary

Systems modeling isn’t prefect. If you’ve already determined your strategy and want to refine the details, then strategy testing is probably a better choice. If you’re trying to understand the dynamics of an envolving ecosystem, then Wardley mapping is a more appropriate tool.

However, if you have the general shape, but lack conviction on how the pieces fit together, systems modeling is a remarkable tool. After this chapter, you know how to select appropriate tooling, and how to use that tooling to model your problem at hand. Next, we’ll work through systems modeling a handful of detailed problems to provide concrete examples of applying this technique.

Eng org seniority-mix model.

Sun, 27 Oct 2024 04:00:00 -0700

One of the trademarks of private equity ownership is the expectation that either the company maintains their current margin and grows revenue at 25-30%, or they instead grow slower and increase their free cash flow year over year. In many organizations, engineering costs have a major impact on their free cash flow. There are many costs to reduce, cloud hosting and such, but inevitably part of the discussion is addressing engineering headcount costs directly.

One of the largest contributors to engineering headcount costs is your organization’s seniority mix: more senior engineers are paid quite a bit more than earlier career engineers. This model looks at how various policies impact an organization’s seniority mix.

In this chapter, we’ll work to:

Summarize this model’s learnings about policy impact on seniority mix
Sketch the model’s stocks and flows
Use lethain/systems to iteratively build and exercise the full model

Time to start modeling.

This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts.

Learnings

An organization without a “backfill at N-1” hiring policy, e.g. an organization that hires a SWE2 to replace a departed SWE2, will have an increasingly top-heavy organization over time.

However, even introducing the “backfill at N-1” hiring policy is insufficient, as our representation in senior levels will become far too high, even if we stop hiring externally into our senior-most levels.

To fully accomplish our goal of a healthy seniority mix, we must stop hiring at senior-most levels, implement a “backfill at N-1” policy, and cap the maximum number of individual at the senior-most level.

Any collection of lower-powered policies simply will not impact the model’s outcome.

Sketch

We’ll start by sketching this system in Excalidraw. It’s always fine to use whatever tool you prefer, but in general the lack of complexity in simple sketching tools focuses you on iterating on the stocks and flows–without getting distracted by tuning settings–much like a designer starting with messy wireframes rather than pixel-perfect designs.

We’ll start with sketching the junior-most level: SWE1.

We hire external candidates to become SWE1s. We have some get promoted to SWE2, some depart, and then backfill those departures with new SWE1s.

As we start sketching the full stocks and flows for SWE2, we also introduce the idea of backfilling at the prior level. As we replicate this pattern for two more career levels–SWE3 and SWE4–we get the complete model.

The final level, SWE4, is simplified relative to the prior levels, as it’s no longer possible to get promoted to a further level. We could go further than this, but the model will simply get increasingly burdensome to work with, so let’s stop with four levels.

Reason

Reviewing the sketched system, a few interesting conclusions come out:

If promotion rates at any level exceed the rate of hiring at that level plus rate of N-1 backfill at that level, then the proportion of engineers at that level will grow over time
If you are not hiring much, then this problem simplifies to promotion rate versus departure rate. A company that does little hiring and has high retention cannot afford to promote frequently. Promotion into senior roles will become financially restrained, even if the policy is explained by some other mechanism
Many companies use the “career level” policy as the mechanism to identify a level where promotions generally stop happening. The rationale is often not explicitly described, but we can conclude it’s likely a financial constraint that typically incentivizes this policy

With those starter insights, now we can get into modeling the details,.

Model & Exercise

We’re going to build this model using lethain/systems. The first version will be relatively simple, albeit with a number of stocks given the size of the model, and then we’ll layer on a number of additional features as we iteratively test out a number of different scenarios.

I’ve chosen to combine the Model and Exercise steps to showcase how each version of the model can inspire new learnings that prompt new questions, that require a new model to answer.

If you’d rather view the full model and visualizations, each iteration is available on github.

Backfill-at-level

The first policy we’re going to explore is backfilling a departure at the same level. For example, if a SWE2 departs, then you go ahead and backfill them at SWE2. This intuitively makes sense, because you needed a SWE2 before to perform the work, so why would you hire something less senior?

There are two new systems concepts introduced in this model:

For easier iteration, we’re going to use the systems modeling concept of an “information link”, which is basically using a stock as a variable to define a flow, Specifically, we’ll create a stock named HiringRate with a size of two. Then we’ll use that stock’s size as to define hiring flows at each career level. In programming terms, you can think as defining a reusible variable, but you can use any stock’s size to define flows.
There are effectively an infinite number of potential candidates for your company, so we’re going to use an infinite stock, represented by initializing a new stock surroundined by [ and ]. Specifically in this case this is [Candidates], if we wanted a fixed size stock with 100 people in it, we could have initialized it as Candidates(100). Depending on what you’re modeling both options are useful.

With those in mind, our initial model is defined as:

HiringRate(2)
[Candidates] > SWE1(10) @ HiringRate
SWE1 > DepartedSWE1 @ Leak(0.1)
DepartedSWE1 > SWE1 @ Leak(0.5)
Candidates > SWE2(10) @ HiringRate
SWE1 > SWE2 @ Leak(0.1)
SWE2 > DepartedSWE2 @ Leak(0.1)
DepartedSWE2 > SWE2 @ Leak(0.5)
Candidates > SWE3(10) @ HiringRate
SWE2 > SWE3 @ Leak(0.1)
SWE3 > DepartedSWE3 @ Leak(0.1)
DepartedSWE3 > SWE3 @ Leak(0.5)
Candidates > SWE4(0) @ HiringRate
SWE3 > SWE4 @ Leak(0.1)
SWE4 > DepartedSWE4 @ Leak(0.1)
DepartedSWE4 > SWE4 @ Leak(0.5)

To confirm that we’ve done something reasonable, we can model this using Graphviz.

That looks like the same model we sketched before, without the downlevel backfill flows that we haven’t yet added to the model, so we’re in a good spot.

With that confirmed, lets inspect the four distinct flows happening for the SWE2 stock. In order they are:

External candidates being hired at the SWE2 level, at the fixed HiringRate defined here as 2 hires per round
SWE1s being promoted to SWE2 at a 10% rate. This is a leak because someone being promoted to SWE2 doesn’t mean the other SWE1s disappear
SWE2s who are leaving the company at a 10% rate
Backfill hires of departed SWE2s, who are rehired at the same level

Running that model, we can see how the populations of the various levels grow over time.

Alright, so we can tell that this backfill at level policy is pretty inefficient, because our organization just becomes more and more top-heavy with SWE4s over time. Something needs to change.

Backfill at N-1

To reduce the number of SWE4s in our company, let’s update the model to backfill all hires at the level below the departed employee. For example, a departing SWE2 would cause hiring a SWE1. This specifically means replacing all these lines:

DepartedSWE2 > SWE2 @ Leak(0.5)

To instead hire into the prior level.

DepartedSWE2 > SWE1 @ Leak(0.5)

The one exception is that SWE1s are still backfilled as SWE1s: as it’s the junior-most level, there’s no lower level to backfill into.

Running this updated model, we get a better looking organization.

We’re still top-heavy, but we’ve turned an exponential growth problem into a linear growth problem, so that’s an improvement. However, this is still a very expensive engineering organization to run, and certainly not an organization that’s reducing costs.

No hiring

One reason our model shows so many SWE4s is because we’re hiring at an even rate across all levels, which isn’t particularly realistic. Also, it’s unlikely that we’re growing headcount at all to the extent that we’re aiming to reduce our engineering costs over time.

We can model this by setting a HiringRate of zero, and then setting more representative initial values for each cohort of engineers (note that I’m only showing the changed lines, check on github for the full model):

HiringRate(0)
[Candidates] > SWE1(100) @ HiringRate
Candidates > SWE2(100) @ HiringRate
Candidates > SWE3(100) @ HiringRate
Candidates > SWE4(10) @ HiringRate

Now we’re starting out with 100 SWE1s, SWE2, and SWE3s. We have a smaller cohort of SWE4s, with just ten initially. Running the model gives us a updated perspective.

We can see that eliminating hiring improves the ratio of SWE4s to the other levels, but it’s still just too high. We’re ending up with roughly 1.25 SWE1s for each SWE4, when the ratio should be closer to five to one.

Capped size of SWE4s

Finally, we’re going to introduce a stock with a maximum size. No matter what flows want to accomplish, they cannot grow a flow over that maximum. In this case, we’re defining SWE4 as a stock with an initial size of 10, and a maximum size of 20.

SWE4(10, 20)
Candidates > SWE4 @ HiringRate

This could also be combined into a one-liner, although it’s potentially easy to miss in that case:

Candidates > SWE4(10, 20) @ HiringRate

With that one change, we’re getting close to an engineering organization that works how we want.

We ratio of SWE4s to other functions is right, although we can see that the backpressure means that we have a surplus of SWE3s in this organization. You could imagine other policy work that might improve that as well, e.g. presumably more SWE3s depart than SWE2s, because the SWE3s see their ability to be promoted is capped by the departure rate of existing SWE4s. However, I think we’ve already learned quite a bit from this model, so I’m going to end modeling here.

Modeling driving onboarding.

Sat, 19 Oct 2024 04:00:00 -0700

The How should you adopt LLMs? strategy explores how Theoretical Ride Sharing might adopt LLMs. It builds on several models, the first is about LLMs impact on Developer Experience. The second model, documented here, looks at whether LLMs might improve a core product and business problem: maximizing active drivers on their ridesharing platform.

In this chapter, we’ll cover:

Where the model of ridesharing drivers identifies opportunities for LLMs
How the model was sketched and developed using lethain/systems package on Github
Exercise to exercise this model to learn from it

Let’s get started.

Learnings

An obvious assumption is making driver onboarding faster would increase the long-term number of drivers in a market. However, this model show that even doubling the rate that we qualify applicant drivers as eligible has little impact on active drivers over time.

Conversely, it’s clear that efforts to reengage departed drivers has a significant impact on active drivers. We believe that there are potential LLM applications that could encourage departed drivers to return to active driving, for example mapping their rationale for departing against our recent product changes and driver retention promotions could generate high quality, personalized emails.

Finally, the model shows that increasing either reactivation of departed or suspended drivers is significantly less impactful than increasing both. If either rate is low, we lose an increasingly large number of drivers over time.

The only meaningful opportunities for us to increase active drivers with LLMs are improving those two reactivation rates.

Sketch

The first step in modeling a system is sketching it (using Excalidraw here). Here we’re developing a model for onboarding and retaining drivers for a ridesharing application in one city.

The stocks are:

City Population is the total population of a city
Applied Drivers are the number of people who’ve applied to be drivers
Eligible Drivers are the number of applied drivers who meet eligibility criteria (e.g. provided a current drivers license, etc)
Onboarded Drivers are eligible drivers who have successfully gone through an onboarding program
Active Drivers are onboarded drivers who are actually performing trips on a weekly basis
Departed Drivers were active drivers, but voluntarily stopped performing trips (e.g. took a different job)
Suspended Drivers were active drivers, but involuntarily stopped performing trips (e.g. are no longer allowed to drive on platform)

Looking at the left-to-right flows, there is a flow from each of those stocks to the following stock in the pipeline. These are all simple one-to-one flows, with the exception of those coming from Active Drivers leads to two distinct stocks: Departed Drivers and Suspended Drivers. These represent voluntary and involuntary departures

There are a handful of right-to-left, exception path flows to consider as well:

Request missing information represents a driver who can’t be moved from Applied Drivers to Eligible Drivers because their provided information proved insufficient in a review process
Re-engage tracks Departed Drivers who have decided to start driving again, perhaps because of a bonus program for drivers who start driving again
Remove suspension refers to drivers who were involuntarily removed, but who are now allowed to return to driving

This is a fairly basic model, but let’s see what we can learn from it.

Reason

Now that we’ve sketched the system, we can start thinking about which flows are going to have the largest impact, and where an LLM might increase those flows. Some observations from reasoning about it:

If a city’s population is infinite, then what really matters in this model is how many new drivers we can encourage to join the system. On the other hand, if a city’s population is finite, then onboarding new drivers will be essential in the early stages of coming online in any particular city, but long-term reengaging departed drivers is probably at least as important.
LLMs tooling could speed up validating eligible drivers. If we speed that process up enough, we could greatly reduce the rate of the Request missing information flow by identifying missing information in real-time rather than requiring a human to review the information later.
We could potentially develop LLM tooling to craft personalized messaging to Departed Drivers, that explains which of our changes since their departure might be most relevant to their reasons for stopping. This could increase the rate of the Re-engage flow
While we likely wouldn’t want an LLM approving the removal of suspensions, we could have it look at requests to be revalidated, and identify promising requests to focus human attention on the highest potential for approval.
We could build LLM-powered tooling that helps a city resident decide whether they should apply to become a driver by answering questions they might have.

As we exercise the model later, we know that our assumptions about whether this city has already exhausted potential drivers will quickly steer us towards a specific subset of these potential options. If all potential drivers are already tapped, only work to reactivate prior drivers that will matter. If there are more potential drivers, then likely activating them will be a better focus.

Model

For this model, we’ll be modeling it using the lethain/systems library that I wrote. For a more detailed introduction, I recommend working through the tutorial in the repository, but I’ll introduce the basics here as well. While systems is far from a perfect tool, as you experiment with different modeling techniques like spreadsheet-based modeling and SageModeler, I think this approach’s emphasis on rapid development and reproducible, sharable models is somewhat unique.

If you want to see the finished model, you can find the model and visualizations in the Jupyterhub notebook in lethain:eng-strategy-models.. Here we’ll work through the steps behind implementing that model.

We’ll start by creating a stock for the city’s population, with an initial size of 10,000.

# City population is 10,000
CityPop(10000)

Next, we want to initialize the applied drivers stock, and specify a constant rate of 100 people in the city applying to become drivers each round. This will only happen until the 10,000 potential drivers in the city are exhausted, at which point there will be no one left to apply.

# 100 folks apply to become drivers per round
# the @ 100 format is called a "rate" flow
CityPop > AppliedDrivers @ 100

Now we want to initialize the eligible drivers stock, and specify that 25% of the folks in applied drivers will advance to become eligible each round.

Before we used @ 100 to specify a fixed rate. Here we’re useing @ Leak(0.25) to specify the idea of 25% of the folks in applied drivers advancing into eligible driver.

# 25% of applied drivers become eligible each round
AppliedDrivers > EligibleDrivers @ Leak(0.25)

You could write this as @ 0.25, but you’d actually get different behavior, That’s because @ 0.25 is actually short-hand for @ Conversion(0.25), which is similar to a leak but destroys the unconverted portion.

Using an example to show the difference, let’s imagine that we have 100 applied drivers and 100 eligible drivers, and then see the consequences of applying a leak versus a conversion:

Leak(0.25) would end with 75 applied drivers and 125 eligible drivers
Conversion(0.25) would end with 0 applied drivers and 125 eligible drivers

Depending on what you are modeling, you might need leaks, conversions or both.

Moving on, next we model out first right-to-left flow. Specifically, the request missing information flow where some eligible drivers end up not being eligible because they need to provide more information.

# This is "Request missing information", with 10%
# of folks moving backwards each round
EligibleDrivers > AppliedDrivers @ Leak(0.1)

Note that the syntax for left-to-right and right-to-left flows is identical, without making a distinction.

Now, 25% of eligible drivers become onboarded drivers each round.

# 25% of eligible drivers onboard each round
EligibleDrivers > OnboardedDrivers @ Leak(0.25)

Then 50% of onboarded drivers become active drivers, actually providing rides.

# 50% of onboarded drivers become active
OnboardedDrivers > ActiveDrivers @ Leak(0.50)

The active drivers stock is drained by two flows: drivers who voluntarily depart become departed drivers, and drivers who are suspended become suspended drivers. Both flows take 10% of active drivers each round.

# 10% of active drivers depart voluntarily and involuntarily
ActiveDrivers > DepartedDrivers @ Leak(0.10)
ActiveDrivers > SuspendedDrivers @ Leak(0.10)

Finally, we also see 5% of departed drivers returning to driving each round. Similarly, we unsuspend 1% of suspended drivers.

# 5% of DepartedDrivers become active
DepartedDrivers > ActiveDrivers @ Leak(0.05)
# 1% of SuspendedDrivers are reactivated
SuspendedDrivers > ActiveDrivers @ Leak(0.01)

We already sketched this model out earlier, but it’s worth noting that systems will allow you to export models via Graphviz. These diagrams are generally harder to read than a custom drawn one, but it’s certainly possible to use this toolchain to combine sketching and modeling into a single step.

Now that we have the model, we can get to exercise it to learn its secrets.

Exercise

Base model:

Now let’s imagine that our LLM-powered tool can speed up eligible drivers, doubling the speed that we move applied drivers to eligible drivers. Instead of 25% of applied drivers becoming eligible each round, we’ll instead see 50%.

# old
AppliedDrivers > EligibleDrivers @ Leak(0.25)
# new
AppliedDrivers > EligibleDrivers @ Leak(0.50)

Unfortunately, we can see that even doubling the speed at which we’re onboarding drivers to eligible has a minimal impact.

To finish testing this hypothesis, we can eliminate the Request missing information flow entirely and see if this changes things meaningfully, commenting out that line.

Unfortunately, even eliminating the missing information rate has little impact on the number of active drivers. So it seems like the opportunity for our LLM solutions to increase active drivers are going to need to focus on reactivating existing drivers.

Specifically, let’s go from 5% of departed drivers reactivating to 20%.

# 20% of DepartedDrivers become active
# DepartedDrivers > ActiveDrivers @ Leak(0.05)
# DepartedDrivers > ActiveDrivers @ Leak(0.2)

For the first time, we’re seeing a significant shift in impact. We reach a much higher percentage of drivers at peak, and even after we exhaust all drivers in a city, the total number of active reaches a higher equilibrium.

Presumably increasing the rate that we reactivate suspended drivers from 1% to 2.5% would have a similar, meaningful but smaller impact on active drivers over time. So let’s model that change.

# 2.5% of SuspendedDrivers are reactivated
#SuspendedDrivers > ActiveDrivers @ Leak(0.01)
SuspendedDrivers > ActiveDrivers @ Leak(0.025)

However, surprisingly, the impact of increasing the reactivate of suspended drivers is actually much higher than reengaging departed drivers.

This is an interesting, and somewhat counter-intuitive result. Increasing the rate for both suspended and departed rates is more impactful than increasing either, because ultimately there’s a growing population of drivers in the slower deflating stock. This means, surprisingly, that a tool that helps us quickly determine which drivers could be unsuspended might matter more than the small size of the flow indicates.

At this point, we’ve probably found the primary story that this model wants to tell us: we should focus efforts on reactivating departed and suspended drivers. Changes elsewhere might reduce operational costs of our business, but they won’t solve the problem of increasing active drivers.

Modeling impact of LLMs on Developer Experience.

Sun, 06 Oct 2024 04:00:00 -0700

In How should you adopt Large Language Models? (LLMs), we considered how LLMs might impact a company’s developer experience. To support that exploration, I’ve developed a system model of the developing software at the company.

In this chapter, we’ll work through:

Summary results from this model
How the model was developed, both sketching and building the model in a spreadsheet. (As discussed in the overview of systems modeling, I generally would recommend against using spreadsheets to develop most models, but it’s educational to attempt doing so once or twice.)
Exercise the model to see what it has to teach us

Let’s get into it.

Learnings

This model’s insights can be summarized in three charts. First, the baseline chart, which shows an eventual equilibrium between errors discovered in production and tickets that we’ve closed by shipping to production. This equilibrium is visible because tickets continue to get opened, but the total number of closed tickets stop increasing.

Second, we show that we can shift that equilibrium by reducing the error rate in production. Specifically, the first chart models 25% of closed tickets in production experiencing an error, whereas the second chart models only a 10% error rate. The equilibrium returns, but at a higher value of shipped tickets.

Finally, we can see that even tripling the rate that we start and test tickets doesn’t meaningfully change the total number of completed tickets, as modeled in this third chart.

The constraint on this system is errors discovered in production, and any technique that changes something else doesn’t make much of an impact. Of course, this is just a model, not reality. There are many nuances that models miss, but this helps us focus on what probably matters the most, and in particular highlights that any approach that increases development velocity while also increasing production error rate is likely net-negative.

With that summary out of the way, now we can get into developing the model itself.

Sketch

Modeling in a spreadsheet is labor intensive, so we want to iterate as much as possible in the sketching phase, before we move to the spreadsheet. In this case, we’re working with Excalidraw.

I sketched five stocks to represent a developer’s workflow:

Open Tickets is tickets opened for an engineer to work on
Start Coding is tickets that an engineer is working on
Tested Code is tickets that have been tested
Deployed Code is tickets than have been deployed
Closed Ticket is tickets that are closed after reaching production

There are four flows representing tickets progressing through this development process from left to right. Additionally, there are three exception flows that move from right to left:

Testing found error represents a ticket where testing finds an error, moving the ticket backwards to Start Coding
Deployment exposed error represents a ticket encountering an error during deployment, where it’s moved backwards to Start Coding
Error found in production represents a ticket encountering a production error, which causes it to move all the way back to the beginning as a new ticket

One of your first concerns seeing this model might be that it’s embarassingly simple. To be honest, that was my reaction when I first looked at it, too. However, it’s important to recognize that feeling and then dig into whether it matters.

This model is quite simple, but in the next section we’ll find that it reveals several counter-intuitive insights into the problem that will help us avoid erroneously viewing the tooling as a failure if time spend testing increases. The value of a model is in refining our thinking, and simple models are usually more effective as refining thinking across a group than complex models, simply because complex models are fairly difficult to align a group around.

Reason

As we start to look at this sketch, the first question to ask is how might LLM-based tooling show an improvement? The most obvious options are:

Increasing the rate that tasks flow from Starting coding to Tested code. Presumably these tools might reduce the amount of time spent on implementation.
Increasing the rate that Tested code follows Testing found errors to return to Starting code because more comprehensive tests are more likely to detect errors. This is probably the first interesting learning from this model: if the adopted tool works well, it’s likely that we’ll spend more time in the testing loop, with a long-term payoff of spending less time solving problems in production where it’s more expensive. This means that slower testing might be a successful outcome rather than a failure as it might first appear.

A skeptic of these tools might argue the opposite, that LLM-based tooling will cause more issues to be identified “late” after deployment rather than early in the testing phase. In either case, we now have a clear goal to measure to evaluate the effectiveness of the tool: reducing the Error found in production flow. We also know not to focus on the Testing found error flow, which should probably increase.
Finally, we can also zoom out and measure the overall time from Start Coding to Closed Ticket for tasks that don’t experience the Error found in production flow for at least the first 90 days after being completed.

These observations capture what I find remarkable about systems modeling: even a very simple model can expose counter-intuitive insights. In particular, the sort of insights that build conviction to push back on places where intuition might lead you astray.

Model

For this model, we’ll be modeling it directly in a spreadsheet, specifically Google Sheets. The completed spreadsheet model is available here. As discussed in Systems modeling to refine strategy, spreadsheet modeling is brittle, slow and hard to iterate on. I generally recommend that folks attempt to model something in a spreadsheet to get an intuitive sense of the math happening in their models, but I would almost always choose any tool other than a spreadsheet for a complex model.

This example is fairly tedious to follow, and you’re entirely excused if you decide to pull open the sheet itself, look around a bit, and then skip the remainder of this section. If you are hanging around, it’s time to get started.

The spreadsheet we’re creating has three important worksheets:

Model represents the model itself
Charts holds charts of the model
Config holds configuration values seperately from the model to ease exercising the model after we’ve built it

Going to the model worksheet, we want to start out by initializing each of the columns to the starting value.

While we’ll use formulae for subsequent rows, the first row should contain literal values. I often start with a positive value in the first column and zeros in the other columns, but that isn’t required. You can start with whatever starting values are more useful for studying the model that you’re building.

With the initial values set, we’re now going to implement the model in two passes. First, we’ll model the left-to-right flows, which represent the standard development process. Second, we’ll model the right-to-left flows, which represent exceptions in the process.

Modeling left-to-right

We’ll start by modeling the interaction between the first two nodes: Open Tickets and Started Coding. We want to have open tickets increased over time at a fixed rate, so let’s add a value in the config worksheet for TicketOpenRate, starting with 1.

Moving to the second stock, we want to start work on open tickets as long as we have at most MaxConcurrentCodingNum open tickets. If we have more than MaxConcurrentCodingNum tickets that we’re working on, then we don’t start working on any new tickets. To do this, we actually need to create an intermediate value (represented using an italics column name) to determine how many should be created by checking if the current in started tickets is at maximum (another value in the config sheet) or if we should increment that by one.

That looks like:

// Config!$B$3 is max started tickets
// Config!$B$2 is rate to increment started tickets
// $ before a row or column, e.g. $B$3 means that the row or column
// always stays the same -- not incrementing -- even when filled
// to other cells
= IF(C2 >= Config!$B$3, 0, Config!$B$2)

This also means that our first column, for Open Tickets is decremented by the number of tickets that we’re started coding:

// This is the definition of `Open Tickets`
=A2 + Config!$B$1 - B2

Leaving us with these values.

Now we want to determine the number of tickets being tested at each step in the model. To do this, we create a calculation column, NumToTest? which is defined as:

// Config$B$4 is the rate we can start testing tickets
// Note that we can only start testing tickets if there are tickets
// in `Started Coding` that we're able to start testing
=MIN(Config!$B$4, C3)

We then add that value to the previous number of tickets being tested.

// E2 is prior size of the Tested Code stock
// D3 is the value of `NumToTest?`
// F2 is the number of tested tickets to deploy
=E2 + D3 - F2

Moving on to deploying code, let’s keep things simple and start out by assuming that every tested change is going to get deployed. That means the calculation for NumToDeploy? is quite simple:

// E3 is the number of tested changes
=E3

Then the value for the Deployed Code stock is simple as well:

// G2 is the prior size of Deployed Code
// F3 is NumToDeploy?
// H2 is the number of deployed changes in prior round
=G2+F3-H2

Now we’re on to the final stock. We add the NumToClose? calculation, which assumes that all deployed changes are now closed.

// G3 is the number of deployed changes
=G3

This makes the calculation for the Closed Tickets stock:

// I2 is the prior value of Closed Tickets
// H3 is the NumToClose?
=I2 + H3

With that, we’ve now modeled the entire left-to-right flows.

The left-to-right flows are simple, with a few constrained flows and a very scalable flows, but overall we see things progressing through the pipeline evenly. All that is about to change!

Modeling right-to-left

We’ve now finished modeling the happy path from left to right. Next we need to model all the exception paths where things flow right to left. For example, an issue found in production would cause a flow from Closed Ticket back to Open Ticket. This tends to be where models get interesting.

There are three right-to-left flows that we need to model:

Closed Ticket to Open Ticket represents a bug discovered in production.
Deployed Code to Start Coding represents a bug discovered during deployment. 3 Tested Code to Start Coding represents a bug discovered in testing.

To start, we’re going to add configurations defining the rates of those flows. These are going to be percentage flows, with a certain percentage of the target stock triggering the error condition rather than proceeding. For example, perhaps 25% of the Closed Tickets are discovered to have a bug each round.

These are fine starter values, and we’ll experiment with how adjusting them changes the model in the Exercise section below.

Now we’ll start by modeling errors discovered in production, by adding a column to model the flow from Closed Tickets to Open Tickets, the ErrorsFoundInProd? column.

// I3 is the number of Closed Tickets
// Config!$B$5 is the rate of errors
=FLOOR(I3 * Config!$B$5)

Note the usage of FLOOR to avoid moving partial tickets. Feel free to skip that entirely if you’re comfortable with the concept of fractional tickets, fractional deploys, and so on. This is an aesthetic consideration, and generally only impacts your model if you choose overly small starting values.

This means that our calculation for Closed Ticket needs to be updated as well to reduce by the prior row’s result for ErrorsFoundInProd?:

// I2 is the prior value of ClosedTicket
// H3 is the current value of NumToClose?
// J2 is the prior value of ErrorsFoundInProd?
=I2 + H3 - J2

We’re not quite done, because we also need to add the prior row’s value of ErrorsInProd? into Open Tickets, which represents the errors’ flow from closed to open tickets. Based on this change, the calculation for Open Tickets becomes:

// A2 is the prior value of Open Tickets
// Config!$B$1 is the base rate of ticket opening
// B2 is prior row's StartCodingMore?
// J2 is prior row's ErrorsFoundInProd?
=A2 + Config!$B$1 - B2 + J2

Now we have the full errors in production flow represented in our model.

Next, it’s time to add the Deployed Code to Start Coding flow. Start by adding the ErrorsFoundInProd? calculation:

// G3 is deployed code
// Config!$B$6 is deployed error rate
=FLOOR(G3 * Config!$B$6)

Then we need to update the calculation for Deployed Code to decrease by the calculated value in ErrorsFoundInProd?:

// G2 is the prior value of Deployed Code
// F3 is NumToDeploy?
// H2 is prior row's NumToClose?
// I2 is ErrorsFoundInDeploy?
=G2 + F3 - H2 - I2

Finally, we need to increase the size of Started Coding by the same value, representing the flow of errors discovered in deployment:

// C2 is the prior value of Started Coding
// B3 is StartCodingMore?
// D2 is prior value of NumToTest?
// I2 is prior value of ErrorsFoundInDeploy?
=C2 + B3 - D2 + I2

We now have the working flow representing errors in production.

Finally, we can added the Tested Code to Started Coding flow. This is pretty much the same as the prior flow we added, starting with adding a ErrorsFoundInTest? calculation:

// E3 is tested code
// Config!$B$7 is the testing error rate
=FLOOR(E3 * Config!$B$7)

Then we update Tested Code to reduce by this value:

// E2 is prior value of Tested Code
// D3 is NumToTest?
// G2 is prior value of NumToDeploy?
// F2 is prior value of ErrorsFoundInTest?
=E2 + D3 - G2 - F2

And update Started Coding to increase by this value:

// C2 is prior value of Started Coding
// B3 is StartCodingMore?
// D2 is prior value of NumToTest?
// J2 is prior value of ErrorsFoundInDeploy?
// F2 is prior value of ErrorsFoundInTest?
= C2 + B3 - D2 + J2 + F2

Now this last flow is instrumented.

With that, we now have a complete model that we can start exercising! This exercise demonstrated both that it’s quite possible to represent a meaningful model in a spreadsheet, but also the challenges of doing so.

While developing this model, a number of errors became evident. Some of them I was able to fix relatively easily, and even more I left unfixed because fixing them makes the model even harder to reason about. This is a good example of why I encourage developing one or two models in a spreadsheet, but I ultimately don’t believe it’s the right mechanism to work in for most people: even very smart people make errors in their spreadsheets, and catching those errors is exceptionally challenging.

Exercise

Now that we’re done building this model, we can final start the fun part: exercising it. We’ll start by creating a simple bar chart showing the size of each stock at each step. We are going to expressly not show the intermediate calculation columns such as NumToTest?, because those are implementation details rather than particularly interesting.

Before we start tweaking the values , let’s look at the baseline chart.

The most interesting thing to notice is that our current model doesn’t actually increase the number of closed tickets over time. We actually just get further and further behind over time, which isn’t too exciting.

So let’s start modeling the first way that LLMs might help, reducing the error rate in production. Let’s shift ErrorsInProd from 0.25 down to 0.1, and see how that impacts the chart.

We can see that this allows us to make more progress on closing tickets, although at some point equilibrium is established between closed tickets and the error rate in production, preventing further progress. This does validate that reducing error rate in production matters. It also suggests that as long as error rate is a function of everything we’ve previously shipped, we are eventually in trouble.

Next let’s experiment with the idea that LLMs allow us to test more quickly, tripling TicketTestRate from 1 to 3. It turns out, increasing testing rate doesn’t change anything at all, because the current constraint is in starting tickets.

So, let’s test that. Maybe LLMs make us faster in starting tickets because overall speed of development goes down. Let’s model that by increasing StartCodingRate from 1 to 3 as well.

This is a fascinating result, because tripling development and testing velocity has changed how much work we start, but ultimately the real constraint in our system is the error discovery rate in production.

By exercising this model, we find an interesting result. To the extent that our error rate is a function of the volume of things we’ve shipped in production, shipping faster doesn’t increase our velocity at all. The only meaningful way to increase productivity in this model is to reduce the error rate in production.

Models are imperfect representations of reality, but this one gives us a clear sense of what matters the most: if we want to increase our velocity, we have to reduce the rate that we discover errors in production. That might be reducing the error rate as implied in this model, or it might be ideas that exist outside of this model. For example, the model doesn’t represent this well, but perhaps we’d be better off iterating more on fewer things to avoid this scenario. If we make multiple changes to one area, it still just represents one implemented feature, not many implement features, and the overall error rate wouldn’t increase.

Testing strategy: avoid the waterfall strategy trap with iterative refinement.

Wed, 25 Sep 2024 17:00:00 -0700

If I could only popularize one idea about technical strategy, it would be that prematurely applying pressure to a strategy’s rollout prevents evaluating whether the strategy is effective. Pressure changes behavior in profound ways, and many of those changes are intended to make you believe your strategy is working while minimizing change to the status quo (if you’re an executive) or get your strategy repealed (if you’re not an executive). Neither is particular helpful.

While some strategies are obviously wrong from the beginning, it’s much more common to see reasonable strategies that fail because they didn’t get the small details right. Premature pressure is one common cause of a more general phenomenon: most strategies are developed in a waterfall model, finalizing their approach before incorporating the lessons that realities teaches when you attempt the strategy in practice.

One effective mechanism to avoid the waterfall strategy trap is explicitly testing your strategy to refine the details. This chapter describes the mechanics of testing strategy:

when it’s important to test strategy (and when it isn’t)
how to test strategy
when you should stop testing
roles in testing strategy: sponsor vs guide
metrics and meetings to run a testing strategy
how to identify a strategy that skipped testing
what to do when a strategy has progressed too far without testing

Let’s get into the details.

Many of the ideas here came together while working with Shawna Martell, Dan Fike, Madhuri Sarma, and many others in Carta Engineering.

When to test strategy

Strategy testing is ensuring that a strategy will accomplish its intended goal at a cost that you’re willing to pay. This means it needs to happen prior to implementing a strategy, usually in a strategy’s early development stages.

Afew examples of when to test common strategy topics:

Integrating a recent acquisition might focus on getting a single API integration working before finalizing how the overall approach goes.
A developer productivity strategy focused on requiring typing in a Python codebase might start by having an experienced team member type an important module.
A service migration might attempt migrating both a simple component (to test migration tooling) and a highly complex component (to test integration complexity) before moving to a broader rollout.

In every case, the two most important pieces are testing before finalizing the strategy, and testing narrowly with a focus on the underlying mechanics of the approach rather than getting caught up in solving broad problems like motivating adoption and addressing conflicting incentives.

That’s not to say that you need to test every strategy. A few of the common cases where you might not want to test a strategy are:

When you’re dealing with a permissive strategy that’s very cheap to apply, testing is often not too important; indeed, you can consider most highly-permissive strategies as a test of whether it’s effective to implement a similar, but less permissive, strategy in the future.
Where testing isn’t viable for some reason. For example, a hiring strategy where you shift hiring into certain regions isn’t something you can test in most cases, it’s something you might need to run for several years to get meaningful signal on results.
There are also cases where you have such high conviction in a given strategy that it’s not worth testing, perhaps because you’ve done something nearly identical at the same company before. Hubris comes before the fall, so I’m generally skeptical of this category.

That said, my experience is that you should try very hard to find a way to test every strategy. You certainly should not try hard to convince yourself testing a strategy isn’t worthwhile. Testing is so, so much cheaper than implementing a bad strategy, that it’s almost always a good investment of time and energy.

How to test strategy

For a valuable step that’s so often skipped, testing strategy is relatively straightforward. The approach I’ve found effective is:

Identify the narrowest, deepest available slice of your strategy, and iterate on applying your strategy to that slice until you’re confident the approach works well.

For example, if you’re testing a new release strategy for your Product Engineering organization, decide to release exactly one important release following the new approach.
As you iterate, identify metrics that help you verify the approach is working; note that these aren’t metrics to measure adoption, instead that measure impact of the change.

For example, metrics that show the new release process reduces customer impact, or drives more top-of-funnel visitors.
Operate from the belief that people are well-meaning, and strategy failures are due to excess friction and poor ergonomics.

For example, assume the release tooling is too complex if people aren’t using it. (Definitely don’t assume that people are too resistent to change.)
Keep refining until you have conviction that your strategy’s details work in practice, or that the strategy needs to be approached from a new direction.

For example, if the metrics you identified before show the new release process has significantly reduced customer impact of the new release.

The most important details are the things not to do. Don’t go broad where impact feels higher but iteration cycles are slower. Don’t get caught up on forcing adoption such that you’re distracted from improving the underlying mechanics. Finally, don’t get so attached to your current approach that you can’t accept that it might not be working. Testing strategy is only valuable because many strategies don’t work as intended, and it’s much cheaper to learn that early.

Testing roles: sponsors and guides

Sometimes the strategy testing process is lead by one individual who is able to sponsor the work (a principal engineer at a smaller company, an executive, etc) and also coordinate the day-to-day work of validating the approach (a principal engineer at a larger company, an engineering manager, a technical program manager, etc). It’s even more common for these responsibilities to split between two roles: sponsor and a guide.

The sponsor is responsible for:

serving as an escalation point to make quick decisions to avoid getting stuck in development stages
pushing past historical decisions and beliefs that prevent meaningful testing
marshalling cross organizational support
telling the story to stakeholders, especially the executive team to avoid getting defunded
preventing overloading of strategy (where people want to make the strategy solve their semi-related problem)
setting pace to avoid stalling out
identifying when energy is dropping and to change phase of stratey (from development to implementation)

The guide is responsible for:

translating the strategy into particulars where testing gets stuck
identifying slowdowns and blockers
escalating frequently to sponsor
tracking goals and workstreams
maintaining the pace set by the sponsor

In terms of filling these roles, there are a few lessons that I’ve learned over time. For sponsors, what matters the most is that they’re genuinely authorized by the company to make the decision they’re making, and that they care enough about the impact that they’re willing to make difficult decisions quickly. A sponsor is only meaningful to the extent that the guide can escalate to the sponsor and they rapidly resolve those escalations. If they aren’t available for escalations or don’t resolve them quickly, they’re a poor sponsor.

For guides, you need someone who can execute at pace without getting derailed by various organizational messes, and has good, nuanced judgment relevant to the strategy being tested. The worst guides are ideological (they reject the very feedback created by testing) or easily derailed (you’re likely testing because there’s friction somewhere, so someone who can’t navigate friction is going to fail by default).

Meetings & Metrics

The only absolute requirement for the strategy testing phase is that the sponsor, guide, and any key folks working on the strategy must meet together every single week. Within that meeting, you’ll iterate on which metrics capture the current areas you’re trying to refine, discuss what you’ve learned from prior metrics or data, and schedule one-off followups to ensure you’re making progress.

The best version of this meeting is debugging heavy and presentation light. Any week that you’re not learning something that informs subsequent testing, or making a decision that modifies approach to testing, should be viewed with some suspicion. It might mean that you’ve underresourced the testing effort, or that your testing approach is too ambitious, but it’s a meaningful signal that testing is converging too slowly to maintain attention.

If all of this seems like an overly large commitment, I’d push you to consider your strategy altitude to adjust the volume or permisiveness of the strategy you’re working on. If a strategy isn’t worth testing, then it’s either already quite good (which should be widely evident beyond its authors) or it’s probably only worth rolling out in a highly permissive format.

Identifying strategies that skipped testing

While not all strategies must be refined by a testing phase, essentially all failing strategies skip the testing phase to move directly into implementation. Strategies that skip testing sound right, but don’t accomplish much. Fully standardizing authorization and authentication across your company on one implementation sounds right, but can still fail if e.g. each team is responsible for its own approach to determining the standard.

One particularly obvious pattern is something I describe as “pressure without a plan.” This is a strategy that is only the “sounds right” aspect with none of the details. Service migrations are particularly prone to this, perhaps due to apocryphal descriptions of Amazon’s service migration in the 2000s, which is often summarized as a top-down zero-details mandate to switch away from the monolith.

Identification comes down to understanding two things:

Are there numbers that show the strategy is driving the desired impact? For example, API requests made into the new authentication service as a percentage of all authentication requests is more meaningful than a spreadsheet tracking teams’ commitments to move to the new service.

Try to avoid proxy metrics when possible, but to instead look at the actual thing that matters.
If the numbers aren’t moving, is there a clear mechanism debugging and solving those issues, and is this team actually making progress? For example, a team who help integration with the new authentication service to understand where limitations are preventing effective adoption, and who are shipping working code.

Because the numbers aren’t moving, you need to find a different source of meaningful evidence to validate that progress is happening. Generally the best bet is either new software running in a meaningful environment (e.g. production for product code). It’s also useful to talk with skeptics or failed integrations, but be cautious of debugging exclusively with skeptics.
They’re almost always right, but often they are out-of-date, such that they’re right but aren’t describing current problems.

Unless one of these two identifications are obviously true, then it’s very likely that you’ve found a strategy that skipped testing.

Recovering from skipped testing

Once you’ve recognized a strategy that skipped testing and is now struggling, the next question is what to do about it. Should we decompose our monolith? looks at recovering from a failing service migration, and is lightly based on my experience dealing with similar, stuck service migration at both Calm and Carta. The answer to a stuck strategy is always: write a new strategy, and make sure not to skip testing this time.

Typically, the first step of this new strategy is explicitly pausing the struggling strategy while a new testing phase occurs. This is painful to do, because the folks invested in the current strategy will be upset with you, but there’s always going to be people who disagree with any change. Long-term, the only thing that makes most people happy is a successful strategy, and anything that delays progress towards is a poor investment.

Sometimes it is difficult to officially pause a struggling strategy, in which case you have to look for an indirect mechanism to implicitly pause without acknowledging it. For example, delaying new services while you take a month to invest into improving service provisioning might give you enough breathing room to test the missing mechanisms from your strategy, without requiring anyone to lose face that the migration is failing. It would be nice to always be able to say these things out loud, but managing personalities is an enduring leadership challenge; even when your an executive, you just have a different set of messy stakeholders.

Summary

Testing doesn’t determine whether a strategy might be good. It exposes the missing details required to translate a directionally accurate strategy into a strategy that works. After reading this chapter, you know how to lead that translation process as both a sponsor and a guide. You can setup and run the necessary meetings to test a strategy, and also put together the bank of metrics to determine if the strategy is ready to leave refinement and move to a broader rollout.

Should we decompose our monolith?

Sun, 15 Sep 2024 06:00:00 -0700

From their first introduction in 2005, the debate between adopting a microservices architecture, a monolithic service architecture, or a hybrid between the two, has become one of the least-reversible decisions that most engineering organizations make. Even migrating to a different database technology is generally a less expensive change than moving from monolith to microservices or from microservices to monolith.

The industry has in many ways gone full circle on that debate, from most hyperscalers in the 2010s partaking in a multi-year monolith to microservices migration, to Kelsey Hightower’s iconic tweet on the perils of distributed monoliths:

2020 prediction: Monolithic applications will be back in style after people discover the drawbacks of distributed monolithic applications. - @KelseyHightower

Even as popular sentiment has generally turned away from microservices, many engineering organizations have a bit of both, often the reminents of one or more earlier but incomplete migration efforts. This strategy looks at a theoretical organization stuck with a bit of both approaches, let’s call it Theoretical Compliance Company, which is looking to determine its path forward.

Here is Theoretical Compliance Company’s service architecture strategy.

Reading this document

More detail on this structure in Making a readable Engineering Strategy document.

Policy

Our policy for service architecture is documented here. All exceptions to this policy must escalate to a local Staff-plus engineer for their approval, and then escalate with that Staff-plus engineer to the CTO. If you have questions about the policies, ask in #eng-strategy.

Our policy is:

Business units should always operate in their own code repository and monolith. They should not provision many different services. They should rarely work in other business units monoliths. There are nuances in the details: make decisions that bring us closer to the preceeding sentence being true.
New integrations across business unit monoliths should be done using gRPC. The emphasis here is on new integrations; it’s desirable but not urgent to migrate existing integrations that use other implementations (HTTP/JSON, etc).

When the decision is subtle (e.g. changes to an existing endpoint), optimize for business velocity rather than technical purity. When the decision is far from subtle (e.g. brand new endpoint), comply with the policy.
Except for new business unit monoliths, we don’t allow new services. You should work within the most appropriate business unit monolith or within the existing infrastructure repositories. Provisioning a new service, unless it corresponds with a new business unit, always requires approval from the CTO in #eng-strategy.

That approval generally will not be granted, unless the new service requires significantly different non-functional requirements than an existing monolith. For example, if it requires significantly higher compliance review prior to changes such as our existing payments service, or if it requires radically higher requests per second, and so on.
Merge existing services into business-unit monoliths where you can. We believe that each choice to move existing services back into a monolith should be made “in the details” rather than from a top-down strategy perspective. Consequently, we generally encourage teams to wind down their existing services outside of their business unit’s monolith, but defer to teams to make the right decision for their local context.

Diagnose

Theoretical Compliance Company has a complex history with decomposing our monolith. We are also increasing our number of business units, while limiting our investment into our core business unit. These are complex times, with a lot of constraints to juggle. To improve readability, we’ve split the diagnosis into two sections: “business constraints” and “engineering constraints.”

Our business constraints are:

We sell business-to-business compliance solutions to other companies on an annual subscription. There is one major, established business line, and two smaller partially validated business lines that are intended to attach to the established business line to increase average contract value.
There are 2,000 people at the company. About 500 of those are in the engineering organization Within that 500, about 150 work on the broadest definition of “infrastructure engineering,” things like developer tools, compute and orchestration, networking, security engineering, and so on.
The business is profitable, but revenue growth has been 10-20% YoY, creating persistent pressure on spend from our board, based on mild underperformance relative to public market comparables. Unless we can increase YoY growth by 5-10%, they expect us to improve free cash flow by 5-10% each year, which jeopardizes our ability to maintain long-term infrastructure investments.
Growth in the primary business line is shrinking. The company’s strategy includes spinning up more adjacent business units to increase average contract value with new products. We need to fund these business units without increasing our overall budget, which means budget for the new business units must be pulled away from either our core business or our platform teams.

In addition to needing to fund our new business units, there’s ongoing pressure to make our core business more efficient, which means either accelerating growth or reducing investment. It’s challenging to accelerate growth while reducing investment, which suggests that most improvement will come from reducing our investment
Our methodology to allocate platform costs against business units does so proportionately to the revenue created by each business unit. Our core business generates the majority of our revenue, which means it is accountable for the majority of our platform costs, even if those costs are motivated by new business lines.

This means that, even as the burden placed on platform teams increases due to spinning up multiple business units, there’s significant financial pressure to reduce our platform spend because it’s primarily represented as a cost to the core business whose efficiency we have to improve. This means we have little tolerance for anything that increases infrastructure overhead.

Our engineering constraints are:

Our infrastructure engineering team is 150 engineers supporting 350 product engineers, and it’s certain that infrastructure will not grow significantly in the forseeable future.
We spun up two new business units in the past six months, and plan to spin up an additional two new business units in the next year. Each business unit is lead by a general manager, with engineering and product within that business unit principally accountable to that general manager. Our CTO and CPO still set practice standards, but it’s situationally-specific whether these practice standards or direction from general manager is the last word on any given debate.

For example, one business unit has been unwilling to support an on-call rotation for their product, because their general manager insists it is a wasteful practice. Consequently, that team often doesn’t respond to pages, even when their service is responsible for impacting the stability of shared functionality.
We’ve modeled how services and monoliths create overhead for both product and infrastructure organizations over time, and have conviction that, in general, it’s more overhead for infrastructure to support more services. We also found that in our organization, the rate of service ownership changing due to team reorganizations counteract much of the initial productivity gains from leaving the monolith.
There is some tension between the two preceeding observations: it’s generally more overhead to have more services, but it’s even more overhead to have unresponsible business units breaking a shared monolithic service. For example, we can much more easily ratelimit usage from a misbehaving service than wrong a misbehaving codepath within a shared service.
We also have a payments service that moves money from customers to us. Our compliance and security requirements for changes to this service are significantly higher than for the majority of our software, because the blast radius is essentially infinite.
Our primary programming language is Ruby, which generally relies on blocking IO, and service-oriented architectures generally spend more time on blocking IO than monoliths. Similarly, Ruby is relatively inefficient at serializing and deserializing JSON payloads, which our service architecture requires as part of cross-service communication.
We’ve previously attempted to decompose, and have a number of lingering partial migrations that don’t align cleanly with our current business unit ownership structure. The number of these new services continues to grow over time, creating more burden on both infrastructure today and product teams in the future as they try to maintain these services through various team reorganizations.

Explore

In the late 2010s, most large or scaling companies adopted services to some extent. Few adopted microservices, with the majority of adopters opting for a service-oriented architecture instead. Kelsey Hightower’s iconic tweet on the perils of distributed monoliths in 2020 captured the beginning of a reversal, with more companies recognizing the burden of operating service-oriented architectures.

In addition to the wider recognition of those burdens, many of the cloud infrastructure challenges that originally motivated service architectures began to mellow. Most infrastructure engineers today only know how to operate with cloud-native patterns, rather than starting from machine oriented approaches. Standard database technologies like PostgreSQL have significantly improved capabilities. Cloud providers have fast local caches for quickly retrieving verified upstream packages. Supply and cost of cloud compute is affordable. Slow programming languages are faster than they were a decade ago. Untyped languages have reasonable incremental paths to typed codebases.

As a result of this shift, if you look at a new, emerging company it’s particularly likely to have a monolith in one backend and one frontend programming language. However, if you look at a five-plus year old company, you might find almost anything. One particularly common case is a monolith with most functionality, and an inconsistent constellation of team-scoped macroservices scattered across the organization.

The shift away from zero interest-rate policy has also impacted trends, as service-oriented architectures tend to require more infrastructure to operate efficiently, such as service meshes, service provisioning and deprovisioning, etc. Properly tuned, service-oriented architectures ought to be cost competitive, and potentially superior in complex workloads, but it’s hard to maintain the required investment in infrastructure teams when in a cost-cutting environment. This has encouraged new companies to restrict themselves to monolithic approaches, and pushed existing companies to attempt to reverse their efforts to decompose their prior monoliths, with mixed results.

Executive translation.

Sat, 07 Sep 2024 08:00:00 -0700

One of my most unexpectedly controversial posts is Extract the Kernel, which argues that executives are generally directionally correct but specifically wrong, and it’s your job to understand the overarching direction without getting distracted by the narrow errors in their idea.

Some executives are skeptical of this idea because they don’t like the implication that they’re usually wrong, but they weren’t the audience that was offended. But the folks who got particularly upset were non-executives who felt it was unfair for them to have to debug the executives’ communication. The fair solution, some argued, is for the executives to become better communicators rather than requiring others around them to become better listeners. For what it’s worth, I agree with them, that would be more fair, but I’ve always found it much more productive to focus on how I can improve my approach than to document ways that others could improve theirs.

Recently I’ve been repeating a similar idea to “Extract the kernel”, but rather than focusing on understanding executives, it’s instead focused on leading change when working with executives. Often you’ll hear an executive say something that you disagree with, and in that moment you have to determine whether you can directly steer the executive towards a better decision. If you can, then do that! If you can’t, then focus on translating the executive’s idea into something useful!

Many high-agency managers try to prevent executives from doing silly things, but it’s almost always more effective to translate their energy for a silly thing into energy for a useful thing. It also leaves the executive feeling supported by your work rather than viewing you as an obstacle to their progress.

Some examples:

Executive is obsessed with adopting LLMs in your product. Translate that into a useful approach rather than fighting that LLMs aren’t useful in most products.
Executive wants to expand into a new business unit without any additional hiring. Use a hackathon to get a sample concept that you can use to validate the business with users.
Executive wants to do a giant rewrite of your product. Translate into a narrow test rewrite of a small feature to support quickly refining the approach.

In each of these cases, the executive’s idea is more likely to succeed based on your actions. The idea is also more likely to fail quickly. In both cases, you’ll have worked to support the executive and the company in a thoughtful, effective way. (These are all good examples of making effective multi-dimensional tradeoffs!)

Video of Developing Eng Leadership Styles.

Sat, 07 Sep 2024 07:00:00 -0700

The last chapter I wrote for Eng Executive’s Primer was this one about developing engineering leadership styles. It’s an interesting chapter to me peronally, precisely because it’s not something I would have agreed with or written five years ago.

This past Friday I gave a conference talk on this topic at LeadingEng New York, 2024. If you’re interested, you can watch a recording of an earlier practice session from a few days before the talk, and can review the slides. I think that the practice session is quite a bit worse than the final talk, which I believe is restricted to LeadingEng attendees.

You can also watch the talk on Youtube directly. The content is similar to the chapter, but takes a bit of a different angle in exploring it.

(Photograph from my former colleague @shidoshi on X.)

Regarding the conference itself, this was my first LeadDev conference that I’d been to, and it was an extremely well put together conference. I personally enjoyed the mix of 20 minute talks (one of which I gave) and 60 minute deliberate working sessions (e.g. Ashley Miller guided through an exercise on honing commercial awareness, and Cat Miller steered refining technical strategy). The working sessions created a nice, ongoing peer learning experience, that reminded me of a transient version of the CTO circle that I run with Uma Chingunde. The working sessions all had written prompts in high-quality printed notebooks, which I thought was quite nice as well, and makes me want to run my next offsite with printed prompts.