Irrational Exuberance

2024 in review.

Sat, 14 Dec 2024 05:00:00 -0700

A lot happened for me this year. I continued learning the details of fund accounting at Carta, which is likely the most complex product domain I’ve worked in. My third book was published, and I did a small speaking tour to support it. We started the unironically daunting San Francisco kindergarten application process. I was diagnosed with skin cancer and had successful surgery to remove it. All things considered, it was a much messier year than I intended, but with many good pockets mixed in with the mess.

(I love to read other folks year-in writeups – if you write one, please send it my way!)

Previously: 2023, 2022, 2021, 2020, 2019, 2018, 2017

Goals

Evaluating my goals for this year and decade:

[Completed] Write at least four good blog posts each year.

How to create software quality, Layers of context, Useful tradeoffs are multi-dimensional, Notes on how to use LLMs in your product, Eng org seniority-mix model.
[Completed] Write another book about engineering or leadership.

I did this in either 2023 or 2024, as I released The Engineering Executive’s Primer this year, and finishing writing it late last year. Either way, it’s complete.
[New] Write three books about engineering or leadership in 2020s.

I’ve already written two this decade, so committing to writing one more feels pretty attainable. At that point, I’ll have written four total including An Elegant Puzzle in 2019, and I’m pretty sure I will be “booked out,” perhaps permanently, but I think I have one more good one left in me on the topic of engineering strategy.
[Completed] Do something substantial and new every year that provides new perspective or deeper practice.

Started working with a physical trainer for first time, significantly helping me improve on a number of my lifts. On a totally different dimension, I finally spent time to learn Wardley mapping which is something I’ve been intending to do, but not doing, for at least five years.
[In progress] 20+ folks who I’ve managed or meaningfully supported move into VPE or CTO roles at 50+ person or $100M+ valuation companies.

This is a decade goal ending in 2029. I previously increased the goal in 2022 from 3-5 to 20. A strict count is at 10 today, so I think I’m on track.

For backstory on these goals: I originally set them in 2019, and then revised them in 2022. I’ve generally come to the point of view that I should be revising these every year, but also not sure it’s worth doing it. Maybe one day!

Published Engineering Executive’s Primer

My third book was published in March, The Engineering Executive’s Primer. I wrote up notes on publishing it with O’Reilly, and I’m altogether very happy to have written and published it. I’m also glad that work is in the past, rather than the present, as finishing a book is a fair amount of work to juggle with a full-time job and parenting.

Relative to my last two books, this is a bit more of a niche topic, so my mental model for sales was that they’d likely be a bit lower, which seems to be accurate so far, with a bit over 10,000 copies sold through October. The real question for a book like this is whether it maintains sales after the first six-month bump, which I’ll be watching with curiosity. It may also be true that book sales are not a particularly effective way to evaluate a book like this, which aims to advance the state of the industry by reaching decision makers within the industry. That said, it’s hard not to evaluate your book that way, even when you know that you shouldn’t.

As an aside, determining accurate numbers for O’Reilly titles is a bit tricky. There are things like the online version of the book, where you’re paid by usage (similar to Amazon Kindle Unlimited) but don’t have a non-dollar indicator of usage. I’m similarly unclear how audiobook usage is calculated, as it doesn’t show up in my royalties. To deal with all that, I’ve just focused on sold physical books and ebooks.

Other book updates

A few other book related updates:

I broke through 1,000 ratings on Amazon for Staff Engineer this year, which is a cool milestone to pass for two distinct books. (AEP passed 1,000 a year or two ago.)
Both earlier books continued to sell fairly well. I believe An Elegant Puzzle passed 100k copies sold either late last 2023 or early 2024 (the sales data I get is not particularly high granularity), which is a nice number. Staff Engineer is also doing well, passing 89k copies sold as of today. I think it’ll pass 100k sometime in 2026, assuming it can mostly maintain its velocity from over the last couple years.
I wrote the High-Context Triad which I view as an additional set of chapters for a second edition of Staff Engineer whenever I get to that project, probably my second side project in the queue after finishing the engineering strategy book. Of those chapters, I think Useful tradeoffs are multi-dimensional and Layers of context are particularly useful.
I’ve been pulling together notes and writing on engineering strategy, which has a real chance of turning into my next book, but there’s no guarantee on these things, e.g Infrastructure Engineering certainly went off the rails.

Public speaking, etc

Generally, I am not very focused on public speaking, but my stance on this topic changes the years when I publish a new book, and I did quite a bit of speaking, writing, podcast attendee-ing, and so on. I was featured on Gergely Orosz’s Pragmatic Engineer mailing list, First Round Review, and Lenny Rachitsky’s Lenny’s Newsletter.

I also gave talks at:

If you’re wondering why I do more speaking on the years I publish new books, it’s pretty straightforward: it helps sell more copies of the books, and books live on momentum. A good start in the first year contributes forever to its sales, as long as it’s a timeless book (“engineering leadership”) rather than a timely one (“details about the current version of a database”).

Videos

I’ve been playing around with creating YouTube videos for my conference talks. My approach is pretty basic: whenever I give a talk, I record a practice run ahead of time, and then post it online after I’ve given the talk at the conference. I don’t ever expect my YouTube channel to go anywhere–hence the low production values–but I do think it’s worthwhile to keep recordings as conference videos often disappear. Worst case, this means I have some recording of my talks since late 2023.

Health

This has been a pretty challenging year for me from a health perspective. I had two major vacations planned this year, and both got derailed by health concerns. The first I was recouperating with stitches from successful skin cancer surgery, and the second I was in bed with a fever for five day straight until my doctors were convinced my sickness was bacterial.

The skin cancer diagnosis was unexpected, and was a pretty unsettling several months to go from detection through surgery. As I write this, I am treating myself with topical chemotherapy to prevent another swath of skin requiring surgery in the future, which is far less serious than it sounds, but unsettling nonetheless. The good news is that a few months out, I am likely on the other side of this batch of skin cancer risk, and one large scar on my neck is a small price to pay for good health.

Probably the most challenging thing with the surgery and recovery, sickness and recovery, and so on, is that it just throws off the larger health routine. For the first time in my life, I’m working with a personal trainer, which has been quite nice. I’ve been an off and on lifter for the past several decades, and feel comfortable withi most lifts, but there have been a handful where I’ve just never quite gotten the form right, and I’m finally getting it now after working with my trainer. Similarly, there are some mobility issues preventing e.g. deep squatting, which I’ve certainly not fixed, but I’m able to better understand and make progress on.

I’m still running 1-2 times a week, although I’ve dropped milleage down to 4 mile runs, and playing basketball for a couple hours each weekend. I’m hopeful I’ll be able to both extend milleage and incorporate a speed session, but I’m honestly struggling to find the time (or perhaps, energy) to make those happen. Fingers crossed for 2025.

Kindergarten search in San Francisco

Without saying too much, I can say that I’ve never experienced anything like the San Francisco process of applying to public and private kindergartens. That said, the schools themselves are just remarkable. Looking back at the schools I went to in North Carolina… comparing them to these schools is almost impossible. I’m excited to see my son start next year, and will be excited to have this process behind us in a few months.

Tracking family cashflow

It’s slightly embarassing to admit, but I’ve had a relatively bad grasp of the details of my financial situation over the past few years. Specifically, I was very comfortable with how we were invested, but was having a hard time with the details of cash management. How much were we cash flowing? How much were our high spend months the result of obvious annual expenses (e.g. property tax) versus one-off purchases?

Our income streams are sufficiently complex, and I shuffle money between several bank accounts (one for bills, one for higher interest), that answering these questions had generally gotten beyond my ability to answer with conviction. This is a problem, because I’m the family CFO, and I became increasingly uncomfortable that I didn’t understand the financial consequences of various theoretical scenarios. For example, if I lost my job (or died, I suppose), where would that leave my family?

I wanted to reclaim my confidence in our finances this year, and did a fair amount of research to determine a tool that would make this possible. Ultimately, I went with Monarch Money (but do your own research before picking one!), and was pretty impressed with how quickly I was able to correctly represent our cashflows. The most important part was marking the various flows to and from bank and investment accounts as transfers rather than in or outflows of cash.

Altogether, I probably spent four or five hours over a week to get it fully configured, and now I can see my family’s cashflow for the first time in my life. This is pretty powerful, and now I have a pretty clear understanding of how much we’re spending (a bit more than I realized), and how much we would need to have invested to fully cover that spend. This has also made it much easier to sit down once or twice a year as a family and update our mental models on how things are going.

Reading

The profession-adjacent related reading I did this year (by which I mean, most of my reading is fiction that I don’t include here):

Financial Accounting, 11th Edition by Weygandt, Kimmel, and Kieso
Hardcore Software by Steven Sinofsky
The Big Score by Michael S. Malone
How Asia Works by Joe Studwell
Loonshots by Safi Bahcall
Chip War by Chris Miller
Practical TLA+: Planning Driven Development by Hillel Wayne (I thought that I’d read this previously, but I’m pretty sure I’d just bought the Kindle version and struggled to get into it in that format. I actually read it this time as a paperback book.)
Architecture Modernization by Nick Tune with Jean-Georges Perrin
Superforecasting by Philip Tetlock and Dan Gardner
Accounting Made Simple by Mike Piper
Partnership Accounting - Financial Accounting by Robert Steele (This is a Udemy course, not a book, but very much professional content.)
Team Topologies by Matthew Skelton and Manuel Pais
Super Founders by Ali Tamaseb
Zero to One by Peter Thiel
Poor Charlie’s Almanack by Charles T. Munger
Domain-Driven Software Distilled by Vaughn Vernon
How Will You Measure Your Life? by Christensen, Allworth, and Dillon
Write Useful Books by Rob Fitzpatrick
Building Green Software by Anne Currie, Sarah Hsu, and Sara Bergman
Unit X by Raj Shah and Christopher Kirchhoff
What You Do Is You Who Are by Ben Horowitz
Read Write Own by Chris Dixon
Co-intelligence: Living and Working with AI by Ethan Mollick
The Cold Start Problem by Andrew Chen
The End of Loyalty by Rick Wartzman
You Deserve a Tech Union by Ethan Marcotte
Build by Tony Fadell

Measuring developer experience, benchmarks, and providing a theory of improvement.

Sun, 08 Dec 2024 07:00:00 -0700

Back in 2020, I wrote a piece called My skepticism towards current developer meta-productivity tools, which laid out my three core problems with developer productivity measurement tools of the time:

Using productivity measures to evaluate rather than learn
Instrumenting metrics required tweaks across too any different tools
Generally I found tools forced an arbitrary, questionable model onto the problem

Two and a half years later, I made an angel investment in DX, which at the time I largely viewed as taking a survey-driven, research-backed approach to developer productivity. I was recently chatting with Abi Noda, co-founder at DX, and thought it would be an interesting time to revise my original thesis both generally and in response to DX’s release of DX Core 4.

Checklist for developer meta-productivity tools

In 2024, I think this is the checklist for a world-class developer meta-productivity tool:

Pull metrics in from most vendors (e.g. Github) automatically. Support converting metrics from primary observability systems (e.g. Datadog). Support custom emitting metrics.
Automatically graph those metrics in a consistent fashion. This includes denormalizing the data across vendors, such that those relying on standard vendors do near-zero customization. (Implicitly this also means rejecting measurements that are inherently complex, looking at you, lead time.)
Plot values against comparable benchmark data, ensuring that all these values have high “judge-ability” such that you, your CTO, or your Board, can judge your company’s performance relative to peer companies.
Provide a “theory of improvement” such that you can roughly translate your scores into the projects to invest in. These should be as narrow as possible. A great example would be: 90% of your incidents involve one of these five files; you should prioritize hardening their contents.

A broader example could be along the lines of: generally teams that score highly on this dimension utilize these practices (e.g. trunk-based development), and those that score lower use these other practices (e.g. long-lived branch development).

I think if you can provide all four of those, then you have provided a highly valuable, effective tool.

Implicitly a network product

This is something of an aside, but I particularly believe that there’s an important breakpoint between products that offer only the first two (ingestion and charting) from products that offer the third (benchmarking). This is because I increasingly believe that developer data is only judge-able in the context of benchmarks, and you can only provide benchmarks if you have a network of companies sharing their data in a common place. Also, they need to be sharing it in a way where the data itself is relatively verifiable (there’s a long history of inaccurate self-supplied datasets powering products that gradually undermine user trust).

If you believe that benchmarks are foundational, then the winning generational developer meta-productivity tool is a network product, and that means that only one or two winners will exist in the exosystem. (Because only so many companies can be vendors to enough of these companies to build good benchmarks.) This is a significant shift away from my earlier thinking here, because now there’s an underlying mechanism to force this market to converge rather than remain fragmented across thousands of small vendors.

Progress from 2020 to 2024

Before testing my four new criteria against recent developments in the developer productivity space, I wanted to do a bit of Wardley mapping to better explain where I’ve seen challenges in these tools, and why I think DX’s approach is relatively novel.

There are three users to consider: the engineer, the developer productivity team, and the engineering executive. The engineer just wants the developer productivity team to do useful things for them, and to spend the minimum amount of time maintaining data integrations for developer productivity. The engineering executive wants reports which help them evaluate the efficiency of their overall engineering organization, which they will use for resourcing decisions and for reporting upwards to the board on their progress. The developer productivity team wants to provide all of those services, and to make targetted investments into improving the development experience.

In 2020, my perspective was that dashboarding and instrumentation remained custom efforts, and that the current wave of tooling aimed to solve that problem through better ingestion of data, which brings us to this second phase of the Wardley map.

As dashboard creation and metric collection became more standardized, you would assume that these tools got significantly more valuable. This seemed like great progress, but it revealed, in my opinion, a misunderstanding about how these metrics are useful.

It’s not particularly interesting to set a goal to increase pull requests per engineer by 10% this half: there’s only so much insight you can get from trying to manipulate these lines in isolation. But it is very interesting to know that you are significantly behind pull requests per engineer of similarly sized companies. (And even more interesting to see where you fall into a scatterplot of those similarly sized companies.)

Importantly, when you think about the engineering executive, it is only the benchmarks which help them adequately report to their board about the impact of their engineering team’s execution: all the other reports don’t solve the engineering executive’s problem at all. Generally, you can provide any reasonable measure to your board, because they are more focused on ensuring you’re capable of instrumenting and maintaining something than on what you instrument itself, but with benchmarks you can actually provide a meaningful way to measure and evaluate your team’s work! And, to the extent the benchmarking is maintained by a vendor rather than something you have to collect informally from peers, it becomes easier to provide a meaningful comparison between your organization and peers than to provide something less meaningful.

The first wave of tooling vendors thought they could solve the engineering productivity problem with better dashboarding, but rather it’s the centralization of benchmark creation that’s the truly valuable part for the most important engineering executive user.

Summarizing a bit, the core insight behind DX’s approach is that standardizing the benchmarking process across companies, such that their customers can benefit from those benchmarks without collecting them themselves, is exceptionally more valuable than the dashboarding or collection components themselves. In many senses, this is applying the learnings from CultureAmp to a new space. It also means that being a standalone offering, rather than an owned component of a larger metrics platform within Google or Github, expands the addressible companies that can be included for benchmarking.

Lastly, if you accept that benchmarking is the most important feature, it means that your number one criteria for things to measure is your ability to consistently include them into a benchmark across many different companies.

Should metrics provide a Theory of Improvement?

The biggest reason I love benchmarks is that any metric with good benchmarks has high judge-ability. Judge-ability is whether you can judge if something is going well by understanding the value over time. Almost all metrics meet this criteria if you show your value for that metric within in a scatterplot of values for peer companies. For example, DX Core 4 suggests “diffs per engineer” as the metric to evaluate engineering speed, and while this is an extremely hard metric to evaluate in isolation, it’s very easy to look at your organization’s diffs per engineer against peer companies’ diffs per engineer and make an informed judgment about whether things are going well for you.

Conversely, I’m much less confident that the goal-ability of diffs per engineer is high. Goal-ability is whether you can usefully set a target based on changing your current value. For example, we’re showing 2.5 diffs per engineer per week, and we want to increase that to 3.5 diffs. Is that a good goal? Do we have a theory of improvement that would guide how we should approach the work?

I think it’s probably not a good goal, and we probably don’t have a theory of improvement, because this is an output metric (also known as a lagging metric). Output metrics are very effective at measuring if something has improved, but very bad at focusing attention towards improving that metric. Revenue is the ultimate output metric: yes, if your new product feature drives revenue, it’s accomplishing something valuable, but merely having a goal for revenue doesn’t provide any direction on how to accomplish that usefully.

That means if your developer productivity metrics are all output metrics, then you still haven’t provided a “theory of improvement” to teams trying to determine their roadmap to improve on their own metrics. (In general, lacking a theory of improvement is equivalent to not having a relevant strategy for the problem at hand.)

All of the DX Core 4 metrics are good output metrics, which allow you to evaluate your organization across dimensions to determine if you’re operating effectively, but none of them have a strong point of view on what you should do about it. This is one of the triumphs of Accelerate, which provided not only the output metrics to evaluate your organization, but also provided a set of practices that correlated highly with stronger performance.

It’s possible that providing a theory of improvement is asking too much from a batch of metrics. For example, at Carta in the past year I’ve spent a fair amount of time working with a cross-team group of engineers on improving code quality. We followed the working modeled I describe as “strategy testing”, and it took us a while to narrow in on what we believe was the ultimate problem: frontend testing support for one portion of our codebase was too challenging, and contributed to fewer frontend tests than necessary for maintaining high code quality within a complex product. It took us about three months of hypothesis forming and testing before we had confidence this was the foundational issue to focus on, and some of our earlier theories of improvement were meaningfully wrong–hence the need for strategy testing before committing to any given approach.

Perhaps the biggest win in the increasingly standardized productivity metrics are allowing developer productivity teams to shift away from the view that maintaining those metrics is their core work, and to instead recognize that refining their company’s theory of improvement for developer productivity is the much more valuable opportunity to work on instead.

Commentary on the actual metrics themselves

Taking some time to comment on the core metrics themselves, I’m reminded that there’s a Reddit thread where someone says they immediately stopped reading my first book because I suggested PRs per engineer as a bad but usable starting place for starting to understand how your engineers worked. It’s sort of funny to see that diffs per engineer is DX Core 4’s first metric.

In general, I find the core metrics to be very reasonable. The only top-level metric I’d directly quibble with is % of time spent on new capabilities which is a bit of a tracking nightmare in my personal experience, but it certainly is a powerful metric if you believe it’s accurate.

There are a number of others that I haven’t seen before, but which are in retrospect obviously useful and easy to measure, such as:

Time to 10th PR is a fantastic onboarding metric, and not one that I’d considered before. In particular it’s quite easy to measure, and easy to benchmark across many different companies.
R&D as a % of revenue is a foundational chart that boards like to use to evaluate you, you should be measuring yourself on it as well.

Using a synthetic/composite metric of Developer Experience Index (DXI) is both a smart move, and also an expensive one. The upside is that it allows them to include many different components into it, and to revise the composite metric over time as research evolves. For example, they could ask new questions in their surveys, and weigh them into DXI numbers calculated after the first survey including those results, and so on, without impacting the validity of their benchmarks.

The downside is that no one understands composite metrics, and it means teams will end up explaining, and explaining, and explaining what the metric means. Even then, skeptical people just don’t trust composite metrics particularly well, and both executives and boards are heavily staffed by skeptical people. That said, presenting the data within the context of a scatterplot ought to address many of the concerns about composite metrics, so it’s a reasonable experiment from my perspective.

2025+

Thinking about where the developer productivity space is today, there’s a few things that I hope are true over the next couple of years. First, I hope that framework proliferation slows down a bit. DORA, SPACE, DX Core 4 and so on are all great frameworks, but hopefully we reach a point where creating another framework is obviously not worth it. I think DX Core 4 could particularly contribute to this by emphasizing the calculation mechanism for DXI, and ensuring that access to that calculation is predictably preserved for even competitive companies to use.

Second, I’d love to see every one of my four criteria for a world-class developer meta-productivity tool move into the “product” segment of the Wardley map and entirely out of the “custom” section. That means that these tools will need to not only provide metrics collection, dashboarding, and benchmarks out-of-the-box, but they’ll also need to provide a theory of improvement. I don’t think this is out of reach, e.g. you can do a series of studies on companies that moved from low-to-high in each of the metrics and understanding what they prioritized to do so, and from that you can generate broad recommendations. Then you can work on incrementally sharpening those recommendations to be more and more directly applicable to each customer.

If that could happen, I think we reach a pretty remarkable place where developer productivity teams are primarily doer teams rather than analysis teams, which is a foundational shift from those teams’ roles in the industry today.

Rough notes on learning Wardley Mapping.

Sat, 30 Nov 2024 07:00:00 -0700

In my ongoing efforts to draft a book on engineering strategy, I’ve finally reached the point where I need to transition “Wardley Mapping” from a topic to consider including into a topic that I either do or do not include. The first step on that line is getting much deeper at understanding how it works. This is rather different than systems modeling, which is a technique I’ve been using off-and-on at work for 15-plus years, but I will make a solid go at it.

My starting point is having read The Value Flywheel Effect by David Anderson, and attempted a few times unsuccessfully to read Wardley Maps by Simon Wardley on Medium. Let’s see how it goes.

Markus pointed me towards his excellent list of Wardley mapping resources
What Is Wardley Mapping? by Mike Lamb is a quick, very high-level introduction to the concepts.
The Easiest Way to Do Wardley Mapping (A Tutorial) is a good first starting point.

Within this example, there are two templates to work from: lists & value chain template and Wardley map template. You can then use the template in whatever diagramming tool you prefer, the video uses Miro, and I used OmniGraffle.

It also links to this table for determining stage of a given activity within a Wardley Map, and this table describing opportunities. Both of these are useful in various steps of creating/refining a Wardley Map.
How to read a Wardley Map. Bit of a repeat of above.
The Problem with Wardley Mapping. Talks about the various ways that people often get stuck using Wardley Maps: taking “all or nothing” approaches to both method and scope. Rooted in this article, Nobody Cares About Your Precious Framework.

First, it describes the number of “Wardley mapping practices” and how it’s valuable to simplify initially. Specifically, start by just building lists of users and needs, as opposed to trying everything all at once. Second, it describes how users can have many needs, there might be three or four users each with ten needs, which gets overwhelming. To solve that, focus on doing one user and one need, and then expand from there to prove out the value.
Top 5 Wardley Mapping Tools in 2024 on YouTube, and also available in blog post. Of these, particularly interested in damonsk/onlinewardleymaps which seems like a Graphviz approach, and reminds me a bit of my experiment with lethain/systems that I build for systems modeling.

That said, I’ll probably give Mapkeep a try first. My issue with systems modeling tooling is primarily that it’s very confusing to use, and Mapkeep seems like it might have the lowest overhead to usage of these Wardley mapping tools.
Situation Normal, Everything Must Change by Simon Wardley on YouTube. This is a good overview of Wardley mapping, although a bit surface level. A good first watch.
Wardley Mapping Part 1-4 by Simon Wardley on YouTube. A more detailed introduction to the same content as the Situation Normal, Everything Must Change talk above. The first two are a bit redundant with some of the prior talks, but the third and fourth are additive and quite useful.
Mapping Your Stack by Adrian Cockroft. Much of this is repeat of the above, but around 22:20 it goes into mapping a concrete example that I found quite interesting.
Wardley Maps for Software Developers by Markus Harrer. Most helpfully, it gets into concrete maps quickly, and walks through a number of distinct examples.
At this point, I signed up for a free Mapkeep, and started mapping.
My first project was this post on the evolution of the developer meta-productivity space, which gave me the opportunity to play around with a simple map and the evolution of that map over the past four years.

At this time, that’s still just a draft post, but was still a good opportunity for mapping practice. I also think it would have been much harder to form and articulate my point of view without the map, which is a good signal.
Next, I did some exploratory mapping of Gitlab’s strategy (e.g. why focus on bundling all-in versus Github’s strategy of supporting best-in-breed unbundling). This was quick, and instructive. It’s also giving me a better sense of what sort of problems Wardley mapping is useful for exploring (and which it isn’t so useful at digging into).

At this point, I feel pretty comfortable jumping into Mapkeeper and mapping something out. That’s not to say that I can’t improve, I certainly can improve a lot, but my sense is that this is going to primarily depend on doing more mapping, and less on some sort of magical realization.

There’s also a lot of stuff bundled into Wardley mapping that I think is quite useful, but isn’t as important to me as the components I’ve focused on here. For example, Wardley’s idea of doctrine and gameplay are great, but I generally would approach these sorts of things as “one particular kind of strategy” rather than a different category. That’s not a refutation of Wardley’s framing, just a different one that I’ve found more useful for the sorts of problems I’m currently focused on.

Video of practice run of QCon SF 2024 talk on Principal Engineers.

Thu, 21 Nov 2024 07:00:00 -0700

Yesterday at QCon, I got to give a talk with my colleague Dan Fike about the Principal Engineer role. You can also watch the video on YouTube.

The content itself is largely based on the High-Context Triad of blog posts I wrote in late 2023:

I hope you enjoy the talk! (Unfortunately I don’t have a recording of Dan Fike’s portion, but QCon will post their official video at some point, which will have both parts.)

How to get more headcount.

Sat, 16 Nov 2024 07:00:00 -0700

One of the recurring challenges that teams face is getting headcount to support their initiatives. A similar problem is the idea that a team can’t get a favored project into their roadmap. In both cases, teams often create a story about how clueless executives don’t understand why their work is important.

I understand why dumb executives are such an appealing explanation to problems: it fits perfectly into the Karpman drama triangle by making executives the villian and the team the victim, but I generally find that these sorts of misalignments are the result of basic communication challenges rather than something more exciting.

When there’s significant misalignment between a team and an executive, my experience is that it often manifests in discussion about a particular project, but it’s often rooted in a much broader topic rather than whatever is currently being discussed. Because the disagreement is about the larger topic, there’s no way to resolve it while discussing the narrow project at hand, and teams struggle to make progress because they’re arguing on the wrong layer of context.

To solve disagreement, the general hierarchy of alignment is:

Do we agree on the problem to be solved?

e.g. We’re having too many incidents and it’s impacting user perception and developer productivity.
Do we agree on the general approach to solving that problem?

e.g. We’re increaing end-to-end coverage and rejecting PRs that reduce coverage.
What evidence do we have that the team is executing well today?

e.g. Here are metrics on: how many user-impacting incidents we’ve had over time, how we’ve increased end-to-end coverage over time, and developer survey feedback on incidents from the last three quarterly internal surveys.
Alignment on the particular topic at hand: headcount, roadmapping, prioritization of a specific project, and so on.

e.g. To speed up further work on this, we’re requesting two more engineers.

If you are misaligned on any of the first three topics, addressing the fourth project is folly. For example, if the executive believes that your team is not executing your current project well, then they won’t believe that giving you more headcount is useful, because you’re already screwing up. To convince them to approve a headcount request, you need to first find evidence that your team is doing good work today.

Similarly, if the executive doesn’t agree with you on your problem or general approach, your headcount request is dead in the water. This is one of the reasons that I see bottoms-up “team mission” initiatives fail so frequently. Teams define their mission, and then tell executives what they’ve decided to focus on, but that’s a general misunderstanding of why teams exist: teams exist to solve a company need, not to solve the problem that the team itself wants to solve. When teams lean on their self-selected problem or approach to defend to an executive why they won’t do something, they dig in on a foundational misalignment that prevents addressing more nuanced discussions like project prioritization.

The solution here is obvious, always make sure you agree on the problem and general solution, and provide evidence the team is working well. These can be an appendix of a document or appendix slides, and should take little to no time to prepare as the first two are core decisions for your team, and the later is a set of metrics or plans that you should already be maintaining as part of operating your team.

If you refuse to engage on the first three topics, and skip to the fourth topic directly in aligning with an executive, then you are generally falling back to relying on social dynamics and executives’ general view of your prior work–what some might call politics–rather than having a joint problem solving session together. If you’re complaining about politics, and not taking the time to answer the first three, then perhaps you are inadvertantly contributing to the political environment that you dislike.

As always when discussing challenges communicating with executives, it’s true that executives should get better at explaining where they’re confused or struggling with the rationale. However, it’s a lot more useful to simply get better at this yourself than to spend time bemoaning how executives could, at a universal group, improve their communication. I know a lot of people who improved their company or moved into more senior roles by improving their own communication. I know zero folks who did either of those by complaining that executives are bad communicators, although they certainly weren’t wrong about it!

Navigating Private Equity ownership.

Mon, 11 Nov 2024 06:00:00 -0700

In 2020, you could credibly argue that ZIRP explains the world, but that’s an impossible argument to make in 2024 when zero-interest rate policy is only a fond memory. Instead, we’re seeing a number of companies designed for rapid expansion learning to adapt to a world that expects immediate free cash flow rather than accepting the sweet promise of discounted future cash flow.

This chapter wants to tackle that problem head-on, taking the role of an engineering organization attempting to navigate new ownership by a private equity group. It’s an increasingly frequent scenario: after many years of learning to operate under the direction of its original founders, and the brief excitement of going public, now there’s a short runway to change operating models. Let’s call this company Fungible Ecommerce Company. It’s a platform for supporting online commerce, and this is their Engineering Leadership team’s attempt to think through their options while waiting for new ownership to provide concrete guideposts.

This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts.

Reading this document

To apply this strategy, start at the top with Policy. To understand the thinking behind this strategy, read sections in reserve order, starting with Explore, then Diagnose and so on. Relative to the default structure, this document has been refactored in two ways to improve readability: first, Operation has been folded into Policy; second, Refine has been embedded in Diagnose.

More detail on this structure in Making a readable Engineering Strategy document.

Policy

Our policy for managing our new ownership structure is:

We believe our new ownership will provide a specific target for Research and Development (R&D) operating expenses during the upcoming financial year planning. We will revise these policies again once we have explicit targets, and will delay planning around reductions until we have those numbers to avoid running two overlapping processes.

That said, looking at our R&D investment relative to comparably growing peer set, we believe that we’ll get pressure to moderately reduce our spend. We aim to accomplish that reduction through a series of policies and one-off infrastructure projects, without requiring a major reduction in headcount spend.
We will move to an “N-1” backfill policy, where departures are backfilled with a less senior level. We will also institute a strict maximum of one Principal Engineer per business unit, with any exceptions approved in writing by the CTO–this applies for both promotions and external hires. These policies are effective immediately, and is based on our model of engineering-org seniority-mix.

We commit to this policy reducing headcount costs by approximately 5% YoY every year for the forseeable future.
We evaluated a number of potential changes to our geographical hiring strategy, but we believe that staffing engineers with cross-functional partners (Product, Marketing, Sales, and so on) is a priority. We have not been able to reach an agreement cross-functionally, and as such we are not changing our geographical hiring strategy at this time.

If we can agree on a policy here, we could accomplish 10-20% reduction in cost over 2-3 years, but the details matter a great deal, so we cannot commit to a specific outcome until we get more cross-functional alignment.
Our infrastructure spend has grown significantly more slowly than revenue for the past two years, meaning that we’ve successfully implement our infrastructure spend strategy of growing infrastructure costs more slowly than revenue. We will continue our current infrastructure efficiency strategy, and believe they are relatively few high impact efficiency opportunities at this point.

We commit to growing infrastructure spend at no more than 5% YoY, significantly lower than our projected revenue increase of 25% YoY.
There are two narrow infrastructure spend opportunities, both related to the integration of prior acquisitions into our shared infrastructure and away from one-off approaches. We will prioritize the post-acquisition integration work next quarter, with the goal of fully standarding all infrastructure across the company into the stack maintained by our centralized Infrastructure Engineering team.

We commit to a one-time reduction in infrastructure of 3% YoY.
We believe there are significant opportunities to reduce R&D maintenance investments, but we don’t have conviction about which particular efforts we should prioritize. We will kickoff a working group to identify the features with the highest support load.

Diagnose

We’ve diagnosed Fungible Ecommerce Company’s current state as:

Fungible Ecommerce Company’s revenue has grown 20-25% YoY for the past two years, and our target for next year is 25% YoY revenue growth. While this is not a guarantee, we grew slower than 25% last year, it’s a defensible goal that we have a good chance of achieving.
Our Engineering headcount costs have grown by 15% YoY this year, and 18% YoY the prior year. Headcount grew 7% and 9% respectively, with the difference between headcount and headcount costs explained by salary band adjustments (4%), a focus on hiring senior roles (3%), and increased hiring in higher cost geographic regions (1%).
Based on general practice, it seems likely that our new Private Equity ownership will expect us to reduce R&D headcount costs through a reduction. However, we don’t have any concrete details to make a structured decision on this, and our approach would vary significantly depending on the size of the reduction.
Infrastructure engineering spend (including vendors) has grown by 4-5% YoY for the past three years. We made a significant push on reducing costs three years ago, and have grown significantly slower than revenue since then.

There are few remaining opportunities to significantly reduce infrastructure costs, but we’ve made several acquisitions since our prior infrastructure consolidation, that represent significant potential savings: roughly one-time 1.5% YoY reductions for each of two largest opportunities.
A significant portion of our current R&D spend goes into maintaining our existing functionality, particularly functionality related to earier geo-expansion efforts that only apply narrows to some small markets. We suspect there’s an opportunity to reduce maintenance overhead here.

However, we lack believable metrics on both (1) time spent maintaining the software and (2) time that would be saved by these cleanup efforts. As a result, it’s hard to pitch projects of this sort as revenue saving with much conviction.

Explore

Financial markets evaluate companies in comparison to their peers. This is most obvious in public markets, where there’s significant information transparency about business performance, and sufficient liquidity to allow markets to revalue companies in something approaching real-time. While private equity firms generally take controlling interest of private businesses, or with the intent of taking the business private if it happens to be public, they value businesses in the same way.

In this exploration, we’re going to dig into two particular questions. First, we’re going to dig into a dataset on the performance of public technology companies, and then second we’re going to look into the concrete example of Zendesk, who were taken private in 2022 after being bought by two private equity firms.

Comparable companies

Exploring the benchmarking question first, most investors evaluate engineering within the context of the overall Research & Development (R&D) investment. They generally judge that spend by constructing a scatterplot of R&D spend versus year-over-year revenue growth for a cohort of similar companies. Perfectly similar companies don’t exist, so this cohort is generally constructed from companies in similar industries, with similar revenue, and operating in the same regions.

We have reached out to our investors to see if they can provide the internal datasets they use for this analysis, but in the mean-time we’ve developed a directionally useful dataset using the 2023 R&D Investment Scoreboard, with some rough cutting of the data to remove outliers. (If we repeat this process, we will use the SEC’s EDGAR database to pull a more specifically helpful dataset, but this has been a useful starting point.)

This isn’t a perfect dataset, we prefer revenue growth over rgowth in operating profit, but it’s the best option within the dataset that we were able to quickly pull down. Nonetheless, there’s a clear strong performer quadrant in top-left that we can plot ourselves into to understand our general performance, which is discussed further in the diagnosis section above.

Zendesk

The second topic of exploration we dug into is understanding the general sequence of steps taken by private equity ownership after taking ownership of a company. For an example with available public documentation, we focused on the purchase of Zendesk in 2022.

To start, we pulled Zendesk’s final 10-Q before going private.

Taking those values, we can reformat them into a chart focusing on the year-over-year changes in the 6 months period ending in 2022 versus the same period in 2021.

The changes are a bit concerning. Sales and Marketing costs have grown more slowly than revenue, which is positive, but Research and Development (R&D) expenses have grown about 50% faster than revenue, and General and Administration (G&A) charges have grown more than twice as quickly as revenue.

From those growth rates, we would assume that the new ownership might push to aggressively reduce spend in those two areas, which is indeed what history suggests happend, with a November, 2022 reduction, followed some months later by a May, 2023 reduction. It’s hard to get precise data here, but it’s our impression that these reductions focused on areas where expenses were growing quickly, with particular focus on G&A functions.

Using systems modeling to refine strategy.

Mon, 04 Nov 2024 07:00:00 -0700

While I was probably late to learn the concept of strategy testing, I might have learned about systems modeling too early in my career, stumbling on Donella Meadows’ Thinking in Systems: A Primer before I began my career in software. Over the years, I’ve discovered a number of ways to miuse systems modeling, but it remains the most effective, flexible tool I’ve found to debugging complex problems.

In this chapter, we’ll work through:

when systems model is a useful technique, and when it’s better to rely on other refinement techniques like Wardley mapping or strategy testing instead
a two minute primer on the basics of systems modeling, along with resources for those looking for a deeper exploration of the foundational topics
a discussion on systems modeling tooling, why there’s no perfect systems modeling tool out there, and how I recommend picking the tool that you build proficiency with
the steps to build a systems model for a problem you’re engaging with
how to document your learnings from a systems model to maximize the chance that others will pay attention to it rather than ignoring it due to the unfamiliarity or complexity of the tooling
what systems modeling can’t do, even if you really want to believe it can

After working through this chapter’s overview of systems modeling, you can see the approaches implemented in a number of system models created to refine the strategies throughout this book. The theory of systems modeling is certainly interesting, but hopefully seeing real models in support of concrete engineering strategies will be even more useful.

When is systems modeling useful?

Although refinement is an important step of developing any strategy, some refinement techniques work better for any given strategy. Systems modeling is extremely useful in three distinct scenarios:

When you’re unsure where leverage points might be in a complex system, modeling allows you to cheaply test which levers might be meaningful. For example, modeling onboarding drivers in a ride-sharing app showed that improving onboarding was less important than reengaging departed drivers.
When you have significant data to compare against, which allows you to focus in on the places where the real data and your model are in tensions. For example, I was able to model the impact of hiring on Uber’s engineering productivity, and then compare that with internal data.
When stakeholder disagreements are based in their unstated intuitions, models can turns those intuitions into something structured that can be debated more effectively.

In all three categories, modeling makes it possible iterate your thinking much faster than running a live process or technology experiment with your team. I sometimes hear concerns that modeling slows things down, but this is just an issue of familiarity. The more you practice, modeling can be faster than asking for advice fro industry peers. The actual models I’ve developed for this book took less than an hour. (With one notable exception: modeling Large Language Models (LLMs) impacts on developer experience, which took much longer because I deliberately used an impractical tool to reveal the importance of good tooling).

Additionally, systems modeling will often expose counter-intuitive dimensions to the problem you’re working on. For example, the model I mentioned above on LLMs’ impact on developer experience suggests that effective LLMs might cause us to spend more time writing and testing code (but less fixing issues discovered post-production). This is a bit unexpected, as you might imagine they’d reduce testing time, but reducing testing time is only valuable to the extent that issues identified in production remains–at worst–constant; if issues found in production increases, then reduced testing time does not contribute to increased productivity.

Modeling without praxis, creates unsubstantiated conviction. However, in combination with praxis, I’ve encountered few other techniques that can similar accelerate learning.

That doesn’t mean that it’s always the ideal refinement technique. If you already have conviction on the general approach, and want to refine the narrow details, then strategy testing is a better option. If you’re trying to understand the evolution of a wider ecosystem, then you may prefer Wardley mapping.

Two minute primer

If you want an exceptional introduction to systems thinking, there’s no better place to go than Donella Meadows’ Thinking in Systems. If you want a worse, but shorter, introduction, I wrote a short Introduction to systems thinking available online and in An Elegant Puzzle.

If you want something even shorter, then here’s the briefest that I can manage.

Accumulations are called stocks. For example, each of the boxes (Requests, Server, etc) in the above diagram is a stock. Changes to stocks are called flows. Every arrow (OK, Error in server, etc) between stocks in the diagram is a a flow.

Systems modeling is the practice of using various configurations of stocks and flows to understand circumstances that might otherwise have surprising behavior or are too slow to understand from measurement.

For example, we can use the above model to explore the tradeoffs between a load balancer that does and does not cap throughput to a load-sensitive service behind it.

Without a model, you might get into a philosophical debate about how rediculous it is that the downstream server is load-sensitive. With the model, it’s immediately obvious that it’s worthwhile protecting it, even if it certainly is concerning that it is so sensitive. This is what models do: they create a cheap way to understand reality when fully understanding reality is cumbersome.

More systems thinking resources

Thinking in Systems: A Primer by Donella Meadows

Business Dynamics: Systems Thinking and Modeling for a Complex World by John D. Sterman

An Introduction to Systems Thinking by Barry Richmond

Tooling

For an idea that’s quite intuitive, the tools of systems modeling are a real obstacle to wider adoption. Perhaps a downstream consequence of many early, popular systems modeling tools being quite expensive, the tooling ecosystems for systems modeling has remained fragmented for some time. There also appears to be a mix of complex requirements, patent consolidation, and percieved small market size that’s discouraged a modern solutions from consolidating the tooling market.

Earlier, I mentioned that system modeling is extremely quick, but that many folks find it a slow, laborous process. Part of that is an issue of practice, but I suspect that the quality of modeling tooling is at least a big a part of the challenge. In the LLMs impact on developer experience model, I go about the steps of building the model in an increasingly messy spreadsheet. This was slow, challenging, and extremely brittle. Even after finishing the model, I couldn’t extend it effectively to test new ideas, and I inadvertently introduced a number of bugs into the implementation.

Going in the opposite direction, I explored using a handful of tools, such as Sagemodeler or InsightMaker, which seemed like a potentially simpler toolchains than the one I typically rely on. There are so many of these introductary toolchains for systems modeling, but I generally find that they’re either constrained in their capabilities, have a fairly high learning curve, or make it difficult to share your model with others.

In the end, I wound up back at the toolchain that I use, which happens to be one that I wrote some years ago,lethain/systems. This is far from a perfect toolchain, but I think it’s a relatively effective mechanism for demonstrating systems modeling for a few reasons:

quick to create models and iterate on those models
easy to share those models with others for inspection and their own exploration
relatively low surface area for bugs in your models
free, open-source, self-hosted toolchain that integrates well with Jupyter ecosystem for diagramming, modeling and so on

You should absolutely pick any tool that feels right to you, and practice with it until you feel confident quickly modeling scenarios. Afterwards, I wouldn’t recommend spending too much time thinking about tools at all: the most important thing is to build models and learn from them quickly, and almost any tool will be sufficient to that goal with some deliberate practice.

How to model

Learning to system model takes some practice, so we’ll approach the details of learning to model from two directions. First, by documenting a general structure for approaching modeling, and second by providing breadcrumbs to the models developed in this book for deeper exploration of particular modeling ideas.

The structure to systems modeling that I find effective is:

Sketch the stocks and flows on paper or a diagramming application (e.g. Excalidraw, Figma, Whimsical, etc). Use whatever you’re comfortable with.
Reason about how you would expect a potential change to shift the flows through the diagram. Which flows do you expect to go up, and which down, and how would that movement help you evaluate whether your strategy is working?
Model the stocks and flows in your spreadsheet tool of choice. Start by modeling the flows from left to right (e.g. the happy path flows). Once you have that fully working, then start modeling the right to left flows (e.g. the exception path flows).

See the Modeling impact of LLMs on Developer Experience model for a deep dive into the particulars of creating a model.
Exercise the model by experiment with a number of different starting values and determining which rates really impact the model’s values. This is essentially performing sensitivity analysis
Document the work done in the above sections into a standalone writeup. You can then link to that writeup from strategies that benefit from a given model’s insights. You might link to any section of your strategy, depending on what topic the particular model explores. I recommend decoupling models from specific strategies, as generally the details of any given model are a distraction from understanding a strategy, and it’s best to avoid that distraction unless a reader is surprised by the conclusion, in which case, the link out supports drilling into the details.

As always, this is the sequence of steps that I’d encourage you to follow, and the sequence that I generally follow, but you should adapt them to solve the particular problems at hand. Over time, my experience is that most of these steps–excluding documentation–turn into a single iterative process, and that I document everything after several iterations.

Breadcrumbs for deeper exploration

Now that we’ve covered the overarching approach to system modeling, here are the breadcrumbs to specific models that go deeper on particular elements:

Modeling driver onboarding explores how the driver lifecycle at Theoretical Ride Sharing might be improved with LLMs, and introduces using the lethain/systems library for modeling
Modeling impact of LLMs on Developer Experience looks at how LLMs might impact developer experience at Theoretical Ride Sharing, and is demonstrates (the downsides of) modeling with a spreadsheet
Modeling engineering backfill strategy studies the financial consequences of various policies for how we backfill departed engineers in an engineering organization, and introduces further lethain/systems features

Beyond these models, you can find other systems models that I’ve written on my blog’s systems-thinking category, and there are numerous, great examples in the materials references in the systems modeling primer section above.

How to document a model

Much like documenting strategy is challenging, communicating with models in a professional setting is challenging. The core problems is that there are many distinct groups of model readers. Some will lack familiarity with the tooling you use to develop models. Others will try to refine, or invalidate, your model by digging into the details.

I navigate those mismatches by focusing first on the audience who is least likely to dig into the model. I still want to keep all the details handy, ideally in the rawest form possible to allow others to manipulate the model themselves, but it’s very much my second goal when documenting a model.

From experience, I recommended this order (it’s also the order used in the models in this book, so you’ll see it in practice a number of times):

start with learning section, with charts showing what model has taught you
sketch and explaing the stocks and flows
reason about what the sketch itself teaches you
explain how you developed the model, with an emphasis on any particularly complex portions
exercise the model by testing how changing the flows and stocks leads to different outcomes

If you remember nothing else, your document should reflect the reality that most people don’t care how you built the model, and just want the insights. Give them the insights early, and assume no one will trust your model nearly as much as you do. Models are an input into the strategy, never a reliable sole backer for a strategy.

What systems modeling isn’t

Although I find systems modeling a uniquely powerful way to accelerate learning, I’ve also encountered many practioners who believe that their models are reality rather than reflecting reality. Over time, I’ve developed a short list of cautions to help would-be modelers avoid overcommitting to their model’s insights:

When your model and reality conflict, reality is always right. At Stripe, we developed a model to guide our reliability strategy. The model was intuitively quite good, but its real-world results were mixed. Attachment to our early model distracted us (too much time on collecting and classifying data) and we were slow to engage with the most important problems (maximizing impact of scarce mitigation bandwidth, and growing mitigation bandwidth). We’d have been more impactful if we engaged directly with what reality was teaching us rather than looking for reasons to disregard reality’s lessons.
Models are immutable, but reality isn’t. I once joined an organization investing tremendous energy into hiring but nonetheless struggling to hire. Their intuitive model pushed them to spend years investing into top of funnel optimization, and later steered them to improving the closing process. What they weren’t able to detect was that misalignment in interviewer expectations was the largest hurdle in hiring.
Every model omits information; some omit critical information. The service migration at Uber is a great example: modeling clarified that we had to adopt a more aggressive approach to our service migration in order to succeed. Subsequently, we did succeed at the migration, but the model didn’t study the consequences of completing the migration, which were a very challenging development environment. The model captured everything my team cared about, as the team responsible for running the migration, but did nothing to evaluate whether the migration was a good idea overall.

In each of those situations, two things are true: the model was extremely valuable, and the model subtly led us astray. We would have been led astray even without a model, so the key thing to remember isn’t that models are inherently misleading, instead the risk is being overly confident about your model. A powerful tool to use in tandem with judgment, not a replacement.

Summary

Systems modeling isn’t prefect. If you’ve already determined your strategy and want to refine the details, then strategy testing is probably a better choice. If you’re trying to understand the dynamics of an envolving ecosystem, then Wardley mapping is a more appropriate tool.

However, if you have the general shape, but lack conviction on how the pieces fit together, systems modeling is a remarkable tool. After this chapter, you know how to select appropriate tooling, and how to use that tooling to model your problem at hand. Next, we’ll work through systems modeling a handful of detailed problems to provide concrete examples of applying this technique.