Systems modeling to refine strategy.

strategy (22), eng-strategy-book (7)

This is a work-in-progress draft!

While I was probably late to learn the concept of effectively testing strategy, I might have learned about systems modeling too early in my career, stumbling on Donella Meadows’ Thinking in Systems: A Primer before I began my career in software. Over the years, I’ve discovered a number of ways to miuse systems modeling, but it remains the most effective, flexible tool I’ve found to debugging complex problems.

In this chapter, we’ll work through:

  • when to use system modeling
  • five minute primer on systems modeling itself (along with resources for a deeper exploration)
  • tooling for doing systems modeling
  • the steps to build a systems model for a problem
  • how to write up a model
  • what systems modeling can’t do, even if you really want to believe it can

After working through this chapter’s overview of systems modeling, this book also includes a number of specific models in support of various strategies, which will help cement these models use and creation.

This is an exploratory, draft chapter for a book on engineering strategy that I’m brainstorming in #eng-strategy-book. As such, some of the links go to other draft chapters, both published drafts and very early, unpublished drafts.

When is systems modeling useful?

Systems modeling is extremely useful in three distinct scenarios:

  1. When you have significant data to compare against, which allows you to focus in on the places where the real data and your model are in tensions. For example, I was able to model the impact of hiring on Uber’s engineering productivity, and then compare that with internal data.
  2. When you’re unsure where leverage points might be in a complex system, modeling allows you to cheaply test which levers might be meaningful. For example, I was able to model how limiting work-in-progress impacted the rate that we finished work, which helped me identify hypotheses to prove or disprove.
  3. When stakeholder disagreements are based in their unstated intuitions, models can turns those intuitions into something structured that can be debated more effectively.

In all three categories, modeling makes it possible to rapidly iterate your thinking on a given topic, much faster than running a live process or technology experiment with your team might. As you combine a model with testing your strategy, you can learn much more quickly than using either technique alone.

Additionally, systems modeling will often expose counter-intuitive dimensions to the problem you’re working on. For example, modeling the impact of Large Language Models (LLMs) on developer productivity suggests that effective LLMs might cause us to spend more time writing and testing code (but less fixing issues discovered post-production). This is a bit unexpected, as you might imagine they’d reduce testing time, but reducing testing time is only valuable to the extent that issues identified in production remains–at worst–constant; if issues found in production increases, then reduced testing time does not contribute to increased productivity.

Modeling without praxis, creates unsubstantiated conviction. However, in combination with praxis, I’ve encountered few other techniques that can similar accelerate learning.

Five minute primer

Link to systems thinking

Tooling

TODO: actually, I think that I want to show a variety of different tools, because TBD modeling with a spreadsheet is super painful and very unfriendly for beginners. I think any approach that solely shows spreadsheet modeling is basically doomed

My advice is:

  1. start with sagemolder or another free, simple modeling tooling online, as described in ??. This is quick, and supports easy iteration in your model.
  2. then model a few in a spreadsheet as described in DX LLM example to build your intuitive understanding of the math within the model. This approach is fairly brittle and hard to modify, which makes iterating on the model quite hard, so I can’t recommend it despite the benefit of using familiar tooling.
  3. Eventually move to a toolkit like systems which is quite to write, cheap to iterate, and creates permanent artifacts that can be used to review your model without depending on third-party applications being maintained online.

For an idea that’s quite intuitive, the tools of systems modeling are a real obstacle to wider adoption. Perhaps a downstream consequence of many early, popular systems modeling tools being quite expensive, the tooling ecosystems for systems modeling has remained fragmented for some time. There also appears to be a mix of complex requirements, patent consolidation, and percieved small market size that’s discouraged a modern solutions from consolidating the tooling market.

That said, there are a number of options for someone starting to experiment with systems modeling. SageModeler is a free, browser-based solution that works well. A few years ago, I experimented writing a small language for modeling in the open-source systems repository. There are also many other options out there!

Unfortunately, and I feel somewhat sad to say this, I think the best tools for a new systems modeler aren’t domain-specific, custom solutions. Rather, it’s whatever diagramming and spreadsheet tools your coworkers already know how to use. The diagramming tool might be Graphviz, but it could also be Excalidraw or Figma. The spreadsheet tool is probably Google Sheets, Microsoft Excel, or something along those lines. For examples in this book, I’ll be using the combination of Excalidraw and Google Sheets, but I’ll be using standard functionality that’ll be simple to migrate to tools of your choice.

As much as you can argue that other tools are better–and they are–the ability to share your model with your colleagues is generally more valuable than cleaner UIs and more powerful capabilities. There’s no line in existence that you can show me, without being able to explain the source of the data points, that will change my mind about something important. As such, a familiar, inspectable, understandable toolchain is the most valuable place to start. You should absolutely experiment with other tools as you iterate, but you’ll likely need to do your final work in these familiar, sharable formats.

How to model

I’ve pulled together a number of example models that look into concrete model development:

The general approach to systems modeling that I find effective is:

  1. Sketch the stocks and flows on paper or a diagramming application (e.g. Excalidraw, Figma, Whimsical, etc).

  2. Reason about how you would expect a potential change to shift the flows through the diagram. Which flows do you expect to go up, and which down, and how would that movement help you evaluate whether your strategy is working?

  3. Model the stocks and flows in your spreadsheet tool of choice. Start by modeling the flows from left to right (e.g. the happy path flows). Once you have that fully working, then start modeling the right to left flows (e.g. the exception path flows).

    See the Modeling impact of LLMs on Developer Experience model for a deep dive into the particulars of creating a model.

  4. Exercise the model by experiment with a number of different starting values and determining which rates really impact the model’s values. This is essentially performing sensitivity analysis

  5. Document the work done in the above sections into a standalone writeup. You can then link to that writeup from strategies that benefit from a given model’s insights. You might link to any section of your strategy, depending on what topic the particular model explores. I recommend decoupling models from specific strategies, as generally the details of any given model are a distraction from understanding a strategy, and it’s best to avoid that distraction unless a reader is surprised by the conclusion, in which case, the link out supports drilling into the details.

How to document a model

What I recommend is:

  • start with learning section, with charts showing what model has taught you
  • sketch and explaing the stocks and flows
  • reason about what the sketch itself teaches you
  • develop the model
  • exercise the model

I guess the key thing is that most people don’t care how you built the model, and just want the insights. Give them the insights early, and assume no one will really believe your model is meaningful. Models are more useful for convicing yourself about something complex, rather than convincing others.

What systems modeling isn’t

Although I find systems modeling a uniquely powerful way to accelerate learning, I’ve also encountered many practioners who believe that their models are reality rather than reflecting reality. Over time, I’ve developed a short list of cautions to help would-be modelers avoid overcommitting to their model’s insights:

  1. When your model and reality conflict, reality is always right. At Stripe, we developed a model to guide our reliability strategy. The model was intuitively quite good, but its real-world results were mixed. Attachment to our early model distracted us (too much time on collecting and classifying data) and we were slow to engage with the most important problems (maximizing impact of scarce mitigation bandwidth, and growing mitigation bandwidth). We’d have been more impactful if we engaged directly with what reality was teaching us rather than looking for reasons to disregard reality’s lessons.
  2. Models are immutable, but reality isn’t. I once joined an organization investing tremendous energy into hiring but nonetheless struggling to hire. Their intuitive model pushed them to spend years investing into top of funnel optimization, and later steered them to improving the closing process. What they weren’t able to detect was that misalignment in interviewer expectations was the largest hurdle in hiring.
  3. Every model omits information; some omit critical information. The service migration at Uber is a great example: modeling clarified that we had to adopt a more aggressive approach to our service migration in order to succeed. Subsequently, we did succeed at the migration, but the model didn’t study the consequences of completing the migration, which were a very challenging development environment. The model captured everything my team cared about, as the team responsible for running the migration, but did nothing to evaluate whether the migration was a good idea overall.

In each of those situations, two things are true: the model was extremely valuable, and the model subtly led us astray. We would have been led astray even without a model, so the key thing to remember isn’t that models are inherently misleading, instead the risk is being overly confident about your model. A powerful tool to use in tandem with judgment, not a replacement.

Summary

…