Guiding broad change with metrics.

Although folks often talk about goals and metrics when they’re writing new plans or reflecting on plans past, my fondest memories of metrics are when I’ve seen them used to drive large organizational change. In particular, I’ve found metrics to be an extremely effective way to lead change with little or no organizational authority, and I wanted to write up how I’ve seen that work.

At both Stripe and Uber, I’ve had the opportunity to manage infrastructure costs (plug for Ryan’s amazing blog post on managing AWS Reserved Instances). Folks who haven’t thought about this problem often default to viewing it as boring, but as you dig into it, I’ve found it rich soil for learning about leading organizational change.

It’s also a good example of how to lead change with metrics!

Infrastructure cost is a great example of a baseline metric; when you’re asked to take responsibility for a company’s overall infrastructure costs, you’re going to start from a goal along the lines of “Maintain infrastructure costs at their current percentage of net revenue of 30%.” (30% is a fictional number for this example’s purposes, the percentage will depend on your industry and maturity, but I have found tying it against net revenue is more useful than pinning it at a specific dollar amount.)

From there, the approach I’ve found effective is:

1. Explore - the first step is to get data in an explorable format in your data warehouse, a SQL database or even an Excel spreadsheet. Once there, spend time looking through it, and getting a feel for it. Your goal in this phase is to identify where the levers for change are. For example, you might find that your batch pipeline is the majority of your costs and that your data warehouse is surprisingly cheap, which will allow you to focus further efforts.
2. Dive - once you know the three or four major contributors, go deep on understanding those areas and the levers that drive them. Batch costs might be sensitive to number of jobs, total data stored, new product development, or it might be entirely driven by a couple expensive jobs. Diving deep helps you build a mental model, and it also kicks off a relationship between you and the teams who you’ll want to partner with most closely.
3. Attribute - for most company level metrics (cost, latency, development velocity, etc), the first step of diving will uncover one team that is nominally accountable for the metric’s performance, but they are typically a cloak. When you pull that cloak aside, that team’s performance is actually driven by dozens of other teams. For example, you might have a Cloud engineering team that is accountable for provisioning VMs, but they’re not the folks writing the code that runs on those VMs. It’s easy to simply pass the cost metric on to that Cloud team, but that’s just abdicating responsibility to them. What’s more helpful is to help them build a system of second degree attribution, allowing you to build data around the teams using the platform. This second degree of attribution is going to allow you to target the folks who can make an impact in the next step.
4. Contextualize - armed with the attribution data, start to build context around each team’s performance. The most general and self-managing tool for this is benchmarking. It’s one thing for a team to know they’re spending $100,000 a month, and it’s something entirely for them to know they’re spending $100,000 a month and that their team spends the second most out of forty-seven teams. Benchmarking is particularly powerful because it automatically adapts to changes in behavior. In some cases, benchmarking against all teams might be too coarse, and it may be useful to benchmark against a small handful of cohorts. For example, you might want to define cohorts for frontend, backend and infrastructure teams, given they’ll have very different cost profiles.
5. Nudge - once you’ve built context around the data so that folks can interpret it, the next step is to start nudging them to action! Dashboards are very powerful to analysis, but the challenge for baseline metrics is that folks shouldn’t need to think about them the vast majority of the time, and that can lead to them forgetting about the baselines entirely. What I’ve found effective is to send push notifications, typically email, to teams whose metric has changed recently, both in terms of absolute change and in terms of their benchmarked performance against their cohort. This ensures each time you push information to a team, it includes important information that they should act on! What’s so powerful about nudges is that simply letting folks know their behavior has changed will typically stir them to action, and it doesn’t require any sort of organizational authority to do so. (For more on this topic, take a look at Nudge.)
6. Baseline - in the best case, you’ll be able to drive the organizational impact you need with contextualized nudges, but in some cases that isn’t quite enough. The next step is to work with the key teams to agree on baseline metrics for their performance. This is useful because it ensures the baselines are top of mind, and it also gives them a powerful tool for negotiating priorities with their stakeholders. In some cases this does require some organizational authority, but I’ve found that folks universally want to be responsible. As long as you can find time to sit down with the key teams and explain why the goal is important, it typically doesn’t require much organizational authority.
7. Review - the final phase, which hopefully you won’t need to reach, is running a monthly or quarterly review that looks at each team’s performance, and reaching out to teams to advocate for prioritization if they aren’t sustaining their agreed upon baselines. This typically requires an executive sponsor, because teams who aren’t hitting their baselines are almost always being prioritized against other goals, and need your help explaining to their stakeholders why the change is important.

I’ve seen this approach work, and more importantly to be very scalable. It enables a company to concurrently maintain many baseline metrics without overloading its teams. This is largely because this approach focuses on driving targeted change within the key drivers, only requiring involvement from a small subset of teams for any given metric, but also because it tries to minimize top-down orchestration in preference of providing information to encourage teams themselves to adjust priorities.