Irrational Exuberance

Learning from Every's Compound Engineering

Mon, 19 Jan 2026 10:00:00 -0700

One of the relatively few AI-native products I use is Cora.computer which summarizes my personal inbox. It’s not perfect, but it’s done a much better job than my collection of filters at managing the ever-growing onslaught of spam and unsolicited email that flows in.

I’ve run into a few issues with Cora, which ended up in me following folks at Every to report the issues, and more recently this led me to see their work on compound engineering and specifically the compound-engineering-plugin.

Compound Engineering is two extremely well-known patterns, one moderately well-known pattern, and one pattern that I think many practitioners have intuited but have not found a consistent mechanism to implement. Those patterns are:

Plan is decoupling implementation from research. This is well understood, e.g. Claude’s plan mode, although it can certainly be done better or worse by being more specific about which resources to consult (specs, PRDs, RFCS, issues, etc)
Work is implementing a plan. This is well understood, and the core of agentic coding. Again, this can be done better or worse, but much of that depends more on the quality of your codebase, tests, and continuous integration harness than the agent itself
Review is asking the agent to review the changes against your best-practices, and identify ways it could be improved. I think most practitioners have some version of this, but standardization is low, even within a given company.
Compound is asking the agent to summarize its learnings from a given task into a well-defined, structured format (basically a wiki) which is consulted by future iterations of the plan pattern. This interplay between the compound and plan steps creates the compounding mechanism.

Many practitioners are implicitly compounding, but it’s often done manually through their own work. For example, I’d often ask the agent to update our AGENTS.md or skills based on a specific problem encountered in a task, but it required my active attention to notice the issue and suggest incorporation.

Taken together, these four steps are not shocking but are an extremely effective way to convert these intuited best-practices into something specific, concrete, and largely automatic within a company by adding a few commands (e.g. workflow:plan, workflow:review, …) and updating your AGENTS.md to instruct the agent when and how to use those commands.

Implementing within Imprint’s frontend and backend monorepos was straightforward, taking about an hour. Most of this was iterating on the last mile of details, for example we want our plans in .claude/plan-*.md format to match our existing .gitignore pattern, and none of it was complex. Most importantly, this frees up a topic that many of our engineers (including me) were trying to find a standard approach. Now we have one, and can move on to the next problem.

If recent history is our guide, it’s a solid guess that many of the practices in compound engineering will get absorbed into the Claude Code and Cursor harnesses over the next couple of months, at which point using these techniques explicitly will be indistinguishable from folks who are entirely unaware they’re using them. But we’ll see. Until then, this is a cheap, useful experiment that you can implement in an hour.

Sharing Claude transcripts.

Sat, 10 Jan 2026 07:15:00 -0800

One of my central premises for supporting internal adoption of LLMs is that adoption depends on easy discovery of what’s possible and what’s good. That is why our internal prompts driving agents are stored in a shared Notion database, but it also begged the question: our most advanced prompting and interactions are happening in Claude Code, which are hard to see.

Thankfully, Simon Willison previously wrote a tool to extract transcripts from Claude Code called claude-code-transcripts, which we were able to wire together into an internal repository of Claude Code sessions and a viewer on Cloudflare pages (and behind Cloudflare authentication into our SSO).

There are three components here. First, an index of all the pages.

That page links into the transcripts generated by Simon’s tool.

Finally, we have an internal CLI named imp that is available on every laptop, which now has an additional tool imp claude share-session that will open claude-code-transcripts, allow you to select a session of choice, and then merge it into the holding repository.

Altogether, this was an hour or two of work, and a bit of an experiment in emergent process design. In the short-term, I am enjoying asking our biggest Claude Code users to share their sessions so that I can cherry-pick their practices.

Moved newsletter from Mailchimp to Buttondown.

Sat, 10 Jan 2026 07:00:00 -0800

In preparation for the release of An Elegant Puzzle, I set up the page to subscribe to my newsletter on January 20th, 2019, heavily inspired by Julia Evans’s approach. I didn’t know anything about releasing a book, but Brie Wolfson coached me through it, and having a newsletter to tell folks about the book seemed like a good idea. My blog had already had an RSS feed for ~12 years at that point, but RSS usage has steadily declined since the golden era of the 2000s.

Following Julia’s post, I set up my newsletter to run on Mailchimp, and that has mostly worked well for me over the following six years. (Looking at Julia’s website, it looks like she subsequently moved to Convertkit at some point.) However, over time I kept running into issues with Mailchimp. Those frustrations slowly mounted:

I could not find the “welcome to this newsletter” template to change the recommended posts. I think this might have been related to them rewriting their UX entirely at some point, but I didn’t really want to become a Mailchimp expert just to update this
Every time I changed jobs, someone would tell me that I needed to update the address for contact, and each time it felt a little bit harder to find the text field to update it
The DMARC changes were confusing to navigate within Mailchimp. DMARC enforcement was absolutely not Mailchimp’s fault, but configuring it with Mailchimp was a fairly confusing process. Presumably the documentation is much improved at this point, but wasn’t great for me at the point I cut over
I was paying $326/month for something that was difficult to tune to work how I wanted. The cost has increased over time, so I wasn’t paying this much the entire time, but back-of-envelope I paid Mailchimp somewhere around $15,000 over six years

Every year or so I’d considered migrating off Mailchimp to something that was more purpose-built for my needs, but never quite got around to it. However, this year I decided to go ahead and migrate. I did some quick research, landed on Buttondown (whose founder I happened to overlap with at Stripe), and the next newsletter on Wednesday will be coming from Buttondown rather than Mailchimp. I’m not quite sure how much I’ll end up paying Buttondown, but it’ll be either $79 or $139/month.

The cutover was very straightforward, including getting to write a bit of Django template syntax for the first time in a decade or so and some DNS setup. Now the imported archive is up at archive.lethain.com, and I’ll start sending this upcoming Wednesday. One small feature that I’ve wanted for a long time on Mailchimp is the ability to change the format when the newsletter has one or more than one post in it, which was a quick win on Buttondown.

This doesn’t mean I have plans to meaningfully change how I’ve been newslettering for the past six years, although I hope it’ll get a bit more interesting in 2026 versus the prior two years, as I’ve completed my book publishing goals for the 2020s, and am excited to return to writing more widely about stuff I’m working on! There’s only so many years of sharing draft chapters before it starts to feel a bit stale.

As a final thought, two years ago I think folks would have been confused by my decision to not move to Substack, just like six years ago they would have been confused by my decision not to move to Medium. The answer here is easy for me: my goals remain consistent ownership of my work, on domains I control. If I was writing to directly build a business, I imagine both of those choices would have been much harder, but at this point I’m surprisingly anchored to my desire to be an operator who writes, which is where I think the most interesting writing happens.

Building internal agents

Thu, 01 Jan 2026 09:00:00 -0800

A few weeks ago in Facilitating AI adoption at Imprint, I mentioned our internal agent workflows that we are developing. This is not the core of Imprint–our core is powering co-branded credit card programs–and I wanted to document how a company like ours is developing these internal capabilities.

Building on that post’s ideas like a company-public prompt library for the prompts powering internal workflows, I wanted to write up some of the interesting problems and approaches we’ve taken as we’ve evolved our workflows, split into a series of shorter posts:

In the same spirit as the original post, I’m not writing these as an industry expert unveiling best practice, rather these are just the things that we’ve specifically learned along the way. If you’re developing internal frameworks as well, then hopefully you’ll find something interesting in these posts.

Building your intuition for agents

As more folks have read these notes, a recurring response has been, “How do I learn this stuff?” Although I haven’t spent time evaluating if this is the best way to learn, I can share what I have found effective:

Reading a general primer on how Large Language Models work, such as AI Engineering by Chip Huyen. You could also do a brief tutorial too, you don’t need the ability to create an LLM yourself, just a mental model of what they’re capable of
Build a script that uses a basic LLM API to respond to a prompt
Extend that script to support tool calling for some basic tools like searching files in a local git repository (or whatever)
Implement a tool_search tool along the lines of Anthropic Claude’s tool_search, which uses a separate context window to evaluate your current context window against available skills and return only the relevant skills to be used within your primary context window
Implement a virtual file system, such that tools can operate on references to files that are not within the context window. Also add a series of tools to operate on that virtual file system like load_file, grep_file, or whatnot
Support Agent Skills, particularly load_skills tool and enhancing the prompt with available skills
Write post-workflow eval that runs automatically after each workflow and evaluates the quality of the workflow run
Add context-window compaction support to keep context windows below a defined size Make sure that some of your tool responses are large enough to threaten your context-window’s limit, such that you’re forced to solve that problem

After working through the implementation of each of these features, I think you will have a strong foundation into how to build and extend these kinds of systems. The only missing piece is supporting code-driven agents, but unfortunately I think it’s hard to demonstrate the need of code-driven agents in simple examples, because LLM-driven agents are sufficiently capable to solve most contrived examples.

Why didn’t you just use X?

There are many existing agent frameworks, including OpenAI Agents SDK and Claude’s Agents SDK. Ultimately, I think these are fairly thin wrappers, and that you’ll learn a lot more by implementing these yourself, but I’m less confident that you’re better off long-term building your own framework.

My general recommendation would be to build your own to throw away, and then try to build on top of one of the existing frameworks if you find any meaningful limitations. That said, I really don’t regret the decision to build our own, because it’s just so simple from a code perspective.

Final thoughts

I think every company should be doing this work internally, very much including companies that aren’t doing any sort of direct AI work in their product. It’s very fun work to do, there’s a lot of room for improvement, and having an engineer or two working on this is a relatively cheap option to derisk things if AI-enhanced techniques continue to improve as rapidly in 2026 as they did in 2025.

Building an internal agent: Iterative prompt and skill refinement

Thu, 01 Jan 2026 08:30:00 -0800

Some of our internal workflows are being used quite frequently, and usage reveals gaps in the current prompts, skills, and tools. Here is how we’re working to iterate on these internal workflows.

This is part of the Building an internal agent series.

When companies push on AI-led automation, specifically meaning LLM agent-driven automation, there are two major goals. First is the short-term goal of increasing productivity. That’s a good goal. Second, and I think even more importantly, is the long-term goal of helping their employees build a healthy intuition for how to use various kinds of agents to accomplish complex tasks.

If we see truly remarkable automation benefits from the LLM wave of technology, it’s not going to come from the first-wave of specific tools we build, but the output of a new class of LLM-informed users and developers. There is nowhere that you can simply acquire that talent, instead it’s talent that you have to develop inhouse, and involving more folks in iterative refinement of LLM-driven systems is the most effective approach that I’ve encountered.

We’ve taken a handful of different approaches here, all of which are currently in use. From earliest to latest, our approaches have been:

Being responsive to feedback is our primary mechanism for solving issues. This is both responding quickly in an internal #ai channel, but also skimming through workflows each day to see humans interacting, for better and for worse, with the agents. This is the most valuable ongoing source of improvement.
Owner-led refinement has been our intended primary mechanism, although in practice it’s more of the secondary mechanism. We store our prompts in Notion documents, where they can be edited by their owners in real-time. Permissions vary on a per-document basis, but most prompts are editable by anyone at the company, as we try to facilitate rapid learning.

Editable prompts alone aren’t enough, these prompts also need to be discoverable. To address that, whenever an action is driven by a workflow, we include a link to the prompt. For example, a Slack message sent by a chat bot will include a link to the prompt, as well a comment in Jira.
Claude-enhanced, owner-led refinement via the Datadog MCP to pull logs into the repository where the skills live has been fairly effective, although mostly as a technique used by the AI Engineering team rather than directly by owners. Skills are a bit of a platform, as they are used by many different workflows, so it may be inevitable that they are maintained by a central team rather than by workflow owners.
Dashboard tracking shows how often each workflow runs and errors associated with those runs. We also track how often each tool is used, including how frequently each skill is loaded.

My guess is that we will continue to add more refinement techniques as we go, without being able to get rid of any of the existing ones. This is sort of disappointing–I’d love to have the same result with fewer–but I think we’d be worse off if we cut any of them.

Next steps

What we don’t do yet, but is the necessary next step to making this truly useful, is to include a subjective post-workflow eval that determines whether the workflow was effective. While we have evals to evaluate workflows, this would be using evals to evaluate individual workflow runs, which would provide a level of very useful detail to understand.

How it’s going

In our experience thus far, there are roughly three workflow archetypes: chatbots, very well understood iterative workflows (e.g. applying :merge: reacji to merged PRs as discussed in code-driven workflows), and not-yet-well-understood workflows.

Once we build a code-driven workflow, they have always worked well for us, because we have built a very focused, well-understood solution at that point. Conversely, chatbots are an extremely broad, amorphous problem space, and I think post-run evals will provide a high quality dataset to improve them iteratively with a small amount of human-in-the-loop to nudge the evolution of their prompts and skills.

The open question, for us anyway, is how we do a better job of identifying and iterating on the not-yet-well-understood workflows. Ideally without requiring a product engineer to understand and implement each of them individually. We’ve not scalably cracked this one yet, and I do think scalably cracking it is the key to whether these internal agents are somewhat useful (frequently performed tasks performed by many people eventually get automated) and are truly transformative (a significant percentage of tasks, even infrequent ones performed by a small number of people get automated).

Building an internal agent: Subagent support

Wed, 31 Dec 2025 09:45:00 -0800

Most of the extensions to our internal agent have been the direct result of running into a problem that I couldn’t elegantly solve within our current framework. Evals, compaction, large-file handling all fit into that category. Subagents, allowing an agent to initiate other agents, are in a different category: I’ve frequently thought that we needed subagents, and then always found an alternative that felt more natural.

Eventually, I decided to implement them anyway, because it seemed like an interesting problem to reason through. Eventually I would need them… right? (Aside: I did, indeed, eventually use subagents to support code-driven workflows invoking LLMs.)

This is part of the Building an internal agent series.

Why subagents matter

“Subagents” is the name for allowing your agents to invoke other agents, which have their own system prompt, available tools, and context windows. Some of the reasons you’re likely to consider subagents:

Provide an effective strategy for context window management. You could provide them access to uploaded files, and then ask them to extract specific data from those files, without polluting your primary agent’s context window with the files’ content
You could use subagents to support concurrent work. For example, you could allow invocation of multiple subagents at once, and then join on the completion of all subagents. If your agent workflows are predominantly constrained by network IO (to e.g. model evaluation APIs), then this could support significant reduction in clock-time to complete your workflows
I think you could convince yourself that there are some security benefits to performing certain operations in subagents with less access. I don’t actually believe that’s meaningfully better, but you could at least introduce friction by ensuring that retrieving external resources and accessing internal resources can only occur in mutually isolated subagents

Of all these reasons, I think that either the first or the second will be most relevant to the majority of internal workflow developers.

How we implemented subagents

Our implementation for subagents is quite straightforward:

We define subagents in subagents/*.yaml, where each subagent has a prompt, allowed tools (or option to inherit all tools from parent agent), and a subset of the configurable fields from our agent configuration
Each agent is configured to allow specific subagents, e.g. the planning subagent
Agents invoke subagents via the subagent(agent_name, prompt, files) tool, which allows them to decide which virtual files are accessible within the subagent, and also the user prompt passed to the subagent (the subagent already has a default system prompt within its configuration)

This has worked fairly well. For example, supporting the quick addition of planning and think subagents which the parent agent can use to refine its work. We further refactored the implementation of the harness running agents to be equivalent to subagents, where effectively every agent is a subagent, and so forth.

How this has worked / what next

To be totally honest, I just haven’t found subagents to be particularly important to our current workflows. However, user-facing latency is a bit of an invisible feature, with it not mattering at all until at some point it starts subtly creating undesirable user workflows (e.g. starting a different task before checking the response), so I believe long-term this will be the biggest advantage for us.

Addendum: as alluded to in the introduction, this subagents functionality ended up being extremely useful when we introduced code-driven workflows, as it allows handing off control to the LLM for a very specific determination, before returning control to the code.

Building an internal agent: Code-driven vs LLM-driven workflows

Wed, 31 Dec 2025 09:30:00 -0800

When I started this project, I knew deep in my heart that we could get an LLM plus tool-usage to solve arbitrarily complex workflows. I still believe this is possible, but I’m no longer convinced this is actually a good solution. Some problems are just vastly simpler, cheaper, and faster to solve with software. This post talks about our approach to supporting both code and LLM-driven workflows, and why we decided it was necessary.

This is part of the Building an internal agent series.

Why determinism matters

When I joined Imprint, we already had a channel where folks would share pull requests for review. It wasn’t required to add pull requests to that channel, but it was often the fastest way to get someone to review it, particularly for cross-team pull requests.

I often start my day by skimming for pull requests that need a review in that channel, and quickly realized that often a pull request would get reviewed and merged without someone adding the :merged: reacji onto the chat. This felt inefficient, but also extraordinarily minor, and not the kind of thing I want to complain about. Instead, I pondered how I could solve it without requiring additional human labor.

So, I added an LLM-powered workflow to solve this. The prompt was straightforward:

Get the last 10 messages in the Slack channel
For each one, if there was exactly one Github pull request URL, extract that URL
Use the Github MCP to check the status of each of those URLs
Add the :merged: reacji to messages where the associated pull request was merged or closed

This worked so well! So, so well. Except, ahh, except that it sometimes decided to add :merged: to pull requests that weren’t merged. Then no one would look at those pull requests. So, it worked in concept–so much smart tool usage!–but in practice it actually didn’t solve the problem I was trying to solve: erroneous additions of the reacji meant folks couldn’t evaluate whether to look at a given pull request in the channel based on the reacji’s presence.

(As an aside, some people really don’t like the term reacji. Don’t complain to me about it, this is what Slack calls them.)

How we implemented support for code-driven workflows

Our LLM-driven workflows are orchestrated by a software handler. That handler works something like:

Trigger comes in, and the handler selects which configuration corresponds with the trigger
Handler uses that configuration and trigger to pull the associated prompt, load the approved tools, and generate the available list of virtual files (e.g. files attached to a Jira issue or Slack message)
Handler sends the prompt and available tools to an LLM, then coordinates tool calls based on the LLM’s response, including e.g. making virtual files available to tools. The handler also has termination conditions where it prevents excessive tool usage, and so on
Eventually the LLM will stop recommending tools, and the final response from the LLM will be used or discarded depending on the configuration (e.g. configuration can determine whether the final response is sent to Slack)

We updated our configuration to allow running in one of two configurations:

# this is default behavior if omitted
coordinator: llm

# this is code-driven workflow
coordinator: script
coordinator_script: scripts/pr_merged.py

When the coordinator is set to script, then instead of using the handler to determine which tools are called, custom Python is used. That Python code has access to the same tools, trigger data, and virtual files as the LLM-handling code. It can use the subagent tool to invoke an LLM where useful (and that subagent can have full access to tools as well), but LLM control only occurs when explicitly desired.

This means that these scripts–which are being written and checked in by our software engineers, going through code review and so on–have the same permission and capabilities as the LLM, although given it’s just code, any given commit could also introduce a new dependency, etc.

How’s it working? / Next steps?

Altogether, this has worked very well for complex workflows. I would describe it as a “solution of frequent resort”, where we use code-driven workflows as a progressive enhancement for workflows where LLM prompts and tools aren’t reliable or quick enough. We still start all workflows using the LLM, which works for many cases. When we do rewrite, Claude Code can almost always rewrite the prompt into the code workflow in one-shot.

Even as models get more powerful, relying on them narrowly in cases where we truly need intelligence, rather than for iterative workflows, seems like a long-term addition to our toolkit.

Building an internal agent: Logging and debugability

Wed, 31 Dec 2025 09:15:00 -0800

Agents are extremely impressive, but they also introduce a lot of non-determinism, and non-determinism means sometimes weird things happen. To combat that, we’ve needed to instrument our workflows to make it possible to debug why things are going wrong.

This is part of the Building an internal agent series.

Why logging matters

Whenever an agent does something sub-optimal, folks flag it as a bug. Often, the “bug” is ambiguity in the prompt that led to sub-optimal tool usage. That makes me feel better, but it doesn’t make the folks relying on these tools feel any better: they just expect the tools to work.

This means that debugging unexpected behavior is a significant part of rolling out agents internally, and it’s important to make it easy enough to do it frequently. If it takes too much time, effort or too many permissions, then your agents simply won’t get used.

How we implemented logging

Our agents run in an AWS Lambda, so the very first pass at logging was simply printing to standard out to be captured in the Lambda’s logs. This worked OK for the very first steps, but also meant that I had to log into AWS every time something went wrong, and even many engineers didn’t know where to find logs.

The second pass was creating the #ai-logs channel, where every workflow run shared its configuration, tools used, and a link to the AWS URL where logs could be found. This was a step up, but still required a bunch of log spelunking to answer basic questions.

The third pass, which is our current implementation, was integrating Datadog’s LLM Observability which provides an easy to use mechanism to view each span within the LLM workflow, making it straightforward to debug nuanced issues without digging through a bunch of logs. This is a massive improvement.

It’s also worth noting that the Datadog integration also made it easy to introduce dashboarding for our internal efforts, which has been a very helpful, missing ingredient to our work.

How is it working? / What’s next?

I’ll be honest: the Datadog LLM observability toolkit is just great. The only problem I have at this point is that we mostly constrain Datadog accounts to folks within the technology organization, so workflow debugging isn’t very accessible to folks outside that team. However, in practice there are very few folks who would be actively debugging these workflows who don’t already have access, so it’s more of a philosophical issue than a practical one.

Irrational Exuberance

Learning from Every's Compound Engineering

Sharing Claude transcripts.

Moved newsletter from Mailchimp to Buttondown.

Building internal agents

Building your intuition for agents

Why didn’t you just use X?

Final thoughts

Building an internal agent: Iterative prompt and skill refinement

Why does iterative refinement matter?

How are we enabling iterative refinement?

Next steps

How it’s going

Building an internal agent: Subagent support

Why subagents matter

How we implemented subagents

How this has worked / what next

Building an internal agent: Code-driven vs LLM-driven workflows

Why determinism matters

How we implemented support for code-driven workflows

How’s it working? / Next steps?

Building an internal agent: Logging and debugability

Why logging matters

How we implemented logging

How is it working? / What’s next?