Irrational Exuberance

Building internal agents

Thu, 01 Jan 2026 09:00:00 -0800

A few weeks ago in Facilitating AI adoption at Imprint, I mentioned our internal agent workflows that we are developing. This is not the core of Imprint–our core is powering co-branded credit card programs–and I wanted to document how a company like ours is developing these internal capabilities.

Building on that post’s ideas like a company-public prompt library for the prompts powering internal workflows, I wanted to write up some of the interesting problems and approaches we’ve taken as we’ve evolved our workflows, split into a series of shorter posts:

In the same spirit as the original post, I’m not writing these as an industry expert unveiling best practice, rather these are just the things that we’ve specifically learned along the way. If you’re developing internal frameworks as well, then hopefully you’ll find something interesting in these posts.

Building your intuition for agents

As more folks have read these notes, a recurring response has been, “How do I learn this stuff?” Although I haven’t spent time evaluating if this is the best way to learn, I can share what I have found effective:

Reading a general primer on how Large Language Models work, such as AI Engineering by Chip Huyen. You could also do a brief tutorial too, you don’t need the ability to create an LLM yourself, just a mental model of what they’re capable of
Build a script that uses a basic LLM API to respond to a prompt
Extend that script to support tool calling for some basic tools like searching files in a local git repository (or whatever)
Implement a tool_search tool along the lines of Anthropic Claude’s tool_search, which uses a separate context window to evaluate your current context window against available skills and return only the relevant skills to be used within your primary context window
Implement a virtual file system, such that tools can operate on references to files that are not within the context window. Also add a series of tools to operate on that virtual file system like load_file, grep_file, or whatnot
Support Agent Skills, particularly load_skills tool and enhancing the prompt with available skills
Write post-workflow eval that runs automatically after each workflow and evaluates the quality of the workflow run
Add context-window compaction support to keep context windows below a defined size Make sure that some of your tool responses are large enough to threaten your context-window’s limit, such that you’re forced to solve that problem

After working through the implementation of each of these features, I think you will have a strong foundation into how to build and extend these kinds of systems. The only missing piece is supporting code-driven agents, but unfortunately I think it’s hard to demonstrate the need of code-driven agents in simple examples, because LLM-driven agents are sufficiently capable to solve most contrived examples.

Why didn’t you just use X?

There are many existing agent frameworks, including OpenAI Agents SDK and Claude’s Agents SDK. Ultimately, I think these are fairly thin wrappers, and that you’ll learn a lot more by implementing these yourself, but I’m less confident that you’re better off long-term building your own framework.

My general recommendation would be to build your own to throw away, and then try to build on top of one of the existing frameworks if you find any meaningful limitations. That said, I really don’t regret the decision to build our own, because it’s just so simple from a code perspective.

Final thoughts

I think every company should be doing this work internally, very much including companies that aren’t doing any sort of direct AI work in their product. It’s very fun work to do, there’s a lot of room for improvement, and having an engineer or two working on this is a relatively cheap option to derisk things if AI-enhanced techniques continue to improve as rapidly in 2026 as they did in 2025.

Building an internal agent: Iterative prompt and skill refinement

Thu, 01 Jan 2026 08:30:00 -0800

Some of our internal workflows are being used quite frequently, and usage reveals gaps in the current prompts, skills, and tools. Here is how we’re working to iterate on these internal workflows.

This is part of the Building an internal agent series.

When companies push on AI-led automation, specifically meaning LLM agent-driven automation, there are two major goals. First is the short-term goal of increasing productivity. That’s a good goal. Second, and I think even more importantly, is the long-term goal of helping their employees build a healthy intuition for how to use various kinds of agents to accomplish complex tasks.

If we see truly remarkable automation benefits from the LLM wave of technology, it’s not going to come from the first-wave of specific tools we build, but the output of a new class of LLM-informed users and developers. There is nowhere that you can simply acquire that talent, instead it’s talent that you have to develop inhouse, and involving more folks in iterative refinement of LLM-driven systems is the most effective approach that I’ve encountered.

We’ve taken a handful of different approaches here, all of which are currently in use. From earliest to latest, our approaches have been:

Being responsive to feedback is our primary mechanism for solving issues. This is both responding quickly in an internal #ai channel, but also skimming through workflows each day to see humans interacting, for better and for worse, with the agents. This is the most valuable ongoing source of improvement.
Owner-led refinement has been our intended primary mechanism, although in practice it’s more of the secondary mechanism. We store our prompts in Notion documents, where they can be edited by their owners in real-time. Permissions vary on a per-document basis, but most prompts are editable by anyone at the company, as we try to facilitate rapid learning.

Editable prompts alone aren’t enough, these prompts also need to be discoverable. To address that, whenever an action is driven by a workflow, we include a link to the prompt. For example, a Slack message sent by a chat bot will include a link to the prompt, as well a comment in Jira.
Claude-enhanced, owner-led refinement via the Datadog MCP to pull logs into the repository where the skills live has been fairly effective, although mostly as a technique used by the AI Engineering team rather than directly by owners. Skills are a bit of a platform, as they are used by many different workflows, so it may be inevitable that they are maintained by a central team rather than by workflow owners.
Dashboard tracking shows how often each workflow runs and errors associated with those runs. We also track how often each tool is used, including how frequently each skill is loaded.

My guess is that we will continue to add more refinement techniques as we go, without being able to get rid of any of the existing ones. This is sort of disappointing–I’d love to have the same result with fewer–but I think we’d be worse off if we cut any of them.

Next steps

What we don’t do yet, but is the necessary next step to making this truly useful, is to include a subjective post-workflow eval that determines whether the workflow was effective. While we have evals to evaluate workflows, this would be using evals to evaluate individual workflow runs, which would provide a level of very useful detail to understand.

How it’s going

In our experience thus far, there are roughly three workflow archetypes: chatbots, very well understood iterative workflows (e.g. applying :merge: reacji to merged PRs as discussed in code-driven workflows), and not-yet-well-understood workflows.

Once we build a code-driven workflow, they have always worked well for us, because we have built a very focused, well-understood solution at that point. Conversely, chatbots are an extremely broad, amorphous problem space, and I think post-run evals will provide a high quality dataset to improve them iteratively with a small amount of human-in-the-loop to nudge the evolution of their prompts and skills.

The open question, for us anyway, is how we do a better job of identifying and iterating on the not-yet-well-understood workflows. Ideally without requiring a product engineer to understand and implement each of them individually. We’ve not scalably cracked this one yet, and I do think scalably cracking it is the key to whether these internal agents are somewhat useful (frequently performed tasks performed by many people eventually get automated) and are truly transformative (a significant percentage of tasks, even infrequent ones performed by a small number of people get automated).

Building an internal agent: Subagent support

Wed, 31 Dec 2025 09:45:00 -0800

Most of the extensions to our internal agent have been the direct result of running into a problem that I couldn’t elegantly solve within our current framework. Evals, compaction, large-file handling all fit into that category. Subagents, allowing an agent to initiate other agents, are in a different category: I’ve frequently thought that we needed subagents, and then always found an alternative that felt more natural.

Eventually, I decided to implement them anyway, because it seemed like an interesting problem to reason through. Eventually I would need them… right? (Aside: I did, indeed, eventually use subagents to support code-driven workflows invoking LLMs.)

This is part of the Building an internal agent series.

Why subagents matter

“Subagents” is the name for allowing your agents to invoke other agents, which have their own system prompt, available tools, and context windows. Some of the reasons you’re likely to consider subagents:

Provide an effective strategy for context window management. You could provide them access to uploaded files, and then ask them to extract specific data from those files, without polluting your primary agent’s context window with the files’ content
You could use subagents to support concurrent work. For example, you could allow invocation of multiple subagents at once, and then join on the completion of all subagents. If your agent workflows are predominantly constrained by network IO (to e.g. model evaluation APIs), then this could support significant reduction in clock-time to complete your workflows
I think you could convince yourself that there are some security benefits to performing certain operations in subagents with less access. I don’t actually believe that’s meaningfully better, but you could at least introduce friction by ensuring that retrieving external resources and accessing internal resources can only occur in mutually isolated subagents

Of all these reasons, I think that either the first or the second will be most relevant to the majority of internal workflow developers.

How we implemented subagents

Our implementation for subagents is quite straightforward:

We define subagents in subagents/*.yaml, where each subagent has a prompt, allowed tools (or option to inherit all tools from parent agent), and a subset of the configurable fields from our agent configuration
Each agent is configured to allow specific subagents, e.g. the planning subagent
Agents invoke subagents via the subagent(agent_name, prompt, files) tool, which allows them to decide which virtual files are accessible within the subagent, and also the user prompt passed to the subagent (the subagent already has a default system prompt within its configuration)

This has worked fairly well. For example, supporting the quick addition of planning and think subagents which the parent agent can use to refine its work. We further refactored the implementation of the harness running agents to be equivalent to subagents, where effectively every agent is a subagent, and so forth.

How this has worked / what next

To be totally honest, I just haven’t found subagents to be particularly important to our current workflows. However, user-facing latency is a bit of an invisible feature, with it not mattering at all until at some point it starts subtly creating undesirable user workflows (e.g. starting a different task before checking the response), so I believe long-term this will be the biggest advantage for us.

Addendum: as alluded to in the introduction, this subagents functionality ended up being extremely useful when we introduced code-driven workflows, as it allows handing off control to the LLM for a very specific determination, before returning control to the code.

Building an internal agent: Code-driven vs LLM-driven workflows

Wed, 31 Dec 2025 09:30:00 -0800

When I started this project, I knew deep in my heart that we could get an LLM plus tool-usage to solve arbitrarily complex workflows. I still believe this is possible, but I’m no longer convinced this is actually a good solution. Some problems are just vastly simpler, cheaper, and faster to solve with software. This post talks about our approach to supporting both code and LLM-driven workflows, and why we decided it was necessary.

This is part of the Building an internal agent series.

Why determinism matters

When I joined Imprint, we already had a channel where folks would share pull requests for review. It wasn’t required to add pull requests to that channel, but it was often the fastest way to get someone to review it, particularly for cross-team pull requests.

I often start my day by skimming for pull requests that need a review in that channel, and quickly realized that often a pull request would get reviewed and merged without someone adding the :merged: reacji onto the chat. This felt inefficient, but also extraordinarily minor, and not the kind of thing I want to complain about. Instead, I pondered how I could solve it without requiring additional human labor.

So, I added an LLM-powered workflow to solve this. The prompt was straightforward:

Get the last 10 messages in the Slack channel
For each one, if there was exactly one Github pull request URL, extract that URL
Use the Github MCP to check the status of each of those URLs
Add the :merged: reacji to messages where the associated pull request was merged or closed

This worked so well! So, so well. Except, ahh, except that it sometimes decided to add :merged: to pull requests that weren’t merged. Then no one would look at those pull requests. So, it worked in concept–so much smart tool usage!–but in practice it actually didn’t solve the problem I was trying to solve: erroneous additions of the reacji meant folks couldn’t evaluate whether to look at a given pull request in the channel based on the reacji’s presence.

(As an aside, some people really don’t like the term reacji. Don’t complain to me about it, this is what Slack calls them.)

How we implemented support for code-driven workflows

Our LLM-driven workflows are orchestrated by a software handler. That handler works something like:

Trigger comes in, and the handler selects which configuration corresponds with the trigger
Handler uses that configuration and trigger to pull the associated prompt, load the approved tools, and generate the available list of virtual files (e.g. files attached to a Jira issue or Slack message)
Handler sends the prompt and available tools to an LLM, then coordinates tool calls based on the LLM’s response, including e.g. making virtual files available to tools. The handler also has termination conditions where it prevents excessive tool usage, and so on
Eventually the LLM will stop recommending tools, and the final response from the LLM will be used or discarded depending on the configuration (e.g. configuration can determine whether the final response is sent to Slack)

We updated our configuration to allow running in one of two configurations:

# this is default behavior if omitted
coordinator: llm

# this is code-driven workflow
coordinator: script
coordinator_script: scripts/pr_merged.py

When the coordinator is set to script, then instead of using the handler to determine which tools are called, custom Python is used. That Python code has access to the same tools, trigger data, and virtual files as the LLM-handling code. It can use the subagent tool to invoke an LLM where useful (and that subagent can have full access to tools as well), but LLM control only occurs when explicitly desired.

This means that these scripts–which are being written and checked in by our software engineers, going through code review and so on–have the same permission and capabilities as the LLM, although given it’s just code, any given commit could also introduce a new dependency, etc.

How’s it working? / Next steps?

Altogether, this has worked very well for complex workflows. I would describe it as a “solution of frequent resort”, where we use code-driven workflows as a progressive enhancement for workflows where LLM prompts and tools aren’t reliable or quick enough. We still start all workflows using the LLM, which works for many cases. When we do rewrite, Claude Code can almost always rewrite the prompt into the code workflow in one-shot.

Even as models get more powerful, relying on them narrowly in cases where we truly need intelligence, rather than for iterative workflows, seems like a long-term addition to our toolkit.

Building an internal agent: Logging and debugability

Wed, 31 Dec 2025 09:15:00 -0800

Agents are extremely impressive, but they also introduce a lot of non-determinism, and non-determinism means sometimes weird things happen. To combat that, we’ve needed to instrument our workflows to make it possible to debug why things are going wrong.

This is part of the Building an internal agent series.

Why logging matters

Whenever an agent does something sub-optimal, folks flag it as a bug. Often, the “bug” is ambiguity in the prompt that led to sub-optimal tool usage. That makes me feel better, but it doesn’t make the folks relying on these tools feel any better: they just expect the tools to work.

This means that debugging unexpected behavior is a significant part of rolling out agents internally, and it’s important to make it easy enough to do it frequently. If it takes too much time, effort or too many permissions, then your agents simply won’t get used.

How we implemented logging

Our agents run in an AWS Lambda, so the very first pass at logging was simply printing to standard out to be captured in the Lambda’s logs. This worked OK for the very first steps, but also meant that I had to log into AWS every time something went wrong, and even many engineers didn’t know where to find logs.

The second pass was creating the #ai-logs channel, where every workflow run shared its configuration, tools used, and a link to the AWS URL where logs could be found. This was a step up, but still required a bunch of log spelunking to answer basic questions.

The third pass, which is our current implementation, was integrating Datadog’s LLM Observability which provides an easy to use mechanism to view each span within the LLM workflow, making it straightforward to debug nuanced issues without digging through a bunch of logs. This is a massive improvement.

It’s also worth noting that the Datadog integration also made it easy to introduce dashboarding for our internal efforts, which has been a very helpful, missing ingredient to our work.

How is it working? / What’s next?

I’ll be honest: the Datadog LLM observability toolkit is just great. The only problem I have at this point is that we mostly constrain Datadog accounts to folks within the technology organization, so workflow debugging isn’t very accessible to folks outside that team. However, in practice there are very few folks who would be actively debugging these workflows who don’t already have access, so it’s more of a philosophical issue than a practical one.

Building an internal agent: Evals to validate workflows

Wed, 31 Dec 2025 09:00:00 -0800

Whenever a new pull request is submitted to our agent’s GitHub repository, we run a bunch of CI/CD operations on it. We run an opinionated linter, we run typechecking, and we run a bunch of unittests. All of these work well, but none of them test entire workflows end-to-end. For that end-to-end testing, we introduced an eval pipeline.

This is part of the Building an internal agent series.

Why evals matter

The harnesses that run agents have a lot of interesting nuance, but they’re generally pretty simple: some virtual file management, some tool invocation, and some context window management. However, it’s very easy to create prompts that don’t work well, despite the correctness of all the underlying pieces. Evals are one tool to solve that, exercising your prompts and tools together and grading the results.

How we implemented it

I had the good fortune to lead Imprint’s implementation of Sierra for chat and voice support, and I want to acknowledge that their approach has deeply informed my view of what does and doesn’t work well here.

The key components of Sierra’s approach are:

Sierra implements agents as a mix of React-inspired code that provide tools and progressively-disclosed context, and a harness runner that runs that software.
Sierra allows your code to assign tags to conversations such as “otp-code-sent” or “lang-spanish” which can be used for filtering conversations, as well as other usecases discussed shortly.
Every tool implemented for a Sierra agent has both a true and a mock implementation. For example, for a tool that searches a knowledge base, the true version would call its API directly, and the mock version would return a static (or locally generated) version for use in testing.
Sierra names their eval implementation as “simulations.” You can create any number of simulations either in code or via the UI-driven functionality.
Every evaluation has an initial prompt, metadata about the situation that’s available to the software harness running the agent, and criteria to evaluate whether an evaluation succeeds.
These evaluation criteria are both subjective and objective. The subjective criteria are “agent as judge” to assess whether certain conditions were met (e.g. was the response friendly?). The objective criteria are whether specific tags (“login-successful”) were, or were not (“login-failed”) added to a conversation.

Then when it comes to our approach, we basically just reimplemented that approach as it’s worked well for us. For example, the following image is the configuration for an eval we run.

Then whenever a new PR is opened, these run automatically along with our other automation.

While we largely followed the map laid out by Sierra’s implementation, we did diverge on the tags concept. For objective evaluation, we rely exclusively on tools that are, or are not, called. Sierra’s tag implementation is more flexible, but since our workflows are predominantly prompt-driven rather than code-driven, it’s not an easy one for us to adopt

Altogether, following this standard implementation worked well for us.

How is it working?

Ok, this is working well, but not nearly as well as I hoped it would. The core challenge is the non-determinism introduced by these eval tests, where in practice there’s very strong signal when they all fail, and strong signal when they all pass, but most runs are in between those two. A big part of that is sloppy eval prompts and sloppy mock tool results, and I’m pretty confident I could get them passing more reliably with some effort (e.g. I did get our Sierra tests almost always passing by tuning them closely, although even they aren’t perfectly reliable).

The biggest issue is that our reliance on prompt-driven workflows rather than code-driven workflows introduces a lot of non-determinism, which I don’t have a way to solve without the aforementioned prompt and mock tuning.

What’s next?

There are three obvious follow ups:

More tuning on prompts and mocked tool calls to make the evals more probabilistically reliable
I’m embarrassed to say it out loud, but I suspect we need to retry failed evals to see if they pass e.g. “at least once in three tries” to make this something we can introduce as a blocking mechanism in our CI/CD
This highlights the general limitation of LLM-driven workflows, and I suspect that I’ll have to move more complex workflows away from LLM-driven workflows to get them to work more consistently

Altogether, I’m very glad that we introduced evals, they are an essential mechanism for us to evaluate our workflows, but we’ve found them difficult to consistently operationalize as something we can rely on as a blocking tool rather than directionally relevant context.

Building an internal agent: Triggers

Wed, 31 Dec 2025 08:00:00 -0800

An internal agent only provides value when its workflows are initiated. Building out a library of workflow initializations, which we call triggers, is a core part of building an internal agent.

This is part of the Building an internal agent series.

Why triggers matter

While there’s a lot of chatter about AI empowering employees to trivially automate their day-to-day workflows, building these workflows requires an intuition about how agents work that requires iterative learning to develop, and few workplaces are providing the tools to facilitate that iterative learning. Easy triggers, combined with easy prompt iteration, are a foundational part of iterative learning.

More practically, triggers are also the mechanisms that initiate workflows, so nothing else matters if they aren’t effective and usable.

Why not Zapier or n8n?

The obvious question here is “why not Zapier?” and “why not n8n?”, both of which would have solved this triggering problem in its entirety. For what it’s worth, you still could use Zapier or n8n to trigger agent workflows using a custom webhook trigger, so these approaches aren’t incompatible. That said, for me there are only a small number of workflows I thought would matter, and I wanted to have full control of the nuances as we worked on the problem.

The “full control” piece ties back to one of my underlying theses of this work: the quality details facilitate adoption in a way that Zapier integration’s constraints simply do not. This is why I think internal agents need to spend so much time managing things like Slack-entity resolution to become a seamless experience.

How we implemented triggers

We’ve implemented these triggers, and in this order:

Notion webhooks can be configured to fire on any page or database modifications, including things going from “draft” status to “ready for review” status or other more nuanced changes. We’ve updated all our Request for Comment and other structured document databases to hook into these.

The triggers drive a variety of workflows. On the simpler end, they can be used to assign additional reviewers to a document based on topic, and on the more complex end they can use a prompt to provide in-line comments with suggestions.
Slack messages can trigger responses or other workflows. This is dependent on which channels the bot has been invited into, and I subsequently made a trigger to capture channel creation to support auto-joining new channels to increase availability. (Channels that are auto-joined have a simple, default prompt that only responds when the bot is mentioned directly, to avoid wearing out its welcome.)

We also got this working in private channels (not the auto-join component, just responding to messages). The hard part was ensuring that logging didn’t capture any evidence of joining or responding in those private channels. Exfiltrating private messages would have been a very easy way to quickly lose trust.
Jira webhooks can be configured to trigger notifications on issue creation, updates, comments and so on.
Slack reacji were a later addition, where we added support for listening to Slack reacji in either a given channel or in any channel where the bot is present. This has made it possible to quickly implement a trigger where the :jira: reacji in any channel turns a thread into a ticket, routing it using centralized routing instructions (which are imperfect, but much better than the average individual’s ability to route across many different teams).
Scheduled events are our most recent addition, allowing periodic triggers. I’ve done two distinct v0 implementations, with the first relying on Slack workflows publishing into new channels as triggers, and the other relying on Github actions. Both work well enough, despite being a bit messy.

Now that we have a fairly large category, and have updated our AGENTS.md to reflect our existing integrations, adding new sorts of triggers is generally pasting documentation into Claude and waiting a few minutes.

Trigger authn and authz

When it comes to authentication and authorization, where possible, we’ve adopted the OAuth2 approach of authorization tokens and SSL. However, that isn’t possible for scenarios like Slack where we can’t inject an authorization token into the requests. In those situations we rely on the application-specific security mechanisms provided by the underlying platform, such as Slack’s mechanisms for verifying requests.

How is it working? What’s next?

Our current set of triggers are working well. The next task here–probably a two sentence prompt and ten minutes of Claude code away–is adding a trigger on more generic webhooks that includes the whole incoming message into the context window. This hasn’t been necessary yet, but is a useful catch-all to support future workflows.

In particular, it would be straightforward to pair with an existing Zapier subscription, which would offload supporting certain esoteric triggers to their large catalog of integrations, while still getting the control and nuance of the internal agents.

Building an internal agent: Context window compaction

Fri, 26 Dec 2025 09:00:00 -0700

Although my model of choice for most internal workflows remains ChatGPT 4.1 for its predictable speed and high-adherence to instructions, even its 1,047,576-token context window can run out of space. When you run out of space in the context window, your agent either needs to give up, or it needs to compact that large context window into a smaller one. Here are our notes on implementing compaction.

This is part of the Building an internal agent series.

Why compaction matters

Long-running workflows with many tool calls or user messages, along with any workflow dealing with large files, often run out of space in their context window. Although context window exhaustion is not relevant in most cases you’ll find for internal agents, ultimately it’s not possible to implement a robust, reliable agent without solving for this problem, and compaction is a straightforward solution.

How we implemented it

Initially, in the beautiful moment where we assumed compaction wouldn’t be a relevant concern to our internal workflows, we implemented an extremely naive solution to compaction: if we ever ran out of tokens, we discarded older tool responses until we had more space, then continued. Because we rarely ran into compaction, the fact that this worked poorly wasn’t a major issue, but eventually the inelegance began to weigh on me as we started dealing with more workflows with large files.

In our initial brainstorm on our 2nd iteration of compaction, I initially got anchored on this beautiful idea that compaction should be sequenced after implementing support for subagents, but I was never able to ground that intuition in a concrete reason why it was necessary, and we implemented compaction without subagent support.

The gist of our approach to compaction is:

After every user message (including tool responses), add a system message with the consumed and available tokens in the context window. In that system message, we also include the updated list of available files that can be read from
User messages and tool responses greater than 10,000 tokens are exposed as a new “virtual file”, with only their first 1,000 tokens included in the context window. The agent must use file manipulation tools to read more than those first 1,000 tokens (both 1k and 10k are configurable values)
Add a set of “base tools” that are always available to agents, specifically including the virtual file manipulation tools, as we’d finally reached a point where most agents simply could not operate without a large number of mostly invisible internal tools. These tools were file_read which can read entire files, lines ranges within a file, or byte ranges within a file, and file_regex which is similar but performs a regex scan against a file up to a certain number of matches.

Every use of a file is recorded in the files data, so the agent knows what has and hasn’t been read into the context window (particularly relevant for preloaded files), along the lines of:
```
<files>
 <file id='a' name='image.png' size='32kb'>
 <file_read />
 </file>
 <file id='a' name='image.png' size='32kb'>
 <file_read start_line=10 end_line=20 />
 </file>
</files>
```
This was surprisingly annoying to implement cleanly, mostly because I came onto this idea after iteratively building the agent as a part-time project for several months. If I could start over, I would start with files as a core internal construct, rather than adding it on later.
If a message pushed us over 80% (configurable value) of the model’s available context window, use the compaction prompt that Reddit claims Claude Code uses. The prompt isn’t particularly special, it just already exists and seems pretty good
After compacting, add the prior context window as a virtual file to allow the agent to retrieve pieces of context that it might have lost

Each of these steps is quite simple, but in combination they really do provide a fair amount of power for handling complex, prolonged workflows. Admittedly, we still have a configurable cap on the number of tools that can be called in a workflow (to avoid agents spinning out), but this means that agents dealing with large or complex data are much more likely to succeed usefully.

How is it working? / What’s next?

Whereas for most of our new internal agent features, there are obvious problems or iterations, this one feels like it’s good enough to forget for a long, long time. There are two reasons for this: first, most of our workflows don’t require large context windows, and, second, honestly this seems to work quite well.

If context windows get significantly larger in the future, which I don’t see too much evidence will happen at this moment in time, then we will simply increase some of the default values to use more tokens, but the core algorithm here seems good enough.

Building an internal agent: Progressive disclosure and handling large files

Fri, 26 Dec 2025 08:00:00 -0700

One of the most useful initial extensions I made to our workflows was injecting associated images into the context window automatically, to improve the quality of responses to tickets and messages that relied heavily on screenshots. This was quick and made the workflows significantly more powerful.

More recently, there are a number of workflows attempting to operate on large complex files like PDFs or DOCXs, and the naive approach of shoving them into the context window hasn’t worked particularly well. This post explains how we’ve adapted the principle of progressive disclosure to allow our internal agents to work with large files.

This is part of the Building an internal agent series.

Large files and progressive disclosure

Progressive disclosure is the practice of limiting what is added to the context window to the minimum necessary amount, and adding more detail over time as necessary.

A good example of progressive disclosure is how agent skills are implemented:

Initially, you only add the description of each available skill into the context window
You then load the SKILL.md on demand
The SKILL.md can specify other files to be further loaded as helpful

In our internal use-case, we have skills for JIRA formatting, Slack formatting, and Notion formatting. Some workflows require all three, but the vast majority of workflows require at most one of these skills, and it’s straightforward for the agent to determine which are relevant to a given task.

File management is a particularly interesting progressive disclosure problem, because files are so helpful in many scenarios, but are also so very large. For example, requests for help in Slack are often along the lines of “I need help with this login issue ”, which is impossible to solve without including that image into the context window. In other workflows, you might want to analyze a daily data export in a very large PDF which is 5-10MB as a PDF, but only 10-20kb of tables and text when extracted from the PDF. This gets even messier when the goal is to compare across multiple PDFs, each of which is quite large.

Our approach

Our high-level approach to the large-file problem is as follows:

Always include metadata about available files in the prompt, similar to the list of available skills. This will look something like:
```
Files:
 - id: f_a1
 name: my_image.png
 size: 500,000
 preloaded: false
 - id: f_b3
 name: ...
```
The key thing is that each id is a reference that the agent is able to pass to tools. This allows it to operate on files without loading their context into the context window.
Automatically preload the first N kb of files into the context window, as long as they are appropriate mimetypes for loading (png, pdf, etc). This is per-workflow configurable, and could be set as low as 0 if a given workflow didn’t want to preload any files.

I’m still of mixed minds whether preloading is worth doing, as it takes some control away from the agent.
Provide three tools for operating on files:
- load_file(id) loads an entire file into the context window
- peek_file(id, start, stop) loads a section of a file into the context window
- extract_file(id) transforms PDFs, PPTs, DOCX and so on into simplified textual versions
Provide a large_files skill which explains how and when to use the above tools to work with large files. Generally, it encourages using extract_file on any PDF, DOCX or PPT file that it wants to work with, and otherwise loading or peeking depending on the available space in the context window

This approach was quick to implement, and provides significantly more control to the agent to navigate a wide variety of scenarios involving large files. It’s also a good example of how the “glue layer” between LLMs and tools is actually a complex, sophisticated application layer rather than merely glue.

How is this working?

This has worked well. In particular, one of our internal workflows oriented around giving feedback about documents attached to a ticket, in comparison to other similar, existing documents. The workflow simply did not work at all prior to this approach, and now works fairly well without workflow-specific support for handling these sorts of large files, because the large_files skill handles that in a reusable fashion without workflow authors being aware of it.

What next?

Generally, this feels like a stand-alone set of functionality that doesn’t require significant future investment, but there are three places where we will need to continue building:

Until we add subagent support, our capabilities are constrained. In many cases, the ideal scenario of dealing with a large file is opening it in a subagent with a large context window, asking that subagent to summarize its contents, and then taking that summary into the primary agent’s context window.
It seems likely that extract_file should be modified to return a referencable, virtual file_id that is used with peek_file and load_file rather than returning contents directly. This would make for a more robust tool even when extracting from very large files. In practice, extracted content has always been quite compact.
Finally, operating within an AWS Lambda requires pure Python packages, and ultimately pure Python is not very fast at parsing complex XML-derived document formats like DOCX. Ultimately, we could solve this by adding a layer to our lambda with the lxml dependencies in it, and at some point we might.

Altogether, a very helpful extension for our internal workflows.

Building an internal agent: Adding support for Agent Skills

Fri, 26 Dec 2025 07:00:00 -0700

When Anthropic introduced Agent Skills, I was initially a bit skeptical of the problem they solved–can we just use prompts and tools?–but I’ve subsequently come to appreciate them, and have explicitly implemented skills in our internal agent framework. This post talks about the problem skills solves, how the engineering team at Imprint implemented them, how well they’ve worked for us, and where we might work with them next.

This is part of the Building an internal agent series.

What problem do Agent Skills solve?

Agent Skills are a series of techniques that solve two important workflow problems:

use progressive disclosure to more effectively utilize the constrained context windows, minimizing conflicting or unnecessary context in the context window
provide reusable snippets for solving recurring problems to avoid individual workflow-creators having to solve recurring problems like e.g. Slack formatting or dealing with large files

These problems initially seemed very insignificant when we started building out our internal workflows, but once the number of internal workflows reached into the dozens, both become difficult to manage. Without reusable snippets, I lost the leverage to improve all workflows at once, and without progressive disclosure the agents would get a vast amount of irrelevant content that could confuse them, particularly when it came to things like inconsistencies between Markdown and slack’s mrkdwn formatting language, both of which are important to different tools used by our workflows.

How we implemented Agent Skills

As a disclaimer, I recognize that it’s not necessary to implement agent skills, as you can integrate with e.g. Claude’s Agent Skills support for APIs. However, one of our design decisions is being largely platform agnostic, such that we can switch across model providers, and consequently we decided to implement skills within our framework.

With that out of the way, we started implementing by reviewing the Agent Skills documentation at agentskills.io, and cloning their Python reference implementation skills-ref into our repository to make it accessible to Claude Code.

The resulting implementation has these core features:

Skills are in skills/ repository, with each skill consisting of its own sub-directory with a SKILL.md

Each skill is a Markdown file with metadata along these lines:

---
name: pdf-processing
description: Extract text and tables...
metadata:
 author: example-org
 version: "1.0"
---

The list of available skills–including their description from metadata–is injected into the system prompt at the beginning of each workflow, and the load_skills tool is available to the agent to load the entire file into the context window.
Updated workflow configuration to optionally specify required, allowed, and prohibited skills to modify the list of exposed skills injected into the system prompt.

My guess is that requiring specific skills for a given workflow is a bit of an anti-pattern, “just let the agent decide!”, but it was trivial to implement and the sort of thing that I could imagine is useful in the future.
Used the Notion MCP to retrieve all the existing prompts in our prompt repository, identify existing implicit skills in the prompts we had created, write those initial skills, and identify which Notion prompts to edit to eliminate the now redundant sections of their prompts.

Then we shipped it into production.

How they’ve worked

Humans make mistakes all the time. For example, I’ve seen many dozens of JIRA tickets from humans that don’t explain the actual problem they are having. People are used to that, and when a human makes a mistake, they blame the human. However, when agents make a mistake, a surprising percentage of people view it as a fundamental limitation of agents as a category, rather than thinking that, “Oh, I should go update that prompt.”

Skills have been extremely helpful as the tool to continue refining down these edge cases where we’ve relied on implicit behavior because specifying the exact behavior was simply overwhelming. As one example, we ask that every Slack message end with a link to the prompt that drove the response. That always worked, but the details of the formatting would vary in an annoying, distracting way: sometimes it would be the equivalent of [title](link), sometimes link, sometimes [link](link). With skills, it is now (almost always) consistent, without anyone thinking to include those instructions in their workflow prompts.

Similarly, handling large files requires a series of different tools that benefit from In-Context Learning (aka ICL, which is a fancy term for including a handful of examples of correct and incorrect usage), which absolutely no one is going to add to their workflow prompt but is extremely effective at improving how the workflow uses those tools.

For something that I was initially deeply skeptical about, I now wish I had implemented skills much earlier.

Where we might go next

While our skills implementation is working well today, there are a few opportunities I’d like to take advantage of in the future:

Add a load_subskill skill to support files in skills/{skill}/* beyond the SKILL.md. So far, this hasn’t been a major blocker, but as some skills get more sophisticated, the ability to split varied use-cases into distinct files would improve our ability to use skills for progressive disclosure
One significant advantage that Anthropic has over us is their sandboxed Python interpreter, which allows skills to include entire Python scripts to be specified and run by tools. For example, a script for parsing PDFs might be included in a skill, which is extremely handy. We don’t currently have a sandboxed interpreter handy for our agents, but this could, in theory anyway, significantly cut down on the number of custom skills we need to implement.

At a minimum, it would do a much better job at operations that require reliable math versus relying on the LLM to do its best at performing math-y operations.

I think both of these are actually pretty straightforward to implement. The first is just a simple feature that Claude could implement in a few minutes. The latter feels annoying to implement, but could also be implemented in less than an hour by running a second lambda running Nodejs with Pyodide, and exposing access to that lambda as a tool. It’s just so inelegant for a Python process to call a Nodejs process to run sandboxed Python that I haven’t done it quite yet.

Irrational Exuberance

Building internal agents

Building your intuition for agents

Why didn’t you just use X?

Final thoughts

Building an internal agent: Iterative prompt and skill refinement

Why does iterative refinement matter?

How are we enabling iterative refinement?

Next steps

How it’s going

Building an internal agent: Subagent support

Why subagents matter

How we implemented subagents

How this has worked / what next

Building an internal agent: Code-driven vs LLM-driven workflows

Why determinism matters

How we implemented support for code-driven workflows

How’s it working? / Next steps?

Building an internal agent: Logging and debugability

Why logging matters

How we implemented logging

How is it working? / What’s next?

Building an internal agent: Evals to validate workflows

Why evals matter

How we implemented it

How is it working?

What’s next?

Building an internal agent: Triggers

Why triggers matter

Why not Zapier or n8n?

How we implemented triggers

Trigger authn and authz

How is it working? What’s next?

Building an internal agent: Context window compaction

Why compaction matters

How we implemented it

How is it working? / What’s next?

Building an internal agent: Progressive disclosure and handling large files

Large files and progressive disclosure

Our approach

How is this working?

What next?

Building an internal agent: Adding support for Agent Skills

What problem do Agent Skills solve?

How we implemented Agent Skills

How they’ve worked

Where we might go next