Facilitating AI adoption at Imprint

I’ve been working on internal “AI” adoption, which is really LLM-tooling and agent adoption, for the past 18 months or so. This is a problem that I think is, at minimum, a side-quest for every engineering leader in the current era. Given the sheer number of folks working on this problem within their own company, I wanted to write up my “working notes” of what I’ve learned.

This isn’t a recommendation about what you should do, merely a recap of how I’ve approached the problem thus far, and what I’ve learned through ongoing iteration. I hope the thinking here will be useful to you, or at least validate some of what you’re experiencing in your rollout. The further you read, the more specific this will get, ending with cheap-turpentine-esque topics like getting agents to reliably translate human-readable text representations of Slack entities into mrkdwn formatting of the correct underlying entity.

Prework: building my intuition

As technologists, I think one of the basics we owe our teams is spending time working directly with new tools to develop an intuition for how they do, and don’t work. AI adoption is no different.

Towards that end, I started with a bit of reading, especially Chip Huyen’s AI Engineering, and then dove in a handful of bounded projects: building my rudimentary own agent platform using Claude code for implementation, creating a trivial MCP for searching my blog posts, and an agent to comment on Notion documents.

Each of these projects was two to ten hours, and extremely clarifying. Tool use is, in particular, something that seemed like magic until I implemented a simple tool-using agent, at which point it become something extremely non-magical that I could reason about and understand.

Our AI adoption strategy

Imprint’s general approach to refining AI adoption is strategy testing: identify a few goals, pick an initial approach, and then iterate rapidly in the details until the approach genuinely works. In an era of crushing optics, senior leaders immersing themselves in the details is one of our few defenses.

Shortly after joining, I partnered with the executive team to draft the above strategy for AI adoption. After a modest amount of debate, the pillars we landed on were:

1. Pave the path to adoption by removing obstacles to adoption, especially things like having to explicitly request access to tooling. There’s significant internal and industry excitemetn for AI adoption, and we should believe in our teams. If they aren’t adopting tooling, we predominantly focus on making it easier rather than spending time being skeptical or dismissive of their efforts towards adoption.
2. Opportunity for adoption is everywhere, rather than being isolated to engineering, customer service, or what not. To become a company that widely benefits from AI, we need to be solving the problem of adoption across all teams. It’s not that I believe we should take the same approach everywhere, but we need some applicable approach for each team.
3. Senior leadership leads from the front to ensure what we’re doing is genuinely useful, rather than getting caught up in what we’re measuring.

As you see from those principles, and my earlier comment, my biggest fear for AI adoption is that they can focus on creating the impression of adopting AI, rather than focusing on creating additional productivity. Optics are a core part of any work, but almost all interesting work occurs where optics and reality intersect, which these pillars aimed to support.

As an aside, in terms of the components of strategy in Crafting Engineering Strategy, this is really just the strategy’s policy. In addition, we used strategy testing to refine our approach, defined a concrete set of initial actions to operationalize it (they’re a bit too specific to share externally), and did some brief exploration to make sure I wasn’t overfitting on my prior work at Carta.

Documenting tips & tricks

My adoption step was collecting as many internal examples of tips and tricks as possible into a single Notion database. I took a very broad view on what qualified, with the belief that showing many different examples of using tools–especially across different functions–is both useful and inspiring.

I’ve continued extending this, with contributions from across the company, and it’s become a useful resource for both humans and bots alike to provide suggestions on approaching problems with AI tooling.

Centralizing our prompts

One of my core beliefs in our approach is that making prompts discoverable within the company is extremely valuable. Discoverability solves four distinct problems:

1. Creating visibility into what prompt’s can do (so others can be inspired to use them in similar scenarios). For example, that you can use our agents to comment on a Notion doc when it’s created, respond in Slack channels effectively, triage Jira tickets, etc
2. Showing what a good prompt looks like (so others can improve their prompts). For example, you can start moving complex configuration into tables and out of lists which are harder to read and accurately modify
3. Serving as a repository of copy-able sections to reuse across prompts. For example, you can copy one of our existing “Jira-issue triaging prompts” to start triaging a new Jira project
4. Prompts are joint property of a team or function, not the immutable construct of one person. For example, anyone on our Helpdesk team can improve the prompt responding to Helpdesk requests, not just one person with access to the prompt, and it’s not locked behind being comfortable with Git or Github (although I do imagine we’ll end up with more restrictions around editing our most important internal agents over time)
5. Identifying repeating prompt sub-components that imply missing or hard to use tools. For example, earlier versions of our prompts had a lot of confusion around how to specify Slack users and channels, which I got comfortable working around, but others did not

My core approach is that every agent’s prompt is stored in a single Notion database which is readable by everyone in the company. Most prompts are editable by everyone, but some have editing restrictions.

Here’s an example of a prompt we use for routing incoming Jira issues from Customer Support to the correct engineering team.

Here’s a second example, this time of responding to requests in our Infrastructure Engineering team’s request channel.

Pretty much all prompts end with an instruction to include a link to the prompt in the generated message. This ensures it’s easy to go from a mediocre response to the prompt-driving the response, so that you can fix it.

Adopting standard platform

In addition to collecting tips and prompts, the next obvious step for AI adoption is identifying a standard AI platform to be used within the company, e.g. ChatGPT, Claude, Gemini or what not.

We’ve gone with OpenAI for everyone. In addition to standardizing on a platform, we made sure account provisioning was automatic and in place on day one. To the surprise of no one who’s worked in or adjacent to IT, a lot of revolutionary general AI adoption is… really just account provisioning and access controls. These are the little details that can so easily derail the broader plan if you don’t dive into them.

Within Engineering, we also provide both Cursor and Claude. That said, the vast majority of our Claude usage is done via AWS Bedrock, which we use to power Claude Code… and we use Claude Code quite a bit.

Other AI tooling

While there’s a general industry push towards adopting more AI tooling, I find that a significant majority of “AI tools” are just SaaS vendors that talk about AI in their marketing pitches. We have continued to adopt vendors, but have worked internally to help teams evaluate which “AI tools” are meaningful.

We’ve spent a fair amount of time going deep on integrating with AI tooling for chat and IVR tooling, but that’s a different post entirely.

Metrics

Measuring AI adoption is, like all measurement topics, fraught. Altogether, I’ve found measuring tool adoption very instruction to identify the right questions to ask. Why haven’t you used Cursor? Or Claude Code? Or whatever? These are fascinating questions to dig into. I try to look at usage data at least once a month, with a particular focus on two questions:

1. For power adopters, what are they actually doing? Why do they find it useful?
2. For low or non-adopters, why aren’t they using the tooling? How could we help solve that for them?

At the core, I believe folks who aren’t adopting tools are rational non-adopters, and spending some time understanding the (appearance of) resistance goes further than top-down mandate. I think it’s often an education gap that is bridged easily enough. Conceivably, at some point I’ll discover a point of diminishing returns, where the lack of progress is stymied on folks who are rejecting AI tooling–or because the AI tooling isn’t genuinely useful–but I haven’t found that point yet.

Building internal agents

The next few sections are about building internal agents. The core implementation is a single stateless lambda which handles a wide variety of HTTP requests, similar-ish to Zapier. This is currently implemented in Python, and is roughly 3,000 lines of code, much of it dedicated to oddities like formatting Slack messages, etc.

For the record, I did originally attempt to do this within Zapier, but I found that Zapier simply doesn’t facilitate the precision I believe is necessary to do this effectively. I also think that Zapier isn’t particularly approachable for a non-engineering audience.

What has fueled adoption (especially for agents)

As someone who spent a long time working in platform engineering, I still want to believe that you can build a platform, and users will come. Indeed, I think it’s true that a small number of early adopters will come, if the problem is sufficiently painful for them, as was the case for Uber’s service migration (2014).

However, what we’ve found effective for driving adoption is basically the opposite of that. What’s really worked is the intersection of platform engineering and old-fashioned product engineering:

1. (product eng) find a workflow with a lot of challenges or potential impact
2. (product eng) work closely with domain experts to get the first version working
3. (platform eng) ensure that working solution is extensible by the team using it
4. (both) monitor adoption as indicator of problem-solution fit, or lack thereof

Some examples of the projects where we’ve gotten traction internally:

• Writing software with effective AGENTS.md files guiding use of tests, typechecking and linting
• Powering initial customer questions through chat and IVR
• Routing chat bots to steer questions to solve the problem, provide the the answer, or notify the correct responder
• Issue triaging for incoming tickets: tagging them, and assigning them to the appropriate teams
• Providing real-time initial feedback on routine compliance and legal questions (e.g. questions which occur frequently and with little deviation)
• Writing weekly priorities updates after pulling a wide range of resources (Git commits, Slack messages, etc)

For all of these projects that have worked, the formula has been the opposite of “build a platform and they will come.” Instead it’s required deep partnership from folks with experience building AI agents and using AI tooling to make progress. The learning curve for effective AI adoption in important or production-like workflows remains meaningfully high.

Configuring agents

Agents that use powerful tools represent a complex configuration problem. First, exposing too many tools–especially tools that the prompt author doesn’t effectively understand–makes it very difficult to create reliable workflows. For example, we have an exit_early command that allows terminating the agent early: this is very effective in many cases, but is also easy to break your bot. Similarly, we have a slack_chat command that allows posting across channels, which can support a variety of useful workflows (e.g. warm-handoffs of a question in one channel into a more appropriate alternative), but can also spam folks. Second, as tools get more powerful, they can introduce complex security scenarios.

To address both of these, we currently store configuration in a code-reviewed Git repository. Here’s an example of a JIRA project.

Here’s another for specifying a Slack responder bot.

Compared to a JSON file, we can statically type the configuration, and it’s easy to extend over time. For example, we might want to extend slack_chat to restrict which channels a given bot is allowed to publish into, which would be easy enough. For most agents today, the one thing not under Git-version control is the prompts themselves, which are versioned by Notion. However, we can easily require specific agents to use prompts within the Git-managed repository for sensitive scenarios.

After passing tests, linting and typechecking, the configurations are automatically deployed.

Resolving foreign keys

It’s sort of funny to mention, but one thing that has in practice really interfered with easily writing effective prompts is making it easy to write things like @Will Larson which is then translated into <@U12345> or whatever the appropriate Slack identifier is for a given user, channel, or user group. The same problem exists for Jira groups, Notion pages and databases, and so on.

This is a good example of where centralizing prompts is useful. I got comfortable pulling the unique identifiers myself, but it became evident that most others were not. This eventually ended with three tools for Slack resolution: slack_lookup which takes a list of references to lookup, slack_lookup_prefix which finds all Slack entities that start with a given prefix (useful to pull all channels or groups starting with @oncall-, for example, rather than having to hard-code the list in your prompt), and slack_search_name which uses string-distance to find potential matches (again, useful for dealing with typos).

If this sounds bewildering, it’s largely the result of Slack not exposing relevant APIs for this sort of lookup. Slack’s APIs want to use IDs to retrieve users, groups and channels, so you have to maintain your own cache of these items to perform a lookup. Performing the lookups, especially for users, is itself messy. Slack users have a minimum of three ways they might be referenced: user.profile.display_name, user.name, and user.real_name, only a subset of which are set for any given user. The correct logic here is, as best I can tell, to find a match against user.profile.display_name, then use that if it exists. Then do the same for user.name and finally user.real_name. If you take the first user that matches one of those three, you’ll use the wrong user in some scenarios.

In addition to providing tools to LLMs for resolving names, I also have a final mandatory check for each response to ensure the returned references refer to real items. If not, I inject which ones are invalid into the context window and perform an additional agent loop with only entity-resolution tools available. This feels absurd, but it was only at this point that things really started working consistently.

As an aside, I was embarassed by these screenshots, and earlier today I made the same changes for Notion pages and databases as I had previously for Slack.

Formatting

Similarly to foreign entity resolution, there’s a similar problem with Slack’s mrkdwn variant of Markdown and JIRA’s Atlassian Document Format: they’re both strict.

The tools that call into those APIs now have strict instructions on formatting. These had been contained in individual prompts, but they started showing up in every prompt, so I knew I needed to bring them into the agent framework itself rather than forcing every prompt-author to understand the problem.

My guess is that I need to add a validation step similar to the one I added for entity-resolution, and that until I do so, I’ll continue to have a small number of very infrequent but annoying rendering issues, To be honest, I personally don’t mind the rendering issues, but that creates a lot of uncertainty for others using agents, so I think solving them is a requirement.

Logging and debugging

Today, all logs, especially tool usage, are fed into two places. First, it goes into Datadog for full logging visibility. Second, and perhaps more usefully for non-engineers, they feed into a Slack channel, #ai-logs which create visibility into which tools are used and with which (potentially truncated) parameters.

Longer term, I imagine this will be exposed via a dedicated internal web UX, but generally speaking I’ve found that the subset of folks who are actively developing agents are pretty willing to deal with a bit of cruft. Similarly the folks who aren’t developing agents directly don’t really care, they want it to work perfectly every time, and aren’t spending time looking at logs.

Biggest remaining gap: universal platform for accessing user-scope MCP servers

The biggest internal opportunity that I see today is figuring out how to get non-engineers an experience equivalent to running Claude Code locally with all their favorite MCP servers plugged in. I’ve wanted ChatGPT or Claude.ai to provide this, but they don’t really quite get there, Claude Desktop is close, but is somewhat messy to configure as we think about finding a tool that we can easily allow everyone internally to customize and use on a daily basis.

I’m still looking for what the right tool is here. If anyone has any great suggestions that we can be somewhat confident will still exist in two years, and don’t require sending a bunch of internal data to a very early stage company, then I’m curious to hear!

What’s next?

You’re supposed to start a good conclusion with some sort of punchy anecdote that illuminates your overall thesis in a new way. I’m not sure if I can quite meet that bar, but the four most important ideas for me are:

1. We are still very early on AI adoption, so focusing on rate of learning is more valuable than anything else
2. If you want to lead an internal AI initiative, you simply must be using the tools, and not just ChatGPT, but building your own tool-using agent using only an LLM API
3. My experience is that real AI adoption on real problems is a complex blend of: domain context on the problem, domain experience with AI tooling, and old-fashioned IT issues. I’m deeply skeptical of any initiative for internal AI adoption that doesn’t anchor on all three of those. This is an advantage of earlier stage companies, because you can often find aspects of all three of those in a single person, or at least across two people. In larger companies, you need three different organizations doing this work together, this is just objectively hard
4. I think model selection matters a lot, but there are only 2-3 models you need at any given moment in time, and someone can just tell you what those 2-3 models are at any given moment. For example, GPT-4.1 is just exceptionally good at following rules quickly. It’s a great model for most latency-sensitive agents

I’m curious what other folks are finding!