Commenting on Notion docs via OpenAI and Zapier.

llm (20), notion (2)

One of my side quests at work is to get a simple feedback loop going where we can create knowledge bases that comment on Notion documents. I was curious if I could hook this together following these requirements:

  1. No custom code hosting
  2. Prompt is editable within Notion rather than requiring understanding of Zapier
  3. Should be be fairly quickly

Ultimately, I was able to get it working. So a quick summary of how it works, some comments on why I don’t particularly like this approach, then some more detailed comments on getting it working.

General approach

Create a Notion database of prompts.

Create a specific prompt for providing feedback on RFCs.

Create a Notion database for all RFCs.

Add an automation into this database that calls a Zapier webhook.

The Zapier webhook does a variety of things that culminate in using the RFC prompt to provide feedback on the specific RFC as a top-level comment in the RFC.

Altogether this works fairly well.

The challenges with this approach

The best thing about this approach is that it actually works, and it works fairly well. However, as we dig into the implementation details, you’ll also see that a series of things are unnaturally difficult with Zapier:

  1. Managing rich text in Notion because it requires navigating the blocks datastructure
  2. Allowing looping API constructs such as making it straightforward to leave multiple comments on specific blocks rather than a single top-level comment
  3. Notion only allows up to 2,000 characters per block, but chunking into multiple blocks is moderately unnatural. In a true Python environment, it would be trivial to translate to and from Markdown using something like md2notion

Ultimately, I could only recommend this approach as an initial validation. It’s definitely not the right long-term resting place for this kind of approach.

Zapier implementation

I already covered the Notion side of the integration, so let’s dig into the Zapier pieces a bit. Overall it had eight steps.

I’ve skipped the first step, which was just a default webhook receiver. The second step was retrieving a statically defined Notion page containing the prompt. (In later steps I just use the Notion API directly, which I would do here if I was redoing this, but this worked too. The advantage of the API is that it returns a real JSON object, this doesn’t, probably because I didn’t specify the content-type header or some such.)

This is the configuration page of step 2, where I specify the prompt’s page explicitly.

)

Probably because I didn’t set content-type, I think I was getting post formatted data here, so I just regular expressed the data out. It’s a bit sloppy, but hey it worked, so there’s that.

)

Here is using the Notion API request tool to retrieve the updated RFC (as opposed to the prompt which we already retrieved).

)

The API request returns a JSON object that you can navigate without writing regular expressions, so that’s nice.

)

Then we send both the prompt as system instructions and the RFC as the user message to Open AI.

)

Then pass the response from OpenAI to json.dumps to encode it for being included in an API call. This is mostly solving for newlines being \n rather than literal newlines.

)

Then format the response into an API request to add a comment to the document.

Anyway, this wasn’t beautiful, and I think you could do a much better job by just doing all of this in Python, but it’s a workable proof of concept.