Creating a Slack App in Python on GCP.

Last week I chatted with someone working on an application to facilitate better 1:1s and skip-level 1:1s. What struck me most was the thought that it might be both faster and a better user experience if this tool was implemented as a Slack application rather than a web application.

This left me with an interesting question: has the Slack ecosystem and toolchain reached a place where it’s quicker and easier to use the Slack stack than the typical web stack?

With that in mind, I decided to spend some time experimenting with the Slack API and get a feel for its development experience and flexibility. To balance infrastructure complexity with preserving a path to production, I decided to build this as a handful of serverless functions running on GCP’s Cloud Functions.

These are my notes.

commit 5359 in lethain/reflect-slack-app contains this post’s code

Slack app in Python series

1. Creating a Slack App in Python on GCP
2. Adding App Home to Slack app in Python
3. Make Slack app respond to reacji
4. Using Cloud Firestore to power a Slack app
5. Distributing your Slack application

What to build?

For the last year or so I’ve been writing weekly updates at work to keep folks informed about my priorities and to make it easier to write what Julia Evans calls a brag document (a list of accomplishments you use to track your progress and write your self-review).

I’d been experimenting with a simple web application to handle this, but nothing about this problem demands a web application. Indeed, it’s not too hard to imagine a simple implementation that boils down to two commands: /reflect and /recall.

/reflect lets folks record work:

/reflect finished writing reviews #reviews #mgmt
/reflect implemented prototype of load balancing tool #programming

/recall retrieves your reflections:

/recall #mgmt
/recall #programing
/recall last 7 days #mgmt

This is a simple interface, but should be enough.

Getting started

Apps need to be developed within a specific workspace, so I started out by creating a new Slack workspace, and then created a new Slack App named “Reflect.”

Slack calls commands like /recall a Slash Command. Each time you submit a Slash Command in your Slack console, Slack processes the command and sends a POST request to an endpoint that you register to that command.

Your endpoint is able to return either text/plain with text to display in the user’s console, or you’re able to send application/json to support more sophisticated rendering, which we’ll dig into more later.

To map a given command to the correct backend code, you’re able to specify different endpoints for each Slash Command. Alternatively, you’re able to inspect incoming requests from Slack for the command key-value pair and dispatch based on that value.

Configuring Cloud Functions

In order to register our Slash Commands, one of the required fields is Request URL, which is the HTTPS endpoint to receive POST requests each time the command is issued. While Glitch is probably the easiest way to do this, I wanted to simulate a slightly more production-like experience using Google Cloud Functions to host the endpoints.

This began with creating the reflect-slack-app repository on Github and cloning it locally:

git clone git@github.com:lethain/reflect-slack-app.git
cd reflect-slack-app

Then I followed these instructions to create a new GCP project named ReflectSlackApp.

Then I enabled the Cloud Functions API within that new project.

Aside: having to individually enable APIs is absolutely the worst part of onboarding developer experience for GCP. I’m certain there are good reasons around this pattern to avoid accidental billing, etc, but it seems like GCP would be much better off opting folks into everything combined with a default low per-feature budget cap. Which would, admittedly, require them to implement hard budget caps, which I believe they don’t have yet, but I think it would be worth it in terms of increased onboarding velocity and reduced user abandonment.

Next it’s time to scaffold reflect-slack-app/, following the rough structure in this helloworld example.

python3 -m venv venv
source venv/bin/activate
mkdir reflect
emacs reflect/main.py

We’ll start with the simplest possible Slash Command handler, which will ignore the incoming message and return text/plain with a static message. This won’t do anything, but it will let us confirm we’ve wired the connections up properly.

def reflect_post(request):
    return "Reflect!"

Before uploading the function I made sure to point my gcloud configuration to the new project that I just created (ok, ok, I admit that I forgot to do this, but you can learn from my confusion):

gcloud projects list
PROJECT_ID              NAME                    PROJECT_NUMBER
...
reflectslackapp         ReflectSlackApp         0000000000000
...
gcloud config set project reflectslackapp

Then we’re ready to deploy the function:

cd reflect-slack-app/reflect
gcloud functions deploy reflect_post --runtime python37 --trigger-http

The first time this ran, it asked me if I would allow unauthenticated invocations (where authentication here is Google ID token authentication), which you must accept if you want it to be reachable by Slack:

Allow unauthenticated invocations of new function [reflect_post]?
(y/N)?
y

After it was done, and it’s curlable:

curl https://your-url-here.cloudfunctions.net/reflect_post
Reflect!

Awesome, so now we have an endpoint to point our new Slash Command towards, and we can go back to setting up the Slack App.

Should api platforms go into FaaS?

Before we go further, it’s interesting to consider whether API platforms should even ask users to go through the steps of setting up a function-as-a-service environment or if they should just host functions for their users directly.

Google Functions, AWS Lambda and so on are fairly easy to setup at this point, but they still create significant onboarding friction for folks, which is why approaches like Twilio Functions have a huge place in easing onboarding.

Long-term, I believe few companies ought to make the infrastructure investments required to support production workloads – AWS and GCP are going to provide better integration with your overall workflow than Twilio or Slack can – but for early onboarding and experimentation the production quality bar is just friction.

I’d love to see a standardized pattern where platforms like Slack or Twilio are a launchpad for hosting simple integrations, and couple that approach with single-button export to AWS/Azure/GCP as a given user’s integration passes some particular bar for sophistication. This would make it quicker for new users to experiment on these platforms, which is mutually beneficial for both the platforms and the users, without inflicting the full production quality friction on the users or the api platforms.

Perhaps even more important is a hidden benefit to the API platforms themselves: hosting the long tail of integrations gives you access and control over those integrations’ implementation. With access to these integrations, you can programmatically refactor them, for example automatically upgrading them off deprecated functionality even if they’re absentee maintainers. This is surprisingly valuable because slow deprecation in the long-tail is one of the core constraints for successful api platforms.

Register /reflect

With our endpoints provisioned, our next step is registering the Slash Command for /reflect, which we’ll do within the Slack app dashboard.

This requires specifying the Command, Request URL, Short Description and Usage Hint. Of those, the two most important are the command, which in this case should be /reflect and the request url which will have been spit out by the gcloud CLI when you registered the function, something along the lines of:

https://your-url-here.cloudfunctions.net/reflect_post

With that registered, we’re half way to a working application.

Add /recall

Now that we have /reflect implemented, we also need to add the /recall command, first adding another endpoint to support it, and then registering it in our Slack app:

Adding the endpoint:

mkdir reflect
emacs reflect/main.py

Then we add the simplest implementation of recall_post:

def recall_post(request):
    return "Recall!"

And to upload the implementation:

cd reflect-slack-app/reflect
gcloud functions deploy recall_post --runtime python37 --trigger-http

Finally, register it for your Slack App using your new request url returned when you created this function.

Install app into workspace

Next you’ll need to go to “Install App” in the left nav bar and click “Install App to Workspace” to add the application to your workspace to use it. This will ask you to approve some permissions.

Now it’s installed, and we can verify by connecting to the linked workspace and trying the commands.

They work! They don’t do much yet, but we’ve verified the connections and we can start on the actual implementation.

Development loop

This also demonstrates the development feedback loop for iterating on these commands: change the code, deploy the updated function, try the command and see the response.

When things didn’t work out, I found debugging to be pretty straightforward relying on Google’s Stackdriver for error messages. I do wish that Slack would track my last couple hundred error messages and show them in a dashboard to facilitate easier debugging, but this worked well enough using server-side visibility.

Google’s tutorials could also expose a faster development loop: there’s no reason why the examples couldn’t have a simple __main__ implementation that exposes your endpoints locally for quick testing. Admittedly it would be quite easy to unittest these very simple functions and run them before deployment.

Formatting the responses

With the scaffolding set up, now we can work on formatting our response into something useful. Slack messages are composed of formatting blocks, which you can experiment with quickly using the Block Kit Builder.

We want our endpoints to return JSON structured along the lines of:

{
  "blocks": [{}, {}, {}],
}

There are quite a few kinds of blocks, but we’ll rely on section and divider for our initial rendering. Dividers are essentially a horizontal rule, e.g. hr, tag, and are quite simple:

{
  "type": "divider"
}

Sections are able to render a subset of Markdown which Slack calls mrkdwn, and look like:

{
  "type": "section",
  "text": {
    "type": "mrkdwn",
    "text": "This is *sort of* Markdown"
  }
}

A lot of the Markdown syntax you love won’t work, the details are documented in the mrkdwn docs.

Putting all of this together, a full response would look like:

{
  "blocks": [
    {
      "type": "section",
      "text": {
        "type": "mrkdwn",
        "text": "This is *sort of* Markdown"
      }
    },
    {
      "type": "divider"
    }
  ]
}

Now we need to update our Flask response format to follow that structure. A quick reminder, full code is on Github, no need to reconstruct it from these excerpts.

We’re not storing the messages sent to us via /reflect yet, so for now we’ll randomly generate fake messages for rendering:

def recall(team_id, user_id, text):
    "Recall for (`team_id`, `user_id`) filtered by parameters in `text`"
    import random
    return ["I did something {}".format(x)
            for x in range(random.randint(3, 10))]

Which will return a list containing a variable number of strings:

[
  "I did something 0"
  "I did something 1"
  "I did something 2"
]

We’ll use that function in our updated recall_post function:

def recall_post(request):
    data = request.form
    team_id, user_id, text = data['team_id'], data['user_id'], data['text']
    items = recall(team_id, user_id, text)
    items_text = "\n".join(["%s. %s" % (i, x)
                  for i, x in enumerate(items, 1)])

    block_args = [
        ('mrkdwn', "Recalling `{}`".format(text)),
        ('divider',),
        ('mrkdwn', items_text),
        ('divider',),
        ('mrkdwn', "_Trying filtering_ `/recall last 7 days #mgmt`."),
        ('mrkdwn', "_Share your response by adding_ `public`"),
    ]

    resp = {
       "text": items_text,
       "blocks": [block(*args) for args in block_args]
    }
    return jsonify(resp)

The block function is a helper that translates tuples of type and optionally text into sections, for example:

>>> block(tuple('divider'))
{
    "type": "divider"
}

And is implemented as:

def block(typ, text=None):
    if typ == "mrkdwn":
        return {
            "type": "section",
            "text": {
                "text": text,
                "type": "mrkdwn"
            }
        }
    else:
        return {
            "type": typ
        }

Now when we use /reflect the response is well formatted:

This would certainly look better with real response data, but we can see the pieces coming together for how the application would work, and we’ve written staggeringly little boilerplate code (text user interfaces ftw).

Public versus ephemeral

You can control whether your responses are sent to the channel or privately to the individual by using the response_type field in your response. The default is to only show responses to the user issuing the command, which is equivalent to setting the response type to ephemeral:

{
  ...
  "response_type": "ephemeral",
  ...
}

Alternatively, we could show everyone in the channel:

{
  ...
  "response_type": "in_channel",
  ...
}

Experimenting with things a bit, I decided to let the user control where it goes by adding the public parameter anywhere in the message text when they /recall, for example:

/recall last 7 days #mgmt public

Would go to the room instead of just private. This was shoddily implemented as a one-line extension to recall_post above:

resp = {
    "text": items_text,
    "response_type": "in_channel" if "public" in text else "ephemeral",
    "blocks": [block(*args) for args in block_args]
}

In a real implementation you would, certainly, want to actually tokenize the contents to avoid the tag #public in the text flipping the public flag, etc.

Discoverability

I think experimenting with ideas like this public flag are important to successful Slack applications, because my experience working with Slack as a medium, is that one of the biggest challenges for Slash Commands is discoverability. Command discoverability has long plagued other text-only mediums like IRC.

The only text-based medium that I’ve used that handled command discovery particularly well are multi-user dungeons, which expose a command along the lines of commands that give a full list of available actions, and reward folks for learning them through enhanced gameplay. (I believe Slack used to support /commands but have removed it, I imagine working around some kind of implementation constraint.)

Consequently, in tools like Stripe’s internal incident communication Slack app, we’ve come to prioritize creating command discoverability in any text returned to the channel, as well as hinting at other commands whenever a user uses a command. An example of both is when you use the public flag in recall, your response informs others of how to use the command:

Try filtering by tag and date `/recall last 7 days #mgmt`
Share with the room by adding `public` anywhere in your response

You could imagine a learning version of this approach which tracks the number of times you’ve used the command and returns fewer and fewer hints over time. Or perhaps starts adding hints over time once you’ve gotten comfortable with the fundamentals.

Verifying requests

As the very last step before calling this prototype complete, I want to implement request verification to ensure that we’re only handling requests from Slack itself. If we don’t add verification, folks can just spoof traffic to your endpoints, which would be pretty unfortunate.

I wrote up details of generating the signature in this post, so I’ll show the verify function without explanation. verify should be the first step in each of your Slash Command handlers.

def verify(request,secret):
    body = request.get_data()
    timestamp = request.headers['X-Slack-Request-Timestamp']
    sig_basestring = 'v0:%s:%s' % (timestamp, body.decode('utf-8'))
    computed_sha = hmac.new(secret,
                            sig_basestring.encode('utf-8'),
                            digestmod=hashlib.sha256).hexdigest()
    my_sig = 'v0=%s' % (computed_sha,)
    slack_sig = request.headers['X-Slack-Signature']
    if my_sig != slack_sig:
        err_str = "my_sig %s does not equal slack_sig %s" % \
                   (my_sig, slack_sig))
        raise Exception(err_str)

def recall_post(request):
    signing_secret = b'your secret from some config tool'
    verify(request, signing_secret)
    return "Signatures match, yay."

The question then is how should we actually store our signing secret so that it can be pulled into our Cloud Function while avoiding committing it into the git repository which I hope to store publicly on Github.

The easiest way to handle this is adding the secret to an environment variable, using the --env-vars-file flag to specify a file including your secret.

We’ll create reflect/env.yaml which is

SLACK_SIGN_SECRET: your-secret-goes-here

Then update recall_post to read the secret from the env.

def recall_post(request):
    signing_secret = os.environ['SLACK_SIGN_SECRET'].encode('utf-8')
    verify(request, signing_secret)

Now when we deploy the function we need to run:

cd reflect
gcloud functions deploy \
recall_post --env-vars-file env.yaml \
--runtime python37 --trigger-http

Note that you only really need the last two parameters the first time you deploy a function, but they’re harmless to specify later, so feel free to keep or omit depending on what you’re doing.

The approach described in this blog post to store your secrets in Google Cloud Key Management Service is much better from a security perspective. If you’re building something good, you should probably do that rather than stuff the secret into the environment.

Ending thoughts

Having gotten this far, what’s most top of mind for me is all the other things I’d like to do next to make this a complete, functional application. The top four being:

1. Adding a App Home which would show you your reflections from the last few weeks and potentially support some degree of in-line sorting and filtering.
2. Use the Events API to let folks tag messages for later recall. Maybe even let you tag someone else’s message for their recall, letting teams show support for each others work.
3. Maybe, ya know, add a database to track stuff instead of randomly generating data.
4. Distributing Reflect into the application directory so folks could use it.

Hopefully I’ll get around to working through and writing up those parts as well in the next few weeks. Either way, super impressed with the experiences that the Slack ecosystem facilitates, and excited to play around with it more.

series continues in Adding App Home to Slack app in Python