Using Let's Encrypt Certificates with Temporal Cloud

TLS Certificates: The Necessary Evil

In my experience, organizations generally pick one of two styles for managing internal TLS certificates:

Generate a CA on some random server and use that to generate Certs with lifetimes longer than the administrator expects to work at the org.
Use Let’s Encrypt and setup automatic certificate provisioning.

Working with Temporal Cloud, the first workflow is actually pretty easy - just copy/paste the Certificate Authority’s public key into the namespace settings, and you’re good to go.

What about the second camp? This is what Temporal’s documentation has to say about it:

A certificate cannot be a well-known CA (such as DigiCert or Let’s Encrypt) unless the user also specifies certificate filters.

Ah — the user needs to manage it. After reading this, my only thought was “OK, how do I do that?” Without next steps for well-known Certificate Authorities, you might give up and generate your own cert with a lifetime longer than you plan to work at the org.

Hopefully, if you’re in the same spot I was, this guide should help get Temporal Cloud setup for Let’s Encrypt, or whatever other well-known CA you’re using.

To make an apple pie from scratch, you must first invent the universe

There’s only one mandatory pre-requisite: The full CA certificate’s Chain of Trust. This is all of the certificates starting at the CA, and ending at the intermediary certificate that signed your end-entity certificate. It’s not difficult to find, but in the case of Let’s Encrypt there’s a small process for it:

Go to the Chains of Trust page for Let’s Encrypt
Download all of the certificates by clicking on the links titled “pem”.
Concatenate all of the pem files you downloaded into one single pem file.

If you want to test, you’ll need A valid end-entity Certificate and the temporal CLI.

Read the Friendly Manual

The documentation becomes a lot more digestible with the pre-requisites.

Open Temporal Cloud and log in.
Go to the settings for the namespace.
Click “Namespaces” in the left sidebar.
Click the “snowman” on the right of the table for the namespace you want to edit.
Click “Edit”.
Paste the full certificate chain in the “CA Certificates” box (you’ll probably need to expand it by clicking on the arrow).
Click the “Certificate Filters” expander, and then “Add a Certificate Filter”. Fill out any of the required details in the form that displays.
Only the Common Name is mandatory. It supports wildcards, so you can do something like *.temporal.fakedomain.com.
It’s quite import that this domain is specific - ideally, all of the Common Names for the Temporal certificates should fall under a subdomain. This is the whole point of the Certificate Filter requirement.
Click “Save” in the top right

Well, it works on My Machine!

Jump to TL;DR

Ok, so we’ve set up the namespace for accepting Let’s Encrypt certificates for your domain, how do we confirm it’s working?

The pre-requisite step is to get a certificate to used for testing - The easiest way to do that is with Certbot, which has a tutorial for retreiving certificates from Let’s Encrypt, using DNS Validation. Alternatively, you can copy a certificate that’s already provisioned, as long as it isn’t expired. After you’re done testing, make sure to delete the certificate from your workstation!

You’re going to want to open up a shell - I’m using zsh on MacOS with the GNU coreutils, but this should probably all work in any Bourne-ish shell with either GNU or BSD coreutils (you’d probably know if these commands wouldn’t work. fish users will need to solve their own problems).

For clarity in these commands I’m setting these environment variables. You’ll want to fill these in to whatever values you should be using:

export TEMPORAL_ADDRESS='your-temporal-namespace.xxxxx.tmprl.cloud:7233'
export TEMPORAL_NAMESPACE='your-temporal-namespace.xxxxx'
export TEMPORAL_TLS_CERT='path/to/cert.pem'
export TEMPORAL_TLS_KEY='path/to/key.pem'

Those environment variables are the only special configurations you need. I’m mostly using them to keep the following command shorter, but you can certainly use the corresponding command line flags instead of the environment variables.

From here, you can start a workflow (remember, we’re testing - use the parameters that make you happiest. Just make something up). Here’s the command for that:

$ temporal workflow start \
    --workflow-id "$(< /dev/urandom base64 | tr -C -d '[:alpha:]' | tr '[:upper:]' '[:lower:]' | head -c8)" \
    --type tf2-character \
    --task-queue tf2-characters \
    --input '{"class": "demoman"}'

That sneaky one-liner for the workflow-id just generated an 8 character string of text. If that’s not working for you, you can just put literally any alphanumeric text in there. It doesn’t matter.

Assuming my instructions were clear enough, you should see output like this:

$ temporal workflow start \
    --workflow-id "$(< /dev/urandom base64 | tr -C -d '[:lower:]' | head -c8)" \
    --type tf2-character \
    --task-queue tf2-characters \
    --input '{"class": "demoman"}'

Running execution:
  WorkflowId  tfukauep
  RunId       a1cdc675-1d16-4ecf-9bcc-cd3e977a9c9f
  Type        tf2-character
  Namespace   delago-temporal-tls-blog-post.agbj0
  TaskQueue   tf2-characters

And with that, not only are you done, but you also know how to confirm that an arbitrary certificate is working.

Here’s the full thing as a shell script:

#!/bin/sh

# shell strict mode, never leave home without it.
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -eu

export TEMPORAL_ADDRESS="${TEMPORAL_ADDRESS:-FILL_ME_IN}"
export TEMPORAL_NAMESPACE="${TEMPORAL_NAMESPACE:-FILL_ME_IN}"
export TEMPORAL_TLS_CERT="${TEMPORAL_TLS_CERT:-FILL_ME_IN}"
export TEMPORAL_TLS_KEY="${TEMPORAL_TLS_KEY:-FILL_ME_IN}"

# Generate random bytes in base64
# Remove everything that isn't a lowercase char
# take the first 8 bytes and be done
UNIQ_ID="$(
    base64 < /dev/urandom |
    tr -C -d '[:lower:]' |
    head -c8
)"

temporal workflow start \
    --workflow-id "$UNIQ_ID" \
    --type tf2-character \
    --task-queue tf2-characters \
    --input '{"class": "demoman"}'

So long, and thanks for all the fish Gophers

You did it - You retrieved the full chain for Let’s Encrypt certificates, configured Temporal Cloud to accept certificates signed by Let’s Encrypt, and confirmed beyond a shadow of a doubt that it’s working. Now, go and revise the Jira task, stating that this was a XXL tee shirt or whatever.

As always, Apartment 304 exists within 4 dimensions 👍 . Please ~~call~~ email us.