nick.pearson.dev Blog

Verifying an SSL certificate with Ruby

Sun, 27 Mar 2022 13:43:29 GMT

Last week I needed to verify an SSL certificate’s trust with Ruby and wasn’t able to find a complete answer anywhere. After much poking around, I came up with the following code:

def trusted?(fullchain_pem)
  certs = fullchain_pem.split(/(?=-----BEGIN)/).map do |pem|
    OpenSSL::X509::Certificate.new(pem)
  end
  store = OpenSSL::X509::Store.new
  store.set_default_paths
  store.verify(certs.first, certs[1..-1])
end

This code:

takes a certificate with its intermediary certs as a PEM-format string,
splits it into individual PEM strings and creates an OpenSSL::X509::Certificate object for each one, and
verifies the chain using an OpenSSL::X509::Store with the operating system’s default trust store.

I needed this for an automated deployment script, which checks whether the existing certificate is trusted and should be kept. Why would there ever be an untrusted cert? When the app is initially deployed, a self-signed certificate is generated if one does not yet exist, allowing nginx (and anything else requiring a cert) to start. Alternatively, during initial setup and testing, a “staging” cert might be obtained, e.g. from Let’s Encrypt’s staging environment.

Once we’re ready for the new production environment to go live, we need to obtain a real certificate from Let’s Encrypt. The code above is used for determining whether the certificate we already have can be keept or needs to be replaced.

Validating Elastic Beanstalk worker tier cron schedules

Mon, 14 Feb 2022 01:36:20 GMT

If you’ve worked with Elastic Beanstalk worker tiers, you may be familiar with the cron.yaml file, which sqsd uses to send jobs to your app. I recently deployed a new version of a Rails app to our worker tier environment, and the deploy failed, and hard.

I’ve had bad deploys before, but one nice thing about Elastic Beanstalk is that the previous version of your app continues running if a deployment fails. Not this time though. The deployment failed and took everything down with it, including the previous (and should-be-still-running) version of the app.

This seemed very strange, so I tried re-deploying the previous version, which had been running just a few minutes prior, but that deployment failed too.

Now, I’m writing this based on my memory from a couple months ago, so bear with me here if the details aren’t 100%. I believe I checked systemctl status sqsd.service and was directed to run journalctl -xe. I think I tried starting sqsd manually using the command shown in the journalctl output (/opt/elasticbeanstalk/lib/ruby/bin/aws-sqsd start).

Regardless, at some point I saw this (or a similar) message:

Failed of parsing file 'cron.yaml', because: min out of range (ArgumentError)

I had just added a new job to the cron.yaml file in the latest deployment, but I couldn’t find anything wrong with it. The schedule value seemed fine and was identical to the schedule of another job, but set for a couple minutes later.

After more digging, I found that the problem was coming from inside sqsd, the worker-tier daemon that delivers messages from an SQS queue to the app. I found the sqsd code on the machine¹ and started adding logging output to find exactly where it was failing.

Turns out the failure was occurring when sqsd parses the cron schedules (in schedule_parser.rb) using the parse-cron gem. Here’s the relevant bit showing the part of the code that was failing and raising the error:

begin
  # ...
  schedule = task_def.fetch('schedule')
  parser = CronParser.new(schedule)
  parser.next(Time.now) # test by computing next scheduled time
  parser.last(Time.now) # test by computing last scheduled time
  # ...
rescue => e
  raise ScheduleFileError.new("Failed of parsing file '#{SCHEDULE_FILE_NAME}', because: #{e.message}")
end

After banging my head against the wall trying to figure out how this new job could be failing, I added a line of code to log the schedule variable before it was parsed. That’s when I found it wasn’t the new job with a bad schedule, but one that had been there for months, successfully parsing and deploying all along. What on earth…

Here’s the job that was failing. It should run every 15 minutes from 7:00am until 7:45pm Central².

- name: my-job
  url: /my-job
  schedule: '*/15 12-24 * * *'

If you work with cron schedules much, you’ll probably spot the error right away. I don’t, and didn’t. The original schedule (from years ago) was */15 12-23 * * *, but I decided I wanted it to run for an extra hour each day, so I increased the hours component from 12-23 to 12-24. Without realizing it at the time, I’d made an invalid cron schedule, as Crontab.guru shows.

I hurridly updated the schedule to its correct form, */15 0,12-23 * * *, and deployed the app. It worked.

But I wasn’t satisfied. I had to figure out why the bad schedule had worked all along until now. I started off by trying this in a local IRB session:

parser = CronParser.new('*/15 12-24 * * *')
#=> #<CronParser:... @source="*/15 12-24 * * *", @time_source=Time>
parser.next(Time.now)
#=> (a timestamp)
parser.last(Time.now)
#=> (a timestamp)

No failures… So the CronParser is allowing the invalid 12-24 hour field. More poking around… until it hit me. The (literal) variable here was Time.now.

I had done the (failed) deployment at night but was now trying to reproduce the problem during the day. I tried parser.next(10.hours.from_now), and there it was, the error I’d been searching for: min out of range.

Without further inspecting the parse-cron gem’s code, it appears to be parsing an invalid cron schedule without complaint and then failing when trying to determine the next or previous execution time if the schedule is invalid.

To prevent future failures like this, I now have a test that reads and parses the app’s cron.yaml file and validates its cron schedules using a different gem called crontab_syntax_checker.³ Here’s a simplified version for minitest:

require 'test_helper'
require 'crontab_line'

class CronTest < ActiveSupport::TestCase
  cron_file = Rails.root.join('cron.yaml')
  YAML.load(cron_file.read)['cron'].each do |task|
    test "#{task['name']} has a valid schedule" do
      # this raises an error if the schedule is bad
      CrontabLine.create_by_entry("#{task['schedule']} echo")
    end
  end
end

The echo part is there because this gem expects a command to come after the schedule, so I’ve added a dummy command.

If you wanted to be really sure sqsd won’t choke on a cron schedule, you could write a test that uses the same parse-cron library that sqsd uses and try each schedule in your cron.yaml file for every minute of a two-year period (a leap year and a common year). That’s the approach I started with, but it means calling parser.next (and maybe parser.last) more than a million times for each schedule:

(366 days + 365 days) × 24 hours × 60 minutes = 1,052,640 times

To be sure, this is overkill. But it’s also exhaustive. You could have a test like this and only run it with a special command line flag. I think the above test is enough, but if you’re unhapy with how fast your test suite runs and would like to slow it down, here’s your code:

require 'test_helper'

class CronTest < ActiveSupport::TestCase
  cron_file = Rails.root.join('cron.yaml')
  YAML.load(cron_file.read)['cron'].each do |task|
    test "#{task['name']} has a valid schedule" do
      # guarantee one of the two years is a leap year
      base_time = Time.local(2020, 1, 1).utc.midnight
      0.upto((366 + 365) * 24 * 60) do |add_minutes|
        parser = CronParser.new(task['schedule'])
        Timecop.freeze(base_time + add_minutes.minutes) do
          # calling next and last can cause parser to raise
          # `ArgumentError: min out of range` if the cron
          # schedule syntax contains an out-of-range value
          parser.next
          parser.last
        end
      end
    end
  end
end

At the time of this writing, the code for aws-sqsd lives on worker tiers at /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.6.0/gems/aws-sqsd-3.0.3/ ↩
The instance uses UTC, and I’m on Central Time, which has a -0600 standard offset and -0500 daylight offset. Daylight Saving Time is accounted for in the app so that a “cron” hour of “12” means “7am CST” or “7am CDT” depending on whether DST is in effect. ↩
crontab_syntax_checker defines global methods, so it’s in our Gemfile as:
```
group :test do
  gem 'crontab_syntax_checker', require: false
end
```
Then in the test, I require a specific class (require 'crontab_line') which has everything we need and doesn’t pollute the global namespace. ↩

GitHub Action for Elastic Beanstalk package

Sat, 22 Jan 2022 06:00:00 GMT

When my apps were still on the now-deprecated platform version of Elastic Beanstalk (pre-Amazon Linux 2), my deployment process was simple. I’d create a release on GitHub, download the “Source code (zip)” file GitHub automatically creates, and upload it to the Elastic Beanstalk console.

When I started the migration to the new Amazon Linux 2 platform version, I quickly found that the GitHub zip file no longer worked, because it contains a root directory. Here’s what GitHub creates, and what works on the older version of Elastic Beanstalk:

myapp-1.0.2.zip
↳ myapp/
  ↳ .ebextensions/
    ↳ 01_setup.config

And here’s what the new Elastic Beanstalk requires:

myapp-1.0.2.zip
↳ .ebextensions/
  ↳ 01_setup.config

At first I wrote a simple Bash script that would move the contents of the zip file “up” one directory level. It worked, but I didn’t like the manual step between getting the release from GitHub and uploading to Elastic Beanstalk. I wanted to have confidence that my code hadn’t been tampered with after the release was created, even if I was the one doing the tampering.

I’d seen other projects using GitHub Actions to create release artifacts automatically, and it seemd a good fit, so I made one by piecing together code from the actions of a couple open source projects.

Here’s the package.yml I use now:

# AWS Elastic Beanstalk platform version 3 requires
# files in the deployment zip file to exist at the
# root of the zip file. The zip file GitHub creates
# for a release (named "Source code (zip)") has a
# directory at its root that contains all other
# files and directories in the repository, making
# it unsuitable for uploading to Elastic Beanstalk.
#
# This workflow builds a zip file that conforms to
# the structure that Elastic Beanstalk expects, and
# excludes all git and GitHub files.

name: Package release
on:
  release:
    types: [published]
permissions:
  contents: read
jobs:
  build:
    name: Build release package
    runs-on: ubuntu-latest
    environment: Build release package
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - name: Build zip file
        run: |
          VERSION="$(echo "$GITHUB_REF" | cut -d/ -f3 | cut -dv -f2)"
          REPO_NAME="$(echo $GITHUB_REPOSITORY | cut -d/ -f2)"
          DIR="$(mktemp -d)"
          zip $DIR/release.zip -q -r --symlinks . -x *.git* *.pkg.zip
          mv $DIR/release.zip $REPO_NAME-$VERSION.pkg.zip
      - name: Upload artifact
        uses: actions/upload-artifact@v2
        with:
          name: package-release
          path: "*.pkg.zip"
  upload:
    name: Upload release package
    if: ${{ github.event_name == 'release' }}
    needs: build
    permissions:
      contents: write
    runs-on: ubuntu-latest
    steps:
      - name: Download artifact
        uses: actions/download-artifact@v2
        with:
          name: package-release
      - name: Upload release artifact
        uses: actions/github-script@v5
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          script: |
            const fs = require("fs").promises;
            const { repo: { owner, repo }, sha } = context;
            const tag = process.env.GITHUB_REF.replace("refs/tags/", "");
            const release = await github.rest.repos.getReleaseByTag({
              owner, repo, tag,
            });
            core.info(`Release: ${owner}/${repo}@${tag}`);
            for (let file of await fs.readdir(".")) {
              if (!file.endsWith(".pkg.zip")) continue;
              core.info(`Uploading: ${file}`);
              await github.rest.repos.uploadReleaseAsset({
                owner, repo,
                release_id: release.data.id,
                name: file,
                data: await fs.readFile(file),
              });
            }

This action checks out the repository at the release tag, creates a zip file containing the repository files (excluding .git*), and uploads it as an asset on the release.

The file name matches GitHub’s “Source code (zip)” download file name with .pkg added before .zip (e.g., GitHub’s file is downloaded as myapp-1.0.2.zip and this action creates myapp-1.0.2.pkg.zip).

Now I can create a release in GitHub, wait 30 seconds or so, then download the .pkg.zip file and upload it to Elastic Beanstalk to deploy.

ebenv update

Wed, 12 Jan 2022 06:00:00 GMT

In a previous post I wrote about a small script I call ebenv, which makes AWS Elastic Beanstalk environment properties available to shell commands as environment variables. See that post for the rationale behind this script.

Since that time, I came up with a simpler way to obtain the environment properties, and the new script does not depend on jq.

It turns out Elastic Beanstalk on Amazon Linux 2 stores its environment properties in a file at /opt/elasticbeanstalk/deployment/env, readable by root. Each property is defined on its own line, in shell variable syntax, like this:

RACK_ENV=production
MY_PROP=my-val

When I found this file, I realized I could simplify my ebenv script fairly significantly. All it needs to do is read this file and export each line, then run the supplied command.

There’s one caveat though—it needs to handle special shell characters, namely ', ", and $, because the values in the env file can contain these characters, which will cause problems with shell interpolation and expansion.

The first version prepended the environment variables to the command invocation. If our command was rails c, the command sent to the shell was RACK_ENV=production MY_PROP=my-val rails c. The updated version exports each property and then runs the command, which is simpler and easier to reason about.

Here’s the updated script, with a couple of strange line breaks so that it’s easier to read here.

#!/bin/bash

# This script runs the command (and args) passed to it
# as the EB AppUser, after making Elastic Beanstalk
# environment properties available to the command.
#
# Usage:
#
#   ebenv <command> [arguments]
#
# Example:
#
#   ebenv bin/rails c
#
# When using shell variables in a command, be sure to
# escape or properly quote them so they are expanded by
# ebenv rather than by the shell before the arguments
# reach ebenv. For example:
#
#   ebenv echo $RACK_ENV    #=> [blank]
#   ebenv echo "$RACK_ENV"  #=> [blank]
#   ebenv echo '$RACK_ENV'  #=> production
#   ebenv echo \$RACK_ENV   #=> production
#
# In the first two examples, $RACK_ENV is interpreted
# by the shell before calling ebenv. In the second two
# examples, $RACK_ENV is interpolated inside ebenv.

# exit if there's no command to run
if [[ "$#" == "0" ]]; then
  echo 'Usage: ebenv <command> [arguments]'
  exit 1
fi

# get the user to run the command as
EB_APP_USER="$(
  /opt/elasticbeanstalk/bin/get-config \
    platformconfig -k AppUser
)"

# export each key=value after escaping special chars
while read -r line; do
  key="$(cut -d= -f1 <<< "$line")"
  val="$(cut -d= -f2- <<< "$line" \
       | sed -r "/[\\$\"']/s/[\\$\"']/\\\\\\0/g")"
  export "$key=$val"
done < /opt/elasticbeanstalk/deployment/env

# run the command as the EB AppUser
eval "su -s /bin/bash -c \"$*\" \"$EB_APP_USER\""

Feel free to email me if you have suggestions for further improvement.

Fixing a file streaming bug in a Rails app

Mon, 10 Jan 2022 01:58:09 GMT

A Rails app I work on accepts file uploads from public-facing websites. Uploaded files may contain sensitive information. We offer this so people don’t have to email their sensitive documents to our customers.

When a file is uploaded, our app generates a new asymmetric encryption key pair and uses the public key to encrypt the uploaded file. The encrypted file is then stored, and the private key is stored separately.

The app then makes the original file available to our customers to download. When a download request is initiated, the app retrieves the encrypted file and the private key, decrypts the file, streams it to the client, and then deletes the decrypted file, so that no sensitive info is left lying around on our server.

Here’s a simplified version of the code I’m talking about, showing a Rails controller action (download) that streams a file, and a private method the action depends on (with_decrypted_file) that decrypts a file, yields it to a block, and then deletes it.

def download
  form_post = FormPost.for(current_user).find(params[:id])
  with_decrypted_file(form_post) do |file| # Pathname
    send_file_headers!(
      disposition: 'attachment',
      filename: file.basename)
    response.delete_header('Content-Length')
    self.response_body = Enumerator.new do |chunks|
      file.open('r') do |f|
        chunks << f.read(8192) while !f.eof?
      end
    end
  end
end

private def with_decrypted_file(form_post)
  encrypted_file = form_post.fetch_decrypted_file
  decrypted_file = nil
  begin
    decrypted_file = EncryptionHandler.decrypt(
      encrypted_file, form_post.private_key)
    fail if !decrypted_file.exist?
    yield(decrypted_file)
  ensure
    encrypted_file.delete if encrypted_file&.exist?
    decrypted_file.delete if decrypted_file&.exist?
  end
end

Here is the same code as above, simplified even further to make it easier to see the meat of what’s going on.

def download
  with_decrypted_file do |file|
    self.response_body = Enumerator.new do |chunks|
      file.open('r') do |f|
        chunks << f.read(8192) while !f.eof?
      end
    end
  end
end

private def with_decrypted_file
  decrypted_file = fetch_and_decrypt_file
  yield(decrypted_file)
ensure
  decrypted_file.delete
end

I didn’t have much experience about Rails response streaming when I wrote this code, but it seemed fairly straightforward and worked fine. Encrypted files were fetched, decrypted, streamed, and deleted. Everyone was happy.

Until we started seeing errors, that is. The errors were only happening in some cases, and even then they were sometimes not reproducible. We would get a user report that a download was failing, but we’d try it ourselves and it would work. We’d have the user try it again and it would work. 🤷‍♂️

Stranger still, there were two different errors that were occurring, but only one would occur at a time, the choice seemingly random:

Errno::ENOENT: No such file or directory @ rb_sysopen (at file.open)
IOError: closed stream (at f.eof?)

At first this didn’t make any sense. There’s an explicit fail if the decrypted file doesn’t exist (a sanity check to make sure decryption didn’t fail silently), but it wasn’t being tripped. The file definitely existed, and it wasn’t being deleted (cleaned up) until after the yield block had finished.

Right? Wrong. A look under the hood revealed that Rails streaming happens in a separate thread. So we had a race condition, which explained why this issue only happens sometimes and is difficult to reproduce. Turns out it was happening less often on smaller files (maybe under 250K), when reading the file in one thread could finish before the file was deleted in another thread.

I couldn’t find much in the way of how to solve this, such as a flush method on the response stream, but I found my way to the Response#await_sent method, which is exactly what was needed. (There’s also Response#await_commit, but that wasn’t enough for this case.)

Adding response.await_sent as the last line of the with_decrypted_file block fixed the issue by eliminating the race condition. The decrypted file is now guaranteed to finish being streamed before it gets deleted.

Here’s the updated download action method:

def download
  form_post = FormPost.for(current_user).find(params[:id])
  with_decrypted_file(form_post) do |file| # Pathname
    send_file_headers!(
      disposition: 'attachment',
      filename: file.basename)
    response.delete_header('Content-Length')
    self.response_body = Enumerator.new do |chunks|
      file.open('r') do |f|
        chunks << f.read(8192) while !f.eof?
      end
    end
    # wait to exit the block until the file has been
    # streamed so it doesn't get deleted beforehand
    response.await_sent
  end
end

I hope this post saves someone else some time if they are greeted with intermittent IOError and Errno::ENOENT errors while using a response_body Enumerator to stream files in Rails.

cfgpkg, or how to use age encryption with YAML

Thu, 16 Dec 2021 15:00:00 GMT

I have a case where I need to store a YAML file in an S3 bucket where it’s available to be copied to a server as part of an automated deployment process. The YAML file contains a few sensitive values, like API keys, so it’d be best to keep those values encrypted.

I’d prefer not to have to encrypt the entire file, because that makes it more difficult to work with and update. It would be nice if I could safely keep a local copy of the file, update it as needed, and only have to deal with encryption when I’m adding or changing a sensitive value.

Not being aware of anything that could encrypt and decrypt just a subset of values in a YAML file—and not looking too hard for one, since this seemed like a fun problem to solve. Let’s write a script.

age reached its 1.0.0 release recently, and it looks like a good tool for the job. It has a simple command line interface, and there are binaries for multiple platforms. Plus, age uses asymmetric encryption, which will let us keep a copy of the public key locally for encrypting new and updated values without having to worry about having a private key (capable of decrypting) lying around.

First, the requirements:

Encrypt sensitive values locally and leave others alone
Decrypt sensitive values on the server during deployment

That’s it, really. We should be able to open our .yml file in a text editor, add a key/value pair, mark it as sensitive, and then run the script to encrypt the sensitive value.

Both requirements imply the script will need a way to identify sensitive values—it needs to know what to encrypt and what to decrypt. Since config keys aren’t likely to include an exclamation mark, that seems like a good way to mark sensitive keys. Let’s do it like this: API_KEY!. And to let’s surround encrypted values in an age “tag” like this: <age>c2VjcmV0</age>.

It’s probably easiest to visualize what we’re going for, so let’s start with a simple YAML file at each state we want to handle: (1) the original file with a cleartext sensitive value, (2) the encrypted version that will be stored where a deployment script can get it, and (3) the final decrypted file for our application to use.

The original file:

APP_HOST: app.example.com
API_KEY!: secret_key_abc123

The encrypted file:

APP_HOST: app.example.com
API_KEY: <age>c2VjcmV0X2tleV9hYmMxMjMK</age>

The decrypted file:

APP_HOST: app.example.com
API_KEY: secret_key_abc123

The original file and decrypted file are identical except that the decrypted file does not contain a ! character marking the sensitive key.

Because the script needs to run locally and on our server, let’s write it as a Bash script. Ruby would surely make for simpler code, but that would introduce a deployment dependency our app may not already have.

Let’s start writing some code. At first we can ignore validation and usage instructions and just keep it simple. We know we need to be able to tell the program what to do (encrypt or decrypt), what YAML file to operate on, and what encryption key to use, so let’s start with a simple outline.

#!/bin/bash
cmd="$1"
yaml_file="$2"
if [[ "$cmd" == "enc" ]]; then
  public_key="$3"
  # TODO: encrypt sensitive values in $yaml_file
else
  private_key_file="$3"
  # TODO: decrypt encrypted values in $yaml_file
fi

Now’s would be a good time to see briefly how to interact with the age program using stdout and stdin:

# generate a key pair
$ age-keygen -o key.txt
Public key: age1abc123...

# encrypt and base64-encode a string using
# public key from age-keygen output
$ echo "secret" | age -r age1abc123... | base64
xyz789...

# base64-decode and decrypt string using
# private key from age-keygen run
$ echo "xyz789..." | base64 -d | age -d -i key.txt
secret

Next, let’s implement the encryption part. sed seems like a good tool to use for our in-place file editing.¹ We need to identify each sensitive key, remove the ! key suffix, and encrypt the value and wrap it in our <age>...</age> marker. To keep things simple we’ll assume sensitive values are on a single line with their keys. Indentation won’t matter.² And to avoid accidentally leaving a sensitive value on a commented-out line, let’s process those lines too.

So, let’s first find and extract sensitive keys into an array. (Note that I’m writing this on macOS, so I’ll use sed -E, which is like sed -r on Linux. We’ll address this in our script later. I’m also including extra line breaks to make the script more readable without horizontal scrolling.)

# extract keys ending with '!' from the
# YAML file, including commented-out lines
while IFS='' read -r line; do
  keys+=("$line")
done < <( \
  grep -E '^[# ]*\w+! *:' "$yaml_file" | sed -E \
    '/[a-zA-Z0-9_]+! *:/ s/^[# ]*([a-zA-Z0-9_]+)! *:.*/\1/' \
)

Now let’s iterate over our $keys array to remove its ! suffix and encrypt its value.

# encrypt values for extracted keys,
# and remove the '!' from the keys
for key in "${keys[@]}"; do
  # find the key and extract its value
  val="$(grep -E "^[# ]*$key! *:" "$yaml_file" \
       | head -n 1 | sed -E "s/^[# ]*$key! *: *(.*)/\1/")"
  # encrypt the value and encode as Base64
  enc="$(echo -n "$val" \
       | age -r "$public_key" | base64)"
  # replace the "key!: val" line with "key: <age>...</age>"
  [[ -n "$enc" ]] && sed -E -i \
    "/<age>/! s%^([# ]*$key)!( *: *).*%\1\2<age>$enc</age>%" \
    "$yaml_file"
done

Ok, not too bad, as shell scripts go. How about the decryption part? It’ll be similar to encryption—we just need to build our $keys array by looking for encrypted values instead of sensitive keys.

# extract keys with <age> values from the
# YAML file, including commented-out lines
while IFS='' read -r line; do
  keys+=("$line")
done < <( \
  grep -E '^[# ]*\w+ *: *<age>.*</age>' "$yaml_file" | sed -E \
    '/[a-zA-Z0-9_]+:/ s/^[# ]*([a-zA-Z0-9_]+) *:.*/\1/' \
)

Then we’ll Base64-decode and decrypt each value.

# decrypt values for extracted keys
for key in "${keys[@]}"; do
  enc="$(grep -E "^[# ]*$key *:" "$yaml_file" \
       | sed -E 's:.*<age>(.*)</age>.*:\1:')"
  val="$(base64 -d <<< "$enc" | age -d -i "$private_key_file")"
  [[ -n "$val" ]] && sed -E -i \
    "s%^([# ]*$key *: *).*%\1$val%" "$yaml_file"
done

Before putting it all together, let’s make a function for invoking sed using the arguments it expects based on the OS the script is running on. (I originally saw this in dehydrated.)

# use `-r` or `-E` depending on platform
_sed() {
  if [[ "$(uname)" = "Linux" ]]; then
    sed -r "${@}"
  else
    sed -E "${@}"
  fi
}
# use `-i` or `-i ''` depending on platform
_sed_i() {
  if [[ "$(uname)" = "Linux" ]]; then
    sed -r -i "${@}"
  else
    sed -E -i '' "${@}"
  fi
}

Let’s put together what we have so far, using our new _sed and _sed_i functions and some docs and input validation:

#!/bin/bash

# Encrypts and decrypts sensitive values in a YAML file.
#
# Values can be encrypted locally, such as in preparation
# for placing the file where it's available to a deployment
# process, and then decrypted on a server during deployment.
#
# Sub commands:
#
#   enc: identifies sensitive key-value pairs (any key
#        ending with "!"), then encrypts sensitive values
#        and removes trailing "!" from keys
#
#   dec: decrypts encrypted values
#
# Usage:
#
#   cfgpkg enc <config-file> <public-key>
#   cfgpkg dec <config-file> <key-file>
#
# Examples:
#
#   cfgpkg enc app.yml age1abc123...
#   cfgpkg dec app.yml /path/to/age.key


_sed() {
  if [[ "$(uname)" = "Linux" ]]; then
    sed -r "${@}"
  else
    sed -E "${@}"
  fi
}

_sed_i() {
  if [[ "$(uname)" = "Linux" ]]; then
    sed -r -i "${@}"
  else
    sed -E -i '' "${@}"
  fi
}

cmd="$1"
yaml_file="$2"

# exit if command, file, and key were not specified
if [[ $# -ne 3 ]] || [[ "$cmd" != "enc" && "$cmd" != "dec" ]]; then
  >&2 echo 'Usage: cfgpkg <enc|dec> <yaml-file> <pub-key/key-file>'
  exit 1
elif [[ ! -f "$yaml_file" ]]; then
  >&2 echo "$yaml_file does not exist"
  exit 1
fi


if [[ "$cmd" == "enc" ]]; then
  public_key="$3"
  # extract keys ending with '!' from the
  # YAML file, including commented-out lines
  while IFS='' read -r line; do
    keys+=("$line")
  done < <( \
    grep -E '^[# ]*\w+! *:' "$yaml_file" | _sed \
      '/[a-zA-Z0-9_]+! *:/ s/^[# ]*([a-zA-Z0-9_]+)! *:.*/\1/' \
  )
  # encrypt values for extracted keys,
  # and remove the '!' from the keys
  for key in "${keys[@]}"; do
    # find the key and extract its value
    val="$(grep -E "^[# ]*$key! *:" "$yaml_file" \
         | head -n 1 | _sed "s/^[# ]*$key! *: *(.*)/\1/")"
    # encrypt the value and encode as Base64
    enc="$(echo -n "$val" \
         | age -r "$public_key" | base64)"
    # replace the "key!: val" line with "key: <age>...</age>"
    [[ -n "$enc" ]] && _sed_i \
      "/<age>/! s%^([# ]*$key)!( *: *).*%\1\2<age>$enc</age>%" \
      "$yaml_file"
  done
else
  private_key_file="$3"
  # extract keys with <age> values from the
  # YAML file, including commented-out lines
  while IFS='' read -r line; do
    keys+=("$line")
  done < <( \
    grep -E '^[# ]*\w+ *: *<age>.*</age>' "$yaml_file" | _sed \
      '/[a-zA-Z0-9_]+:/ s/^[# ]*([a-zA-Z0-9_]+) *:.*/\1/' \
  )
  # decrypt values for extracted keys
  for key in "${keys[@]}"; do
    enc="$(grep -E "^[# ]*$key *:" "$yaml_file" \
         | _sed 's:.*<age>(.*)</age>.*:\1:')"
    val="$(base64 -d <<< "$enc" | age -d -i "$private_key_file")"
    [[ -n "$val" ]] && _sed_i \
      "s%^([# ]*$key *: *).*%\1$val%" "$yaml_file"
  done
fi

Ok, we have a working, (*nix) platform-independent script. I named it “cfgpkg” (config packager). Let’s give it a try using the sample YAML from the beginning of this post to ensure it works end-to-end. (You’ll need to install age if you haven’t already.)

Create our .yml file with sample data and generate our age key:

$ echo 'APP_HOST: app.example.com' > app.yml
$ echo 'API_KEY!: secret_key_abc123' >> app.yml
$ age-keygen -o age.key
Public key: age1abc123...

Encrypt the sensitive value in our .yml file:

$ cfgpkg enc app.yml age1abc123...
$ cat app.yml
APP_HOST: app.example.com
API_KEY: <age>c2VjcmV0X2tleV9hYmMxMjMK</age>

Decrypt the sensitive value in our .yml file:

$ cfgpkg dec app.yml age.key
$ cat app.yml
APP_HOST: app.example.com
API_KEY: secret_key_abc123

There are two features I’d like to add and one known bug to fix, but we’ll save these for another time:

[feature] the ability to “undo” an encryption run, essentially restoring our .yml file to its original pre-encryption state (same as the decrypted state, but with the ! sensitive key markers), possibly as a rev sub command
[feature] the ability to extract a single encrypted value without modifying the file, possibly as an ext sub command
[bug] duplicate sensitive keys cause a problem, including on commented-out lines, because sed replaces every matching occurrence it finds

I plan to update this post when these three things have been addressed. If you find other bugs or have an idea for a useful feature, feel free to email me.

When initially researching, I discovered sed has an e (execute) command that allows you to do substitutions using the output of a shell command that runs for each match. This did exactly what I needed, but I’ll save you the time you might have otherwise spent before finding this line in the docs: “This is a GNU sed extension.” This means it won’t work on the version of sed that ships with macOS, so I opted against using it here. ↩
If your YAML file contains a multi-line value, and one of the lines in that value starts with what looks like a sensitve key followed by a colon, this script will incorrectly process that line. To keep things simple, our script will process the file as text using regular expressions. In other words, it won’t use a YAML parser, because that would introduce a dependency. ↩

ebenv, or how to run commands with environment variables on Elastic Beanstalk

Thu, 14 Oct 2021 05:00:00 GMT

Update [Jan 12, 2022]: I came up with a simpler script and wrote about it.

I recently deployed a Rails app onto the latest AWS Elastic Beanstalk platform version running Ruby. I had done this a couple years ago, but a newer platform version (running Amazon Linux 2) is out now, with some significant changes.

One such change: the Elastic Beanstalk application’s environment properties are no longer automatically available in a login shell as environment variables. In the previous platform version, I could start a Rails console like this:

# switch to root, then start a Rails
# console session as the webapp user
sudo su -
cd /var/app/current
su -s /bin/bash -c 'bin/rails c' webapp

To my surprise, when I tried doing this for the first time on an instance launched via the newer Elastic Beanstalk platform version, I was greeted with this error message after trying to query the database:

Mysql2::Error::ConnectionError (Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2))

Odd that it was trying to connect via a local socket rather than my RDS instance… So I tried Rails.env and got back "development". Definitely not correct. It seemed that the environment variables were not being set.

I exited the console and tried echo $MY_VAR to test for a variable defined as an Elastic Beanstalk environment property. Empty. Same for $RACK_ENV, a (Ruby) platform built-in, which I expected to be set to production.

While I could just add RACK_ENV=production MY_VAR=my-val before rails c, this would be a lot to remember and to type each time, and a copy/paste note could easily get out of sync with the list of properties the application needs.

What I needed was a way to start a Rails console (or run other scripts) with my application’s environment variables available. I looked around a bit but didn’t find a way to do this. On a help page, AWS notes:

Environment properties aren’t automatically exported to the shell, even though they are present in the instance. Instead, environment properties are made available to the application through the stack that it runs in, based on which platform you’re using.

Just under that note is a link to Accessing environment properties, which shows how to access individual properties from within an application, but not from a shell. At the bottom, though, there’s a link to the built-in get-config program, which looks promising.

This page shows how to get a single environment property using get-config:

/opt/elasticbeanstalk/bin/get-config environment -k PORT

While not entirely clear from the documentation, all environment properties can be obtained by leaving off the -k option. The results are returned as JSON. Now we just need to convert the JSON to environment variables.

Amazon Linux 2 instances on Elastic Beanstalk (on the Ruby platform, at least) come with jq preinstalled. We can use this to parse the JSON from the get-config program and turn the key/value pairs into export commands.

Assuming get-config returns this JSON:

{"RACK_ENV":"production","MY_VAR":"my-val"}

we want to get these export commands:

export RACK_ENV="production"; export MY_VAR="my-val";

Here’s a command that does just that¹:

/opt/elasticbeanstalk/bin/get-config environment \
  | jq -j 'to_entries | .[] | "export \(.key)=\"\(.value)\"; "'

Rather than having to copy and paste that command each time I ssh to a server, I put it into a script. I named it ebenv because it loads the EB properties into the shell as environment variables.

Note: If you plan on using this code, be sure to first check out the update.

#!/bin/bash

# This script runs the command (and args)
# passed to it as the EB app user, after
# making Elastic Beanstalk environment
# variables available to the command.
#
# Usage:
#
#   ebenv <command> [arguments]
#
# Example:
#
#   ebenv bin/rails c

# exit if there's no command to run
if [[ "$#" == "0" ]]; then
  echo 'Usage: ebenv <command> [arguments]'
  exit 1
fi

# convert env properties to export commands
EXPORTS="$(
  /opt/elasticbeanstalk/bin/get-config environment \
  | jq -j 'to_entries | .[] | "export \(.key)=\"\(.value)\"; "'
)"

# get the user to run the command as
EB_APP_USER="$(
  /opt/elasticbeanstalk/bin/get-config platformconfig -k AppUser
)"

eval "$EXPORTS su -s /bin/bash -c \"$*\" $EB_APP_USER"

After making sure there’s a command to run, this script collects the Beanstalk environment’s properties, converts them to export commands, and then runs the command (preceeded by the exports) as the app user (webapp).

This little script lives happily in my Rails app’s bin directory, and I also have a prebuild platform hook script that copies it to /usr/local/bin/ so that it’s available everywhere.

Note that this is a simple implementation that assumes the property values do not contain quotes, escape characters, or dollar-sign characters. It just slaps a " at the beginning and end of the value. ↩