Triggered CI Builds: Automatically Update your Project's Dependencies

September 10, 2019

I wake up every Tuesday to several Pull Requests on my GitHub repositories. Those PRs are triggered automatically and update my repositories’ dependencies. By the time I get to the office, they’re merged. For this I use CircleCI’s scheduled builds, Nix and niv.

But why though?

I <3 AUTOMATION

That aside, it boils down to two facts:

1. Keeping dependencies up-to-date is important. You get the latest security patches. You avoid having to catch up with breaking changes from three years ago when you urgently have to update some library to get a very needed feature which would halve the total development time of mysundaydress.com, your mom’s new drop shipping platform for Church-wear.
2. I always forget to update dependencies.

Overview

I have a dedicated repository, autoupdate, which has scheduled CircleCI builds. This CI build is responsible for updating my other repositories. For each other repository $repo_name:

clone github.com/nmattia/$repo_name.
update the dependencies. I use Nix — a “programmable package manager” — to specify all my dependencies, and niv to go look for new updates (but you don’t need Nix for this). If there was any update, I …
… fork nmattia/$repo_name, and create a PR, with GitHub’s hub tool. All checks after the update are handled by $repo_name’s CI.

Here’s the plan for the rest of the article.

I’ll first explain how the dependencies get updated.
Then I’ll talk about the CircleCI configuration I use to trigger the weekly updates.
Finally I’ll talk about how I deal with preview deploys for my website. I have to make sure the update didn’t break anything that can’t be programatically checked — no computer system has my innate sense of design.

The Update Script

I have a convention: all my repositories have an update script, ./script/update (a small extension to GitHub’s “Scripts to Rule Them All”). This script’s only job is to update the dependencies, be it:

Code libraries in package.json, Cargo.toml, foo.cabal, etc.
System libraries needed for the build like openssl, libgmp, etc.
System tools needed for the tests and build like curl, redis, etc.

Nix provides the system libraries and tools. On top of that there’s niv, whose job is to automatically update dependencies for Nix projects.

The combination of those two tools means that my update script almost always looks like this:

#!/usr/bin/env nix-shell
#!nix-shell -I nixpkgs=./nix
#!nix-shell -i bash -p bash -p nix -p niv --pure
# vim: filetype=sh

niv update

That takes care of updating system libraries and system tools. For code libraries, YMMV, but should boil down to something like npm update, cargo update, etc.

And now you have a script that updates your system dependencies, tools, and libraries! Let’s now configure the scheduled job to run this script on a weekly basis, on all your repositories.

CircleCI Configuration

Quick recap: all your repos now have a script, ./script/update, performing the update. Now we’ll configure a new repository (mine’s autoupdate), whose job is to checkout your other repositories and run ./script/update. My favorite CI for open-source projects is CircleCI, and I was delighted to discover that they have a “scheduled” build feature.

A snippet speaks a thousand words, let’s start with that:

# "autoupdate" CircleCI configuration
version: 2

jobs:
  update:
    steps:
      - checkout
      - run:
          name: Update repositories
          command: ./script/run

workflows:
  version: 2
  update-workflow:
    triggers:
      - schedule:
          cron: "0 23 * * 1"
          filters:
            branches:
              only:
                - master
    jobs:
      - update

Super simple, right? Well I lied! The real configuration is a little more verbose, but the difference is irrelevant to this article.

Here’s a YAML-to-English translation:

Dear CircleCI,

On every Monday (Day 1), at 11pm (hour 23 and 0 minutes), please run the script ./script/run that you will find in this repository, autoupdate.

Yours Truly

The code to update the other repositories is in ./script/run:

# Clone the repository
git clone "https://github.com/nmattia/$repo_name"
cd $repo_name

# Perform the update
./script/update

# Check if there were any changes
if [[ `git status --porcelain` ]]; then

    # Create a branch for the update
    git checkout -b autoupdate

    # Commit the update
    git commit -am "Update dependencies"

    # Fork the repository
    hub fork

    # Push to our fork
    git push -u nmattia-autoupdate "$branch_name"

    # Create a pull request
    hub pull-request -m "Update dependencies"
else
    echo "No updates for $repo_name"
fi

The reality is not that simple, but that’s the idea. In practice there’s one tricky issue: how to authenticate with GitHub to fork repositories, create pull requests, etc.

I use a machine user, nmattia-autoupdate. I don’t use my main GitHub account because I need to share the GitHub API Token with CircleCI. I do trust them, but if anything were to happen I wouldn’t want my main GitHub account to be compromised.

And that’s it! This configures a weekly build which calls ./script/update on your other repositories and creates update PRs.

Inspecting the Build Result

Ideally a CI build ensures that everything went well, but in some cases you may want to inspect the build result before merging the update. I do this for my website; what if the update broke some CSS and now everything looks crappy? You, my reader, would be terribly disappointed!

In nmattia.com’s build I store the produced HTML as build artifacts, which CircleCI is kind enough to host for me:

version: 2

jobs:
  build:
    machine:
      enabled: true
    steps:
      - run:
          name: Install Nix
          command: ... install Nix ...

      - checkout

      - run:
          name: Build
          command: |
            build_dir=$(nix-build --no-link)
            mkdir -p /tmp/artifacts
            cp -r "$build_dir"/* /tmp/artifacts

      - store_artifacts:
          path: /tmp/artifacts

workflows:
  version: 2
  build:
    jobs:
      - build

Before merging the PR, I have a quick look at the new version, and if everything looks good I merge. The merge then triggers another build which uploads everything to netlify.

Before You Leave

I hear a voice rise from the crowd:

Why don’t you use netlify’s Unique Deploy feature for preview?

Well my dear, then I would have to give my autoupdate machine user my netlify token, or I would have to add the machine user as a contributor to the repo! I barely trust myself with tokens, let alone my machine clones.

Another voice timidly says:

In your autoupdate configuration, you run all updates in a single CircleCI build step. Wouldn’t be nicer to have a one build step per repository?

I completely agree. Go ahead and I’ll steal your configuration!