Triggered CI Builds: Automatically Update your Project's Dependencies
September 10, 2019
I wake up every Tuesday to several Pull Requests on my GitHub repositories. Those PRs are triggered automatically and update my repositories’ dependencies. By the time I get to the office, they’re merged. For this I use CircleCI’s scheduled builds, Nix and niv.
But why though?
That aside, it boils down to two facts:
- 1. Keeping dependencies up-to-date is important. You get the latest
security patches. You avoid having to catch up with breaking changes from
three years ago when you urgently have to update some library to get a very
needed feature which would halve the total development time of
mysundaydress.com, your mom’s new drop shipping platform for Church-wear.
- 2. I always forget to update dependencies.
I have a dedicated repository, autoupdate, which has scheduled CircleCI
CI build is responsible for updating my other repositories. For each
- update the dependencies. I use Nix — a “programmable package manager” — to specify all my dependencies, and niv to go look for new updates (but you don’t need Nix for this). If there was any update, I …
- … fork
nmattia/$repo_name, and create a PR, with GitHub’s hub tool. All checks after the update are handled by
Here’s the plan for the rest of the article.
- I’ll first explain how the dependencies get updated.
- Then I’ll talk about the CircleCI configuration I use to trigger the weekly updates.
- Finally I’ll talk about how I deal with preview deploys for my website. I have to make sure the update didn’t break anything that can’t be programatically checked — no computer system has my innate sense of design.
The Update Script
I have a convention: all my repositories have an update script,
./script/update (a small extension to GitHub’s “Scripts to Rule Them
All”). This script’s
only job is to update the dependencies, be it:
- Code libraries in
- System libraries needed for the build like
- System tools needed for the tests and build like
Nix provides the system libraries and tools. On top of that there’s niv, whose job is to automatically update dependencies for Nix projects.
The combination of those two tools means that my update script almost always looks like this:
#!/usr/bin/env nix-shell #!nix-shell -I nixpkgs=./nix #!nix-shell -i bash -p bash -p nix -p niv --pure # vim: filetype=sh niv update
That takes care of updating system libraries and system tools. For code
libraries, YMMV, but should boil down to something like
cargo update, etc.
And now you have a script that updates your system dependencies, tools, and libraries! Let’s now configure the scheduled job to run this script on a weekly basis, on all your repositories.
Quick recap: all your repos now have a script,
the update. Now we’ll configure a new repository (mine’s
autoupdate), whose job is to
checkout your other repositories and run
./script/update. My favorite CI
for open-source projects is CircleCI, and I was delighted to discover that they
have a “scheduled” build feature.
A snippet speaks a thousand words, let’s start with that:
# "autoupdate" CircleCI configuration version: 2 jobs: update: steps: - checkout - run: name: Update repositories command: ./script/run workflows: version: 2 update-workflow: triggers: - schedule: cron: "0 23 * * 1" filters: branches: only: - master jobs: - update
Super simple, right? Well I lied! The real configuration is a little more verbose, but the difference is irrelevant to this article.
Here’s a YAML-to-English translation:
On every Monday (Day 1), at 11pm (hour 23 and 0 minutes), please run the script ./script/run that you will find in this repository, autoupdate.
The code to update the other repositories is in
# Clone the repository git clone "https://github.com/nmattia/$repo_name" cd $repo_name # Perform the update ./script/update # Check if there were any changes if [[ `git status --porcelain` ]]; then # Create a branch for the update git checkout -b autoupdate # Commit the update git commit -am "Update dependencies" # Fork the repository hub fork # Push to our fork git push -u nmattia-autoupdate "$branch_name" # Create a pull request hub pull-request -m "Update dependencies" else echo "No updates for $repo_name" fi
The reality is not that simple, but that’s the idea. In practice there’s one tricky issue: how to authenticate with GitHub to fork repositories, create pull requests, etc.
I use a machine user,
nmattia-autoupdate. I don’t use my main GitHub account
because I need to share the GitHub API Token with CircleCI. I do trust them,
but if anything were to happen I wouldn’t want my main GitHub account to be
And that’s it! This configures a weekly build which calls
your other repositories and creates update PRs.
Inspecting the Build Result
Ideally a CI build ensures that everything went well, but in some cases you may want to inspect the build result before merging the update. I do this for my website; what if the update broke some CSS and now everything looks crappy? You, my reader, would be terribly disappointed!
In nmattia.com’s build I store the produced HTML as build artifacts, which CircleCI is kind enough to host for me:
version: 2 jobs: build: machine: enabled: true steps: - run: name: Install Nix command: ... install Nix ... - checkout - run: name: Build command: | build_dir=$(nix-build --no-link) mkdir -p /tmp/artifacts cp -r "$build_dir"/* /tmp/artifacts - store_artifacts: path: /tmp/artifacts workflows: version: 2 build: jobs: - build
Before merging the PR, I have a quick look at the new version, and if everything looks good I merge. The merge then triggers another build which uploads everything to netlify.
Before You Leave
I hear a voice rise from the crowd:
Why don’t you use netlify’s Unique Deploy feature for preview?
Well my dear, then I would have to give my autoupdate machine user my netlify token, or I would have to add the machine user as a contributor to the repo! I barely trust myself with tokens, let alone my machine clones.
Another voice timidly says:
In your autoupdate configuration, you run all updates in a single CircleCI build step. Wouldn’t be nicer to have a one build step per repository?
I completely agree. Go ahead and I’ll steal your configuration!