JDHeyburn

Reorganising Obsidian Journal Notes

Wed, 26 Mar 2025 00:00:00 +0000

I finished off my previous post on journalling in Obsidian with a couple of wish lists:

I do have some improvements I wish to make such as moving notes away from their respective flat directories. i.e.:

Daily notes move from planner/daily to be grouped by year and month: planner/daily/YYYY/MM

Weekly notes move from planner/weekly to planner/weekly/YYYY

Monthly notes move from planner/monthly to planner/monthly/YYYY

I’ve been playing around with AI code generation, specifically the Claude 3.7 Sonnet model, and I wanted to see if it could produce scripts to help with this.

Unfortunately I didn’t capture the exact prompts I used, but it was something along these lines:

There are markdown files in the date format YYYY-MM-DD.md found in planner/daily. Create a Python script that will group them in directories by their respective year and month. The Python script should include user confirmation before moving the files.

I repeated this for each note type, and finished with asking it to consolidate all the scripts into one for simplicity.

Consolidate all the scripts you’ve created into one.

You can find all the scripts on the Github gist link below.

https://gist.github.com/jdheyburn/f64effd0b606a68044d7d28960becc77

I strongly recommend backing up your vault and pausing your sync tool of choice before executing any of the scripts.

After the scripts run, notes are now grouped into tidier directories

Group new files

Now that existing files had been moved, I had to change the settings for the Periodic Notes plugin to template new files at their correct location:

I had to update the format of each periodic note as follows:

Daily
- YYYY/MM/YYYY-MM-DD
- e.g.: 2025/03/2025-03-21
Weekly
- YYYY/YYYY-[W]ww
- e.g.: 2025/2025-W12
Monthly
- YYYY/YYYY-MM-[M]
- e.g.: 2025/2025-03-M
Quarterly
- YYYY/YYYY-[Q]Q
- e.g.: 2025/2025-Q1

These get created under their respective note folder, e.g. planner/monthly.

With all but daily notes, I’m prepending YYYY/ to the filename. Periodic Notes will see this as a directory and template out the format. Daily is prepended by YYYY/MM/ to allow for grouping on months too.

Updated explicit daily note links

From my previous post you would have seen daily notes link to adjacent days through an explicit link under planner/daily.

# Monday 20th November 2023
<< [[planner/daily/2023-11-19|2023-11-19]] || [[planner/daily/2023-11-21|2023-11-21]] >>
[[2023-W47]]  || [[2023-11-M]]

These links broke after the restructure because the notes are no longer directly underneath planner/daily. I can’t remember the reason why I set it up this way; it might’ve been because I expected more daily notes in various locations. In either case I don’t have a need for it, so we can change these to implicit links. There are also implicit links to weekly and monthly notes, but these links will work fine as the files are still in the vault, Obsidian will be able to find them.

I also asked Claude to write me a bash one-liner to rename these.

There are markdown files under planner/daily that have text in the format [[planner/daily/2023-11-19|2023-11-19]] || [[planner/daily/2023-11-21|2023-11-21]]. Write a bash command that converts them to [[2023-11-19]] || [[2023-11-21]].

I thought I’d have to be explicit with the date format, but it seemed to understand the context enough to imply it:

find planner/daily -name "*.md" -type f -exec sed -i '' -E 's/\[\[planner\/daily\/([0-9]{4}-[0-9]{2}-[0-9]{2})\|([0-9]{4}-[0-9]{2}-[0-9]{2})\]\]/[[\1]]/g' {} \;

This took just 2 seconds to run! I also made sure to backup my Obsidian vault before hand, in case it went a bit haywire.

My daily note template also needed updating so that new notes follow this:

# <% thisDayLong %>
<< [[<% previousDayFmt %>]] || [[<% nextDayFmt %>]] >>
[[<% thisWeek %>]]  || [[<% thisMonth %>]]

Finally, I enabled syncing and watched as all the files were updated.

Thanks for reading!

Adding aria2 download manager to my NixOS homelab

Sun, 22 Sep 2024 00:00:00 +0000

In this blog post I talk about how I host aria2 on a NixOS server. If you’re coming here for the first time you can see other posts related to nix.

aria2 introduction

aria2 is command-line download tool that can support a wide variety of protocols - that can be ran as a server to allow remote invocation of downloads. Their home page describes it as:

aria2 is a lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink. aria2 can be manipulated via built-in JSON-RPC and XML-RPC interfaces.

Working in the command line is cool and definitely how I prefer doing things, however sometimes its nice to work with a UI for when you don’t have a terminal ready. Additionally you would need to remember the commands to be able to get aria2 to download the files.

AriaNg is a front-end that enables a UI for aria2. It’s written in HTML and JavaScript so there is no complexity in getting it up and running. You tell a web server to host the files, and then configure AriaNg to connect to your aria2 instance!

Also by fronting it with a service, any of the devices on my Tailscale VPN can access aria2 at a nice URL and download files in the background.

The primary use case this helps me solve is downloading music I’ve bought from Bandcamp. Since I download both the flac and mp3 zips, after a bulk purchase I download a load of files at once. My internet speed at home isn’t great so this can take a while. Having a service where I can dump a list of files to download means I can set the download off, then come back to it another time.

Deploying with NixOS

Now that I’m all-in on NixOS for managing my services at home, let’s get together the config for deploying this set up.

Firstly I’ll need to deploy aria2 as a headless daemon, which is available in nixpkgs.

services.aria2 = {
    enable = true;
    rpcSecretFile = config.age.secrets."aria2-password".path;
};

users.users.jdheyburn.extraGroups = [ "aria2" ];

rpcSecretFile expects a file path containing a string of the password which should be set for aria2. I manage my secrets with agenix, but explaining how it works is beyond the scope of this blog post. If you’re trying it out I suggest just creating a temporary file containing the password to use.

I’m also adding my user to the aria2 group so that I have permissions to modify the files after they’ve been downloaded.

After that I need to deploy the frontend, AriaNg. This is available in nixpkgs too, but only as a package and not as a service. Or rather, the package in nixpkgs only contains the HTML and JavaScript files for the frontend. We would need to roll our own web server to be able to host it.

Now I’ve mentioned Caddy on my blog before where I’m using it to reverse proxy my services at home. But here I can use it as a very simple web server - hosting files from a location!

services.caddy.virtualHosts."aria2.svc.joannet.casa" = {
    # Routing config inspired from below:
    # https://github.com/linuxserver/reverse-proxy-confs/blob/20c5dbdcff92442262ed8907385e477935ea9336/aria2-with-webui.subdomain.conf.sample
    extraConfig = ''
        file_server {
            root ${pkgs.ariang}/share/ariang
        }
        tls {
            dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        reverse_proxy /jsonrpc localhost:${toString config.services.aria2.settings.rpc-listen-port}
    '';
};

A big thanks to LinuxServer.io through which the Caddy config was inspired from.

In that config I’m using the file_server directive to specify that the root of the server should be served from the resulting directory that builds the AriaNg package as defined in nixpkgs. You can see that this line in particular is what copies the compiled AriaNg package to the directory. Once this config is built, the directive looks like this:

file_server {
    root /nix/store/i8ln23gyx6icjmv78lfg7pn9nnibrnzp-ariang-1.3.7/share/ariang
}

After file_server we have the tls directive which instructs Caddy how to authenticate that the domain is owned by me - I’ve introduced this in a previous blog post.

Lastly we have a reverse_proxy that will route any calls made to https://aria2.svc.joannet.casa/jsonrpc to the port 6800, which is the port that the aria2 headless service is listening on for remote download invocations. You’ll see us config AriaNg to use this endpoint later.

Once these configs are defined, you can have NixOS deploy out the changes:

nixos-rebuild switch

Configuring AriaNg

Remember when I said that AriaNg was just a bunch of HTML and JavaScript? Well it doesn’t hold any state on the server-side, that is all managed by local storage in your browser. This includes configurations such as authentication, appearance, and even where AriaNg should look for aria2. So if we were to load up the frontend after the install, it won’t be able to connect.

Looking at dev tools we can see why it’s failing.

By default AriaNg is trying to connect to aria2 at https://aria2.svc.joannet.casa:6800/jsonrpc, and failing. This is expected because we configured the reverse proxy for aria2 at https://aria2.svc.joannet.casa:443/jsonrpc - with port 443 being the port that HTTPS runs on (i.e. our Caddy server).

AriaNg has an API for configuring the aria2 RPC (remote procedure call). We take the endpoint that AriaNg is available at and configure it using path parameters in this format:

/#!/settings/rpc/set/${protocol}/${rpcHost}/${rpcPort}/${rpcInterface}/${secret}

Populating with what we have configured previously, we end up with:

https://aria2.svc.joannet.casa/#!/settings/rpc/set/https/aria2.svc.joannet.casa/443/jsonrpc/REPLACE_WITH_ENCODED_PASSWORD

REPLACE_WITH_ENCODED_PASSWORD is the password for aria2 that we set in the password file located at rpcSecretFile, but base64 encoded. You can use the below command to find out what that would be.

# the -n flag will remove the newline from the echo command
echo -n "PASSWORD" | base64

Once we navigate to that URL in our browser, it authenticates us and creates that all important local storage to cache the settings for next time.

Creating an aria2 NixOS module

NixOS is great for packaging all of this together into a module. In my repo I have an aria2 module which contains this:

{ config, pkgs, lib, ... }:

with lib;

let cfg = config.modules.aria2;

in {

  options.modules.aria2 = { enable = mkEnableOption "enable aria2 with a web client for managing downloads"; };

  config = mkIf cfg.enable {

    services.caddy.virtualHosts."aria2.svc.joannet.casa" = {
      # Routing config inspired from below:
      # https://github.com/linuxserver/reverse-proxy-confs/blob/20c5dbdcff92442262ed8907385e477935ea9336/aria2-with-webui.subdomain.conf.sample
      extraConfig = ''
        tls {
          dns cloudflare {env.CLOUDFLARE_API_TOKEN}
        }
        reverse_proxy /jsonrpc localhost:${toString config.services.aria2.settings.rpc-listen-port}
        file_server {
          root ${pkgs.ariang}/share/ariang
        }
      '';
    };

    # TODO retrieve primary user programatically
    # Needed so that I can modify downloaded files
    users.users.jdheyburn.extraGroups = [ "aria2" ];

    age.secrets."aria2-password".file =
      ../../secrets/aria2-password.age;

    services.aria2 = {
      enable = true;
      rpcSecretFile = config.age.secrets."aria2-password".path;
    };
  };
}

Any modules defined in this directory are automatically loaded into each host. For a host to deploy the module, I simply need to add this line.

modules.aria2.enable = true;

Fin

If you’re in the market for a download manager then definitely check out aria2 + AriaNg to see if they fit the bill. I’m not sure if it’s something I’ll stick with forever, but it’s simplicity is a pull factor for me.

The only downside to it is the manual step to get AriaNg configured with the backend. If you have any tips on how that could be improved then feel free to reach out to me!

I'm going to KubeConEU 2024

Fri, 15 Mar 2024 00:00:00 +0000

I was lucky enough to win a DragonflyDB competition for a free ticket to KubeCon Europe 2024 in Paris!

Feel free to reach out if you fancy meeting up for a chat about everything and anything! I’m more than happy to chat the below, with some good starting points.

Kubernetes
- Sample installation of third-party agents on nodes
Redis
- Backups with Velero
- Migrating data into Kubernetes
Observability
- Prometheus, Thanos, and Grafana
- Datadog
- OpenTelemetry
  - I’m really interested in finding out more about this

And anything that I might’ve blogged about before, including Nix more recently.

I’ll be at the Paris Expo from Tuesday for the co-located events. Hope to see some of you there!

Simplifying Caddy configs in NixOS

Thu, 07 Mar 2024 00:00:00 +0000

tl;dr

My Caddy configs before were bloody awfully complicated
I scrapped that way of doing it and made it simpler in this pull request

A gentle reminder

I’ve written before about how I use Caddy to reverse proxy my internal services. Since then I’ve migrated to NixOS and am using that to automatically generate the Caddy config.

However the approach I used was extremely over-engineered and unnecessarily complicates one of the benefits of Caddy, which is the simplicity of its Caddyfile DSL. It was interesting to see how I could generate configs from Nix attribute sets, which served as a learning exercise and the basis for the Nix cheat sheet.

NixOS allows you to specify Caddy config in a way that allows you to modularise it. Migrating to this structure will provide cleaner code for future iterations.

Previously I had created the module for Caddy in such a way that it would discover what services required a reverse proxy fronting it on said host. This snippet from the previous config would look up from the catalog for any services that were due to be hosted on this machine, then build the Caddy config from there.

host_services = attrValues (filterAttrs (svc_name: svc_def:
  svc_def ? "host" && svc_def.host.hostName == config.networking.hostName
  && svc_def.caddify.enable) catalog.services);

I have seen the light

Instead of doing all the building of the Caddy JSON config by hand, I can declare a block in each service module that requires it which will ensure Caddy will create a reverse proxy for it. This means a module now encapsulates everything that is required for this service to function properly.

Such a module would then look like the below, with some lines removed for brevity. You can see it in full in GitHub.

{ config, pkgs, lib, ... }:

with lib;

let cfg = config.modules.plex;
in {

  options.modules.plex = { enable = mkEnableOption "Deploy plex"; };

  config = mkIf cfg.enable {

    services.caddy.virtualHosts."plex.svc.joannet.casa".extraConfig = ''
      tls {
        dns cloudflare {env.CLOUDFLARE_API_TOKEN}
      }
      reverse_proxy localhost:32400
    '';

    services.plex = {
      enable = true;
      openFirewall = true;
    };
  };
}

This is a lot simpler, and it meant that I was able to remove a lot of complicated Nix code in the pull request to achieve the same thing. The end result is a solution that is much easier to understand and pragmatic.

I’m still using a similar method for when writing the AdGuardHome DNS config, but I changed the functions to more appropriate names. You can revisit the previous blog post where I walk through that.

This refactoring to more complete modules isn’t ground-breaking by any means, but I wanted to share this win I gained :thumbs_up:.

How I use Obsidian to journal

Fri, 12 Jan 2024 00:00:00 +0000

Edits

2025-03-26
- Periodic Notes are now grouped in directories by their year (and if applicable, month)
- See blog post: Reorganising Obsidian Journal Notes
2025-01-26
- Updated moment locale from gb to en-gb
- Sundays were incorrectly being templated as the first day of the week, instead of last

tl;dr

I use Obsidian for journalling
You can view the templates I use for creating my daily, weekly, monthly, and quarterly notes
See Deep dive into note templates on how these templates work

Introduction to Obsidian

I’ve been using Obsidian for over 2 years now as my primary note-taking tool. Previously I was using Notion but then I got frustrated with the clunkiness of it (of which I am sure it has improved since then) and also required Internet connectivity to run - none of the files were local. Before that, I’d been keeping a physical notebook which wasn’t actually of much use for recalling information from time gone by.

Obsidian is just one application used in personal knowledge management (PKM) and is a tool for thought (TfT). It is not open-source, however, it is built by a very small, independent team of developers who are passionate about privacy, and user feedback, and who have promised to never accept VC investment to remain independent.

Since kepano became CEO of Obsidian, they have been highlighting what sets Obsidian apart from other PKM tools. Ensuring that it is local-first means that you can use Obsidian without your data being stored anywhere. The files are in markdown format, which is the de-facto standard for documentation - no proprietary file formats are used here. With these main points, if Obsidian were to cease to exist, you could pick your files up and move them to another app. These make Obsidian work blazingly quick - no loading screens or sluggish navigating; everything loads immediately.

Obsidian’s functionality is propelled by the amazing community which has written several plugins that provide it the flexibility to be used for many use cases. Including databases, Kanban boards, project management… there are over 1,000 to choose from.

The application itself is free, but if you want to synchronise notes between different desktops or even the mobile app, they offer a Sync feature for a cost. You could also achieve file sync using a third party, but given that it is a small independent team working on the app, I’m happy to pay for Sync as a means of supporting them too. There are also options for one time payments to help assist them too (I am a Supporter!).

While I don’t see myself necessarily as a power user (at least, my notes are somewhat all-over-the-shop and unlinked), I wanted to write up how I’ve been using it for journalling over the last 2 years. I hope this then becomes a series of my journey through PKM, and to share additional blog posts which I hope inspire others.

Why journal?

Everyone has their reasons for journalling; some people use it to track thoughts, emotions, etc. For myself though as I started approaching my late twenties I felt that life was passing by very quickly - unsure of what I had accomplished or even done on a particular day, and what the general theme of a week or month was to me. I use journalling as a reference back to see how far I have progressed whether in my career or my personal life.

Obsidian has made this extremely easy for me to do with its Daily Notes feature. These are simply notes that are specific to one day only and can be created from a template so that you don’t need to populate them in the boilerplate.

In this post, I’ll walk through the setup that I use for daily journalling, which then bubbles up to weekly, monthly, and then quarterly notes. You can see some sample screenshots below.

Daily note

Weekly note

Monthly note

Quarterly note

This is achieved with these community plugins:

Calendar
- Provides a calendar widget in the right pane of Obsidian
- Clicking on a day or week takes you to its respective note
- I find it’s an easy way to generate historical daily and weekly notes
Dataview
- Enables querying of properties from notes and displaying them into another
Periodic Notes
- Quickly create daily, weekly, monthly, quarterly, and yearly notes
- Configurable to choose the name format of the notes, which template to use, and where to place the notes
Templater
- Creates files from a template
- Allows you to dynamically generate the content of notes from logic

Setting up the workflow

In this section I’ll run through the templates, plugins, and configuration you need to replicate the same workflow in the folder that you have open in Obsidian (known as a vault).

Template files

I use templates extensively for dynamically generating notes, especially on notes that are created every day.

templates/DailySummary.md
templates/WeeklySummary.md
templates/MonthlySummary.md
templates/QuarterlySummary.md

See the links below for the templates needed for journalling.

I run through the sections of each of these templates later on.

Plugins and configuration

Now that we have the template files, we can configure the plugins. Make sure you install them first if you’re following along as a guide. As a reminder the plugins you’ll need are:

Calendar

For this I enable Show week number to let me click on week numbers to create notes from them in the widget.

I like to enable Confirm before creating a new note so that I don’t accidentally create notes I didn’t mean to when clicking in the widget.

Dataview

After installation, there’s no additional config needed for Dataview.

Periodic Notes

I enable all toggles with the exception of Quarterly Notes and Yearly Notes, since I don’t have these templated just yet.

The config for each note is as follows:

Daily Notes
- Format: YYYY-MM-DD (the default)
  - Creates files in the format 2023-11-20, which is theISO 8601 standard
  - A benefit of this format is that alphabetical order is the same as chronological
  - So as daily notes are created, they are already in order
- Daily Note Template: templates/DailySummary.md
- Note Folder: planner/daily
Weekly Notes
- Format: YYYY-[W]ww
  - Creates files in the format 2023-W46, also ISO 8601 compliant
- Weekly Note Template: templates/WeeklySummary.md
- Note Folder: planner/weekly
Monthly Notes
- Format: YYYY-MM-[M]
  - Creates notes in the format 2023-11-M
- Monthly Note Template: templates/MonthlySummary.md
- Note Folder: planner/monthly
Quarterly Notes
- Format: YYYY-[Q]Q
  - e.g. 2023-Q4
- Quarterly Note Template: templates/QuarterlySummary.md
- Note Folder: planner/quarterly

Templater

Other than setting the Template folder location to templates, there is no other config needed.

Connecting everything together

If everything is set up correctly, you should be able to click on a day or week in the Calendar widget and it should generate a note for you. Another way to create these is via the command palette, which can be access through the hotkey Cmd+P or Ctrl+P - this is how I create monthly notes.

Deep dive into note templates

Daily

The daily note is generated from this template, and all daily notes are stored in planner/daily.

Here’s a walkthrough on what’s happening in each section.

Properties

Properties define the metadata of the note, which can be used to query your notes to retrieve data back using Dataview. You define it in your notes via YAML front matter, which is YAML values wrapped around lines of ---.

--- # Front matter starts here
key: value
--- # Front matter ends here

Normal markdown text from here

Templater is used to populate the property values in the front matter, so let’s look at the local variables used in the daily note template.

<%*
moment.locale("en-gb")
const thisDay = window.moment(new Date(tp.file.title));
const previousDay = thisDay.clone().subtract(1, 'days');
const previousDayFmt = previousDay.format("YYYY-MM-DD")
const lastYearDay = thisDay.clone().subtract(1, 'years');
const nextDay = thisDay.clone().add(1, 'days');
const nextDayFmt = nextDay.format("YYYY-MM-DD")
const thisWeek = thisDay.format("YYYY-[W]ww")
const thisMonth = thisDay.format("YYYY-MM-[M]")
const thisDayLong = thisDay.format("dddd Do MMMM YYYY")
const thisYear = thisDay.format("YYYY-[Y]")
-%>

Reviewing each one of these:

thisDay
- Creates a moment.js object for the date that this daily note represents
- Remember that daily notes are created in the format YYYY-MM-DD (2023-11-20), which is already the format for the Date constructor
- By using the file name as the source of truth, it allows me to create daily notes from the Calendar widget just by clicking on the dates
previousDay and previousDayFmt
- Calculate the day previous to this one, and then format it in to YYYY-MM-DD
- Used for navigation
nextDay and nextDayFmt
- Similar to above, but for the succeeding day
lastYearDay
- Calculate the same day last year
- Used to query the journal entry from last year
thisWeek, thisMonth, thisYear
- The week/month/year that this day is in
thisDayLong
- A long format of this date (i.e. Monday 20th November 2023)

You’ll notice that I have to clone thisDay, because add() and subtract() are mutating.

These variables then get placed into the YAML front matter for the note. You’ll notice that there is date and createdAt which is a standard across any note in my vault and represent the date and time the note was created.

Additionally there is a dateShort property which is the date of the note, but formatted in a nicer manner for when the note is recalled.

---
date: <% tp.date.now("YYYY-MM-DD") %>
createdAt: <% tp.file.creation_date() %>
weekly: <% thisWeek %>
monthly:
  - <% thisMonth %>
yearly: <% thisYear %>
dateShort: <% thisDay.format("dddd Do MMMM") %>
aliases:
  - <% thisDayLong %>
---

After templating it looks like:

---
date: 2023-11-20
createdAt: 2023-11-20 18:33
weekly: 2023-W47
monthly:
  - 2023-11-M
yearly: 2023-Y
dateShort: Monday 20th November
aliases:
  - Monday 20th November 2023
---

Since Obsidian has first-class support for properties, it creates a nice interface.

# <% thisDayLong %>
<< [[planner/daily/<% previousDayFmt %>|<% previousDayFmt %>]] || [[planner/daily/<% nextDayFmt %>|<% nextDayFmt %>]] >>
[[<% thisWeek %>]]  || [[<% thisMonth %>]]

This is the main title for the note I’m working in. There are also links to other notes related to this day, such as the adjacent days (although I primarily use the hotkeys Ctrl+1 and Ctrl+3 to navigate previous and next days respectively), along with a link to the weekly and monthly notes for this day.

# Monday 20th November 2023
<< [[planner/daily/2023-11-19|2023-11-19]] || [[planner/daily/2023-11-21|2023-11-21]] >>
[[2023-W47]]  || [[2023-11-M]]

This visualises as:

See below for the hotkeys I have set up for navigating daily notes. Even if I don’t have hotkeys for them, I can still access the command palette at the hotkey Ctrl+P, or Cmd+P or macOS.

Summary

## Summary 🏁

```dataview
TABLE WITHOUT ID DailySummary as "This time last year: <% lastYearDay.format("dddd Do MMMM YYYY") %>"
FROM "planner/daily"
WHERE file.name = "<% lastYearDay.format("YYYY-MM-DD") %>"
```

DailySummary::

After DailySummary:: is where I write the day’s journal entry. Before that though, I have a Dataview query which retrieves the journal entry for the same day, but the year before. I like retrieving the journal entry for the year before to frame time somewhat.

After templating, it looks like this:

## Summary 🏁

```dataview
TABLE WITHOUT ID DailySummary as "This time last year: Sunday 20th November 2022"
FROM "planner/daily"
WHERE file.name = "2022-11-20"
```

DailySummary::

And here it is when rendered.

A bit about Dataview and querying

Dataview is one of the most popular plugins for Obsidian, and there’s good reason for it. It opens up a whole world of dynamically presenting the files in your vault in a database-like manner. You can query for files themselves, or their metadata.

For example, the Dataview query below will retrieve all notes that are tagged with #kubernetes and #worklog.

```dataview
LIST
FROM #kubernetes AND #worklog 
SORT file.name
```

This then is visualised as:

So how do you add metadata? Dataview allows for a couple of methods ; including YAML front matter (aka Properties), or in-line in the file via a key name appended by two colons - e.g. NewProperty::.

I would definitely recommend to check the docs for Dataview, I’ll be covering it in more detail in this post for my particular use-case.

Weekly

The weekly note is used to summarise what happened during a week. In order for me to recall each day, I use a Dataview query to retrieve the DailySummary for each day in that week.

I use this template for my weekly notes, which produces the below.

Properties

These local variables are define in the template:

<%*
moment.locale("en-gb")
const thisWeek = window.moment(tp.file.title, "YYYY-&W WW");
const previousWeek = thisWeek.clone().subtract(1, 'weeks')
const nextWeek = thisWeek.clone().add(1, 'weeks')
const theseMonths = [...new Set([1,2,3,4,5,6,7].map(d=>moment(`${tp.file.title}-${d}`, "YYYY-W-E").format("YYYY-MM-[M]")))]
const thisYear = thisWeek.format("YYYY-[Y]")
-%>

thisWeek
- Creates a moment.js object for the week that this note is for
- We’re parsing the file title (which is in the format 2023-W47) to create the object
- Similarly to the daily note, this allows me to create weekly notes from the Calendar widget by clicking on the week number
previousWeek and nextWeek
- Calculate the weeks that are adjacent to this one
- Used for navigation
theseMonths
- Calculates the months that this week traverses
- It loops over days of the week (i.e. first day in the week), creates a new moment object, then formats it in the month format
thisYear
- The year that this week is in

These variables are then used in the YAML front matter below. There is some extra logic to print out monthly into an array.

---
date: <% tp.date.now("YYYY-MM-DD") %>
createdAt: <% tp.file.creation_date() %>
monthly: <%* theseMonths.forEach(month => tR += `\n - ${month}`) %>
yearly: <% thisYear %>
---

After templating, it looks like this:

---
date: 2023-11-20
createdAt: 2023-11-20 19:35
monthly: 
 - 2023-11-M
yearly: 2023-Y
---

Visualised as:

<< [[<% previousWeek.format("YYYY-[W]ww") %>]] || [[<% nextWeek.format("YYYY-[W]ww") %>]] >><%* theseMonths.forEach(month => tR += `\n[[${month}]]`) %>

I can navigate to adjacent weekly notes, and the monthly notes for this week from this panel. After templating it looks like:

<< [[2023-W46]] || [[2023-W48]] >>
[[2023-11-M]]

Then is visualised as:

Summary

```dataview
TABLE WITHOUT ID ("[["+file.name+"|"+dateShort+"]]") as "Daily Summaries", DailySummary as " "
FROM "planner/daily"
WHERE weekly = this.file.name
SORT file.name ASC
```

## Summary

WeeklySummary::

WeeklySummary:: is where the summary of that week is entered. Above that is a Dataview query for retrieving all the daily entries for the days within that week. This renders the view below:

Let’s now break down what the query is doing, skipping over the first line for now.

FROM "planner/daily"
- limits searching for files in this directory, which is where my daily notes get placed
WHERE weekly = this.file.name
- filters the results to only match files where the weekly property is equal to the name of the file where the query is executed from
- remember that this is executing from my weekly note, which is named 2023-W47
- this will then only return daily notes for the given week
SORT file.name ASC
- arrange the files in alphabetical order
- given we named the files in the ISO8601 date format, these are inherently ordered chronologically

If we were to do a basic LIST output, with the query below…

```dataview
LIST
FROM "planner/daily"
WHERE weekly = this.file.name
SORT file.name ASC
```

Then it would print out a list of all the files with the given property.

While this tells us what days were in the week, we can use the TABLE query to display metadata of the queried files in an easy to view format. So reviewing the first line of our original query again:

TABLE WITHOUT ID ("[["+file.name+"|"+dateShort+"]]") as "Daily Summaries", DailySummary as " "

TABLE WITHOUT ID
- defines this as a TABLE query
- WITHOUT ID will hide the name of the file, i.e. 2023-11-13
  - We want to hide it because we want to use the human friendly format instead

Now we define the name of the columns for the table.

("[["+file.name+"|"+dateShort+"]]") as "Daily Summaries" will create a column named Daily Summaries, with the text being a hyperlink to the file of that daily note, with the display text being the human friendly text.

DailySummary as " " creates a column with an empty name, using the file property DailySummary as the values. Given that we’re setting an in-line property in the daily notes at DailySummary::, the Dataview query is able to display back the values.

Once we’ve written the weekly summary, that bubbles up to the monthly note.

Monthly

The monthly note consolidates all the weekly notes into here so that I can write a summary, and is created from this template.

<%*  
moment.locale("en-gb")  
const thisMonth = window.moment(tp.file.title, "YYYY-MM-&M ");
const thisQuarter = thisMonth.format("YYYY-[Q]Q")
const previousMonth = thisMonth.clone().subtract(1, 'months')  
const nextMonth = thisMonth.clone().add(1, 'months')  
const thisYear = thisMonth.format("YYYY-[Y]")  
-%>

thisMonth
- Creates a moment object for the month that this note is for
- Similar to daily and weekly, we are parsing the title of the note to create the object
thisQuarter
- The quarter that this month is in
previousMonth and nextMonth
- Adjacent months to this one
- Used for navigation
thisYear
- The year that this month is in

These variables get used in the template for the YAML front matter, and navigation.

---
date: <% tp.date.now("YYYY-MM-DD") %>
createdAt: <% tp.file.creation_date() %>
quarterly: <% thisQuarter %>
yearly: <% thisYear %>
---

<< [[<% previousMonth.format("YYYY-MM-[M]") %>]] || [[<% nextMonth.format("YYYY-MM-[M]") %>]] >>
[[<% thisQuarter %>]]
[[<% thisYear %>]]

After templating, it looks like this:

---
date: 2023-11-20
createdAt: 2023-11-20 19:36
quarterly: 2023-Q4
yearly: 2023-Y
---

<< [[2023-10-M]] || [[2023-12-M]] >>
[[2023-Q4]]
[[2023-Y]]

And this is how both of them are rendered:

Summary

## Summaries 
```dataview
Table without id ("[["+file.name+"]]") as "Weekly summaries", WeeklySummary as " "
FROM "planner/weekly"
WHERE contains(monthly, this.file.name)
SORT file.name ASC
```

This Dataview query retrieves all the weekly summaries for the weeks in this month. It’s similar to the query used in weekly notes to retrieve daily summaries, with the exception that the WHERE clause is modified to handle monthly, given it’s an array.

Below that I have a heading for where the monthly summary would go.

> [!hint] 
> Anything written under the **Month summary** section is added to the quarterly review.

### Monthly summary

I tend to write a longer prose for the month summary, and so that wouldn’t fit in nicely in a Dataview query. Instead I embed the contents of the heading in the quarterly note, which I’ll talk about in that section. I include a callout which helps remind me of that set up.

When rendered, it looks like this:

Quarterly

I’ve not really built up a habit of writing a summary for quarterly notes, but I do collate the summaries of all months in a quarter into its note. These are located in planner/quarterly in the format yyyy-[Q]n.md, i.e. 2024-Q1.md.

Properties

<%*
moment.locale("en-gb")
const thisQuarter = window.moment(tp.file.title, "YYYY-&Q Q");
const theseMonths = [...new Set([0,1,2].map(m=>thisQuarter.startOf("quarter").clone().add(m, "months")))]
const thisYear = thisQuarter.format("YYYY-[Y]")
-%>

thisQuarter
- Creates a moment object for the quarter that this note is for
- Similar to the other notes, it parses the title for this
theseMonths
- Creates an array of moment objects for all the months in the quarter
thisYear
- The year that this quarter is in

Out of these, only thisYear is used in the properties…

---
date: <% tp.date.now("YYYY-MM-DD") %>
createdAt: <% tp.file.creation_date() %>
yearly: <% thisYear %>
---

…and here it is after templating:

---
date: 2024-01-04
createdAt: 2024-01-04 20:31
yearly: 2024-Y
---

Summary

For bubbling up the monthly summaries into the quarter note, I embed the contents of each month’s ### Monthly summary heading into the note - which is done by linking to a header in a note, and then embedding it: ![[2024-01-M#Monthly summary]]. This is a core feature of Obsidian and doesn’t require the Dataview plugin.

The reason I chose embedded a note heading is because the monthly summary tends to be a longer prose, whereas the smaller units that make up a month typically are a few sentences and so can be easily captured by a Dataview property.

Taking a look at how the template works, it loops over each element in theseMonths and formats the markdown out for each month:

## Summary

> Insert quarterly summary here

## Months

<%* theseMonths.forEach(month => { %>### <% month.format("MMMM") %>
![[<% month.format("YYYY-MM-[M]") %>#Monthly summary]]

<%* }); %>

And this is how it looks after templating:

## Summary

> Insert quarterly summary here

## Months

### January
![[2024-01-M#Monthly summary]]

### February
![[2024-02-M#Monthly summary]]

### March
![[2024-03-M#Monthly summary]]

All of these monthly summaries can be rather long, so the quarterly summary is written at the top.

Conclusion

This post ended up way more in-depth then I anticipated, and so it was written over several weeks, which might’ve caused some inconsistencies in my screenshots, so apologies for that!

You might be asking where the yearly note gets created (i.e. 2024-Y) - and the truth is that these are not templated yet. I create these ad-hoc whenever I start to write my year review. Even for these, I tend to read through the monthly summaries (given their longer prose) to build together how I think the year went.

However the main purpose of this post is to share my flow for journalling, and what tools I use to help achieve that. I do have some improvements I wish to make such as moving notes away from their respective flat directories. i.e.:

Daily notes move from planner/daily to be grouped by year and month: planner/daily/YYYY/MM
Weekly notes move from planner/weekly to planner/weekly/YYYY
Monthly notes move from planner/monthly to planner/monthly/YYYY

UPDATE 2025-03-26

I wrote a follow-up blog post to group periodic notes into their respective year: Reorganising Obsidian Journal Notes

Hopefully I’ve given enough of a walkthrough to allow you to come up with a flow of your own! In the future I’ll look to document additional functionalities I have in my daily notes such as task management.

Automating service configurations with NixOS

Thu, 02 Feb 2023 00:00:00 +0000

In my last post I mentioned that I use a “service catalog” as a source of truth for what is deployed where and how it should be configured. This catalog is read by a number of services to determine the locations of those services so that can be referred back to.

NixOS makes it easy to get services up and running with sensible defaults. But once those services are online we need a means of routing traffic to them from a nice domain (e.g. service_name.svc.joannet.casa), and monitoring against the service to alert when it’s down.

Since NixOS manages the services and their configuration, we can have it create configurations for services that enable connectivity.

For my use case the tasks it performs are:

DNS rewrites to forward DNS names to the node hosting the service
- I use AdGuardHome as my DNS server, so it generates the config for
Reverse proxy the service to the port the service runs on the host
- Each host has a Caddy instance deployed to it, which reverse proxies all the services that run on that host
- Caddy needs to be aware of the domain to forward traffic to, at the particular port
Dashy for the home dashboard
- A dashboard which let’s you create bookmarks, and more
Monitoring service endpoints
- We want to be informed when services go down, so we can automate writing the Prometheus blackbox exporter config - scraped by VictoriaMetrics
- Then there is a Grafana alert which is configured to alert when the probe fails

When I add a new service to the catalog and deploy the NixOS configs to the nodes, all the above is taken care of for me.

Catalog definitions

In catalog.nix, there is an attribute called services which is an attribute set (attrset) of service names to its service definition. The service name is what Caddy and AdGuardHome will assume to be the domain name to forward requests to, although with an option to make this configurable.

Also in catalog.nix is another attribute nodes which maps hostnames to node definitions. Service definitions in their current form are dependent on node definitions, so let’s dive into the latter first.

Node definitions

A node definition has these attributes:

ip.private
- private IP address
ip.tailscale
- Tailscale IP address
- Not currently used yet, but will look to add functionality when I want to set up connectivity to the services over Tailscale
system
- What architecture the host is on
- Used to determine what flavour of nixpkgs should be used
isNixOS
- boolean representing if this is running NixOS, as I have some legacy hosts which are not yet migrated over
nixosHardware
- optional; any nixos-hardware modules to include into the configuration too

Let’s see an example node definition for the dee host.

nodesBase = {
    dee = {
      ip.private = "192.168.1.10";
      ip.tailscale = "100.127.189.33";
      system = "aarch64-linux";
      isNixOS = true;
      nixosHardware = nixos-hardware.nixosModules.raspberry-pi-4;
    };
};

You’ll notice node definitions are defined under nodesBase , this is because I want nodes to have the hostname enriched in each node definition at hostName. We’re already defining the hostname as the attribute name, we can follow the Don’t Repeat Yourself (DRY) principle by doing some Nixlang work:

  nodes = builtins.listToAttrs (map (hostName: {
    name = hostName;
    value = (nodesBase."${hostName}" // { hostName = hostName; });
  }) (builtins.attrNames nodesBase));

From my Nixlang cheatsheet, we can [[#Add an attribute to an attrset]] to enrich each base node definition with the hostName (the attribute name of nodesBase), and then [[#Converting a list to attributes|convert the returned list to an attrset]]. After this a subset of nodes would be equal to:

nodes = {
    dee = {
      hostName = "dee";
      ip.private = "192.168.1.10";
      ip.tailscale = "100.127.189.33";
      system = "aarch64-linux";
      isNixOS = true;
      nixosHardware = nixos-hardware.nixosModules.raspberry-pi-4;
    };

    # ... remaining nodes here
};

Service definitions

The service definition accepts these attributes:

host
- The node that runs this service
port
- The port this service runs at
blackbox.name
- The display name to use for this service for blackbox
- For when the domain name doesn’t map to the service name
  - i.e. the Dashy service is reachable at home.svc.joannet.casa, but we want blackbox to announce the service as dashy
blackbox.path
- Blackbox by default will take the root domain name as the health check endpoint. If that should not the case then you can add a suffix path here
- e.g. /health
caddify.enable
- Whether Caddy which runs on the same host should reverse proxy to this service at the port defined
caddify.skip_tls_verify
- Whether Caddy should ignore TLS verification when forwarding traffic to this service
- Usually for when the backend service defaults to HTTPS, and I cba to set up a certificate trust
caddify.forwardTo
- Define a node here different to the host to have that node set up reverse proxy instead
- Currently I’m using this to reverse proxy for services where nodes do not have Caddy on them (i.e. non-NixOS nodes)
caddify.paths
- A list of paths, additional path forwarding to ports that
- Used this for testing path forwarding for minio console, but reverted as it didn’t play nice - I’m not using this for anything right now
- path
  - The URL path to forward (e.g. /ui/*)
- port
  - The port to forward to
dashy.section
- What section in Dashy it should fall under
dashy.description
- The description to use in Dashy
dashy.icon
- The icon to display in Dashy

An example service definition would look like this.

servicesBase = {
    adguard = {
      host = nodes.dee;
      port = 3000;
      caddify.enable = true;
      dashy.section = "networks";
      dashy.description = "DNS resolver";
      dashy.icon = "hl-adguardhome";
    };
};

So this will set for us the following:

DNS entry in AdGuardHome for adguard.svc.joannet.casa, which creates an A record to the private IP address of host dee
Configures Caddy on dee to reverse proxy traffic received on adguard.svc.joannet.casa and forward it to 127.0.0.1:3000
- Because I’m using Caddy, I get HTTPS for free
Creates an entry in Dashy, in the section networks with the provided description and gives it a pretty icon

Additionally, the AdGuardHome module will use the port configured at the port to use when starting the service, meaning that I don’t duplicate the port (DRY principle again).

services.adguardhome = {
  enable = true;
  settings = {
    bind_port = catalog.services.adguard.port;
    # ...
  };
};

Similarly to nodes, I add the service name into the definition by duplicating servicesBase to services:

services = builtins.listToAttrs (map (serviceName: {
  name = serviceName;
  value = (servicesBase."${serviceName}" // { name = serviceName; });
}) (builtins.attrNames servicesBase));

Reading catalog definitions

To generate the configs for each service, we have to parse the contents of catalog.nix using Nixlang (have you seen the cheat sheet?).

DNS rewrites

tl;dr: See here for the configuration in GitHub.

AdGuard can respond to DNS requests by answering an IP address for it. The configuration for this is structured like so:

dns:
  rewrites:
    - domain: service.DOMAIN
      answer: IP_ADDRESS

We can use Nix to build this configuration for us.

In my module for DNS, we start by retrieving all services that should be behind Caddy using filterAttrs, and then converting to a list by passing that output to attrValues.

  # Get services which are being served by caddy
  caddy_services = attrValues (filterAttrs (svc_name: svc_def:
    svc_def ? "caddify" && svc_def.caddify ? "enable" && svc_def.caddify.enable)
    catalog.services);

filterAttrs takes two arguments, a lambda to filter the attributes by, and the attrset to filter.

Taking what we learnt about lambdas, we know that the filtering lambda takes two parameters (svc_name and svc_def), and evaluates the body of svc_def ? "caddify" && svc_def.caddify ? "enable" && svc_def.caddify.enable. Since the caddify field in the service definition is optional, we can use shorthand to determine if the field is present in the attribute (svc_def ? "caddify" and svc_def.caddify ? "enable").

When learning new languages I like to make comparisons to other languages so that we can see what’s happening - the equivalent for above would look like this in Python.

caddy_services = {}
for svc_name, svc_def in catalog_services.items():
  if "caddify" in svc_def \
      and "enable" in svc_def["caddify"] \
      and svc_def["caddify"]["enable"]:
    caddy_services[svc_name] = svc_def

# For any Python comprehension fans
caddy_services = {
  svc_name: svc_def
  for svc_name, svc_def in catalog_services.items()
  if "caddify" in svc_def
    and "enable" in svc_def["caddify"]
    and svc_def["caddify"]["enable"]
}

filterAttrs returns us an attrset, whereas we’d like a list so that we can map over each element. We can call attrValues to retrieve the list of values that make up this attrset. It the same as if we were to call caddy_services.values() in Python.

We now need to define a function which will help us to determine the IP address that the domain should be forwarded to. The service definition contains a host attribute which is set to a node definition, however there are some services at home which are not running NixOS and so won’t have Caddy.

For these services they should be routed to a host which does have Caddy, and this is what the caddify.forwardTo is for. We can write a lambda that will return the correct node definition if this attribute exists in the service definition:

getCaddyDestination = service:
  if service.caddify ? "forwardTo" then
    service.caddify.forwardTo
  else
    service.host;

This function is used when we map over caddy_services to generate the attrset containing domain and answer - the format for AdGuard to perform DNS rewrites. As getCaddyDestination returns us a node definition, we can traverse it to retrieve the private IP address.

rewrites = map (service: {
  domain = "${service.name}.svc.joannet.casa";
  answer = (getCaddyDestination service).ip.private;
}) caddy_services;

The last thing we need to do is refer to rewrites when we write the config for AdGuard.

Caddy config

tl;dr: See here for the configuration in GitHub.

Once DNS is configured, any DNS requests for service.svc.joannet.casa will be answered with the IP address of the reverse proxy (Caddy) that can serve that request, so let’s configure Caddy on how to handle that request.

My setup is probably more complex than what’s needed, since Caddy requires config for services running on the same host, as well as any additional services it should forward to other hosts. For the sake of brevity I’ll focus on the former, since the latter is done in a similar manner (look for forward_services and forward_routes).

I use the JSON structure to configure Caddy, since I’m generating these programatically. The config that we want generated for the services is in this format:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "routes": [
            {
              "match": [
                {
                  "host": ["service.svc.joannet.casa"]
                }
              ],
              "terminal": true,
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:PORT"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ]
            }
            // ... other services here
          ]
        }
      }
    }
  },
  "tls": {
    "automation": {
      "policies": [
        {
          "subjects": [
            "service.svc.joannet.casa"
            // ... other services here
          ],
          "issuers": [
            // ... removed for brevity
          ]
        }
      ]
    }
  }
}

That’s a lot of nesting - no wonder why they encourage using a Caddyfile instead - infact here is the Caddyfile I used before I migrated to this.

I could template the config for that instead, however I wanted to play around with learning how to manipulate attrsets, since this is how the majority of options are configured in Nix.

For each service, we need to create a route:

{
  "match": [{
    "host": [ "service.svc.joannet.casa" ]
  }],
  "terminal": true,
  "handle": [{
    "handler": "subroute",
    "routes": [{
      "handle": [
        // ... insert route handler
      ]
    }]
  }]
},

Route functions

Translating this to a function yields this result.

route = { name, port, upstream ? "localhost", skip_tls_verify ? false
  , paths ? [ ] }:
  let
    subroutes = map (service: routeHandler service) (paths ++ [{
      port = port;
      upstream = upstream;
      skip_tls_verify = skip_tls_verify;
    }]);

  in {
    match = [{ host = [ "${name}.svc.joannet.casa" ]; }];
    terminal = true;
    handle = [{
      handler = "subroute";
      routes = subroutes;
    }];
  };

route becomes a function which takes in an attrset with these parameters:

port
- the service port to reverse proxy the request to
upstream
- IP address to forward the request to, defaults to localhost
skip_tls_verify
- whether to ignore TLS certificate verification, defaults to false
path
- whether any query paths should be used instead of the domain
  - i.e. forward to /path instead of /

After then within the let block we can create some local variables. In subroutes we’re mapping over the parameters and creating a routeHandler from them (shown below). Since any defined path forwarding must take precedence, it is prepended to the base path handler.

Then in the return block we return a route entry for this service. Within handle[0].routes we need a “route handler”, this can take different formats depending on whether you want to disable verification of TLS connections to the upstream service:

{
  "handler": "reverse_proxy",
  "upstreams": [
    {
      "dial": "localhost:PORT"
    }
  ],
  // optional below, to disable TLS verification
  "transport": {
    "protocol": "http",
    "tls": {
      "insecure_skip_verify": true
    }
  }
}

We can write a function that can output this for us:

routeHandler =
  { port, upstream ? "localhost", skip_tls_verify ? false, path ? [ ] }:
  let
    base_handle = {
      handler = "reverse_proxy";
      upstreams = [{ dial = "${upstream}:${toString port}"; }];
    };
    handle = base_handle // optionalAttrs (skip_tls_verify) {
      transport = {
        protocol = "http";
        tls.insecure_skip_verify = true;
      };
    };
  in {
    handle = [ handle ];
  } // optionalAttrs (length path > 0) { match = [{ path = path; }]; };

The routeHandler function this takes the same parameters as the route function. We know that each route handler needs at least a handler and upstreams, so let’s define them in the local variable base_handle.

Then if skip_tls_verify is true, we want to append on the transport block - this is making use of the merge attrset shorthand // to do this.

In the return block, we’re returning a attrset with the newly created route handler. Optionally if any additional paths were defined then add a match block to them too.

Constructing Caddy config

To start, we need to pull the services from catalog that are being hosted on this node, then convert it to a list.

host_services = attrValues (filterAttrs (svc_name: svc_def:
  svc_def ? "host" && svc_def.host.hostName == config.networking.hostName
  && svc_def.caddify.enable) catalog.services);

For each service we want to generate a Caddy route for it, using our route function from earlier.

catalog_routes = map (service:
  route {
    name = service.name;
    port = service.port;
    skip_tls_verify = service.caddify ? "skip_tls_verify"
      && service.caddify.skip_tls_verify;
    paths = optionals (service.caddify ? "paths") service.caddify.paths;
  }) host_services;

Then the last thing we need is the list of the domains that can be used for tls.automation.policies[0].subjects.

subject_names = map (service: "${service.name}.svc.joannet.casa")
    (host_services);

Now we can declare these variables in the Caddy service options:

services.caddy = {
  # ... removed for brevity

  configFile = pkgs.writeText "Caddyfile" (builtins.toJSON {
    logging.logs.default.level = "ERROR";
    apps = {
      http.servers.srv0 = {
        listen = [ ":443" ];
        routes = catalog_routes;
      };
      tls.automation.policies = [{
        subjects = subject_names;
        issuers = [
          {
            module = "acme";
            ca = "https://acme-v02.api.letsencrypt.org/directory";
            challenges.dns.provider = {
              name = "cloudflare";
              api_token = "{env.CLOUDFLARE_API_TOKEN}";
            };
          }
          {
            module = "zerossl";
            ca = "https://acme-v02.api.letsencrypt.org/directory";
            challenges.dns.provider = {
              name = "cloudflare";
              api_token = "{env.CLOUDFLARE_API_TOKEN}";
            };
          }
        ];
      }];
    };
  });
};

I’m using agenix to expose my Cloudflare token in Caddys environment variables.

After this is deployed, we will now be able to access services at easy to remember domain names, forwarded to the host on the network.

Monitoring

tl;dr: See here for the configuration in GitHub.

To monitor the services I’m using the Prometheus blackbox exporter - a big thanks to maxanderson’s internetmonitoring for the inspiration.

Blackbox has a multitude of probes you can use - for our use case we want to use the http probe, which returns the HTTP response code, probe duration, and also returns info on TLS certificates.

Blackbox configuration

Given we can build a list of services, we want to write configuration that blackbox can parse to probe these. We can pass a target list of the format:

# format
url;human_name;routing

# examples
https://service_name.svc.joannet.casa;service_name;internal
https://grafana.svc.joannet.casa;grafana;internal
https://loki.svc.joannet.casa/ready;loki;internal
https://bbc.co.uk;bbc.co.uk;external
https://jdheyburn.co.uk;jdheyburn.co.uk;external

url is the endpoint that the probe should hit, followed by a human_name which allows us to make it easily identifiable when querying/alerting, and lastly a routing which can be one of internal or external - which we can use to filter metrics on later.

We can then use Prometheus’s relabel configs to parse these and map them to labels in the probe. So let’s get building by first building the list of services as we’ve done so previously.

caddified_services = attrValues (filterAttrs
  (svc_name: svc_def: svc_def ? "caddify" && svc_def.caddify.enable)
  catalog.services);

And now map these services into the desired format of url;human_name;routing.

internal_https_targets = let
  getPath = service:
    optionalString (service ? "blackbox" && service.blackbox ? "path")
    service.blackbox.path;
  getHumanName = service:
    if service ? "blackbox" && service.blackbox ? "name" then
      service.blackbox.name
    else
      service.name;
in map (service:
  "https://${service.name}.svc.joannet.casa${getPath service};${
    getHumanName service
  };internal") caddified_services;

let blocks can be used in Nix to define local variables when assigning statements (I’ve used them previously in functions). I’m defining two functions in this block, getPath and getHumanName.

getPath checks to see if there is a health check path to append to the service URL, because the health check endpoint may not necessarily be at the root path (/) as we see for the case of Loki (/ready). So we perform a look up to see if a path is defined in the service definition, else we don’t append one (optionalString will return an empty string ("") if the condition is false).

getHumanName checks to see if there’s a human name we should override with, else the default service name is used. This is useful where the domain name doesn’t necessarily map to the service that underlines it, such as home.svc.joannet.casa is the service dashy - this override prevents the human name being set as home.

Blackbox can be used to monitor any endpoint. It can be useful to have it monitor endpoints external to my local services so that I can ensure my Internet is connected. Let’s create that list and merge it with our internal targets.

external_targets = map (url: "https://${url};${url};external") [
  "bbc.co.uk"
  "github.com"
  "google.com"
  "jdheyburn.co.uk"
];

# concatenate the two together
blackbox.https_targets = external_targets ++ internal_https_targets;

Now we need to define some relabelling so that blackbox knows how to parse it.

blackbox.relabel_configs = [
  {
    source_labels = [ "__address__" ];
    regex = "(.*);(.*);(.*)"; # first is the url, thus unique for instance
    target_label = "instance";
    replacement = "$1";
  }
  {
    source_labels = [ "__address__" ];
    regex = "(.*);(.*);(.*)"; # second is humanname to use in charts
    target_label = "humanname";
    replacement = "$2";
  }
  {
    source_labels = [ "__address__" ];
    regex =
      "(.*);(.*);(.*)"; # third state whether this is testing external or internal network
    target_label = "routing";
    replacement = "$3";
  }
  {
    source_labels = [ "instance" ];
    target_label = "__param_target";
  }
  {
    target_label = "__address__";
    replacement = "127.0.0.1:${toString catalog.services.blackboxExporter.port}";
  }
];

The first three configs are using regex to parse the format of url;human_name;routing to map them to labels. We then take the newly created instance label and map it to __param_target, which is the endpoint that blackbox will probe against. Lastly we define the exporter address that Prometheus should scrape at, which is the local blackbox instance running at the defined port.

We’ll now need to add this config to the Prometheus scrape configs:

[
  {
    job_name = "blackbox-https";
    metrics_path = "/probe";
    params.module = [ "http_2xx" ];
    static_configs = [{ targets = blackbox.https_targets; }];
    relabel_configs = blackbox.relabel_configs;
  }
]

So now that we have the scrape config that our scraper can use (i.e. VictoriaMetrics), we’ll need to boot up the blackbox exporter so that there’s something to scrape against.

services.prometheus.exporters.blackbox = {
  enable = true;
  port = catalog.services.blackboxExporter.port;
  configFile = pkgs.writeText "blackbox.json" (builtins.toJSON {
    modules.http_2xx = {
      prober = "http";
      timeout = "5s";
      http.fail_if_not_ssl = true;
      http.preferred_ip_protocol = "ip4";
      http.valid_status_codes = [ 200 401 403 ];
    };
  });
};

Visualising and alerting

It’s all well and good having VictoriaMetrics scrape and poll, but let’s use Grafana to visualise all this. I think I sourced it from this dashboard, in either case I’d recommend to use it!

You’ll notice that some services are responding with 4XX codes, this is because the probes are not being authenticated - but I’m getting some response from the service which shows that something is working. These 4XX codes don’t cause a probe to fail, which is down to how I configured the blackbox exporter in the previous section: http.valid_status_codes = [ 200 401 403 ];.

I also have an alert set up in Grafana to alert on the metric probe_success. This metric will report 1 when it was successful, else 0. Given that I want to be alerted when a service has gone down for 5 minutes, I can give the alert a metric query of max by(humanname) (probe_success{routing="internal"}), which will produce a unique metric for each humanname (i.e. service). This is assigned to the variable A.

I’m only interested in alerting on internally routed services, external is out of my control.

Expression B is then reducing the metric output to a single value, which will be the maximum value for that period.

Lastly expression C checks if B is less than 1, which is what will be produced if the probe failed.

Next up I’m setting how often the alert should poll, and some additional details on the alert, which can be used in the body to link to the blackbox dashboard for diagnosing.

By default all alerts go to my root notification policy, which is to send me an email. I’ve got SMTP set up on my Grafana instance, but I’ll dive into that another time. In the meantime, here’s a screenshot of an email alert!

Dashy

tl;dr: See here for the configuration in GitHub.

Dashy is a customisable dashboard that can act as a homepage for your web browser to help you navigate to services, bookmarks, and more. I’m using it to keep a visual track on the services that I’m running at home.

The desired end result looks like this:

The config for Dashy looks like the below, so let’s get Nix to build it for us!

appConfig:
  # ... removed for brevity

pageInfo:
  title: Joannet
  navLinks:
    - path: https://dashy.to/docs
      title: Dashy Documentation

sections:
  - name: Media
    icon: fas fa-play-circle
    items:
      - title: plex
        description: Watch TV and movies
        icon: hl-plex
        url: https://plex.svc.joannet.casa
  -  # additional sections here

Building Dashy config

We start off with a baseline of sections we want Dashy to look for, since a section requires an icon (the image to the left of the section name in the above screenshot).

sections = [
  {
    name = "Media";
    icon = "fas fa-play-circle";
  }
  {
    name = "Monitoring";
    icon = "fas fa-heartbeat";
  }
  {
    name = "Networks";
    icon = "fas fa-network-wired";
  }
  {
    name = "Storage";
    icon = "fas fa-database";
  }
  {
    name = "Virtualisation";
    icon = "fas fa-cloud";
  }
];

And in what might come across as draw the rest of the fucking owl, we build the sections list…!

sectionServices = let
  isDashyService = section_name: svc_def:
    svc_def ? "dashy" && svc_def.dashy ? "section" && svc_def.dashy.section
    == section_name;

  createSectionItems = services:
    map (service: {
      title = service.name;
      description = service.dashy.description;
      url = "https://${service.name}.svc.joannet.casa";
      icon = service.dashy.icon;
    }) services;

  sectionItems = sectionName:
    createSectionItems (attrValues (filterAttrs
      (svc_name: svc_def: isDashyService (toLower sectionName) svc_def)
      catalog.services));

in map (section: section // { items = sectionItems section.name; }) sections;

Let’s expand on this. Three functions are being defined in this let block:

isDashyService
- Returns true we should include this service in the current iterated section
- Service definitions opt-in to what Dashy section they belong to
createSectionItems
- For a given list of services, create the item definition for that service as required by Dashy
sectionItems
- For the current iterated section:
  - filter on services to be added to the section
  - convert that to a list
  - create section items from it

Then for each element in sections, enrich it with an items attribute with the output of sectionItems function.

This sectionServices variable is then added to dashyConfig:

dashyConfig = {
  pageInfo = {
    title = "Joannet";
    navLinks = [{
      title = "Dashy Documentation";
      path = "https://dashy.to/docs";
    }];
  };

  appConfig = {
    theme = "nord-frost";
    iconSize = "large";
    layout = "vertical";
    preventWriteToDisk = true;
    preventLocalSave = true;
    disableConfiguration = false;
    hideComponents = {
      hideSettings = true;
      hideFooter = true;
    };
  };

  sections = sectionServices;
};

We’ll then need to convert it to a YAML file. This was a bit tricky to set up, but I sought inspiration from this code block.

format = pkgs.formats.yaml { };

configFile =
  pkgs.runCommand "dashy-configuration.yaml" { preferLocalBuild = true; } ''
    cp ${format.generate "dashy-configuration.yaml" dashyConfig} $out
    sed -i -e "s/'\!\([a-z_]\+\) \(.*\)'/\!\1 \2/;s/^\!\!/\!/;" $out
  '';

Then configFile is exposed to the container where the app is running. You can also see below that the port from catalog is used here to expose the service.

virtualisation.oci-containers.containers.dashy = {
  image = "lissy93/dashy:${version}";
  volumes = [ "${configFile}:/app/public/conf.yml" ];
  ports = [ "${toString catalog.services.home.port}:80" ];
};

Deployment configurations

This next section talks about some more advanced features of Nix, of which introducing them is out of scope for this blog post given its length. I’ll discuss how the catalog is used here and link back a more in-depth blog when it is published.

It’s not just service configurations that use the catalog too - I use deploy-rs to deploy these configurations to NixOS nodes, reading from the node definitions in catalog. Given that various services are interdependent on each other across varying nodes, deploy-rs allows me to deploy all configurations at the same time from one command.

# all hosts
nix run github:serokell/deploy-rs -- -s "."

# per host
nix run github:serokell/deploy-rs -- -s ".#dennis"

deploy-rs requires you to enable Nix Flakes in your config, allowing you to fix all your dependencies to a particular version, with a hash. This ensures that you are always able to reproduce the config no matter what rebuilds you do. It primarily is used for locking dependencies of a particular package, but it can also be used for locking dependencies of NixOS configs.

deploy-rs piggybacks on flakes to define what hosts it should deploy too, requiring a deploy.nodes attrset of hostnames to a definition:

deploy = {
  nodes = {
    dennis = {
      hostname = "192.168.1.12";
      profiles = {
        system = {
          user = "root";
          path = deploy-rs.lib."x86_64-linux".activate.nixos self.nixosConfigurations.dennis;
          sshOpts = [ "-o" "IdentitiesOnly=yes" ];
        };
      };
    };
  };
};

Given that I have node NixOS configurations defined in the hosts directory, I can retrieved the hostnames and use these to poll the hosts defined in catalog.nodes to construct a new attrset that deploy-rs requires.

# Defined earlier in the flake
hosts = builtins.attrNames (builtins.readDir ./hosts);

deploy.nodes = builtins.listToAttrs (map (host:
  let node = catalog.nodes.${host};
  in {
    name = host;
    value = {
      hostname = node.ip.private;
      profiles.system = {
        user = "root";
        path = deploy-rs.lib.${node.system}.activate.nixos
          self.nixosConfigurations.${host};
        sshOpts = [ "-o" "IdentitiesOnly=yes" ];
      };
    };
  }) hosts);

What’s happening here is I’m pulling from the node definitions the IP address used to reach the host, as well as the system architecture for that node, so that we can call the correct deploy-rs library. Lastly I feed it its nixosConfiguration that should be deployed to the node - this is a requirement of using a flake to deploy configs.

While brief, I didn’t want to overload this section with nuances of how flakes are set up. You can see my flake.nix if you are keen to see how it all pieces together.

Conclusion and improvements

I don’t like how I have to define which host the service is running on, I think it would be better to have it so that wherever the modules are enabled, then the catalog discovers that. It’s only a tiny bit of duplication so its not been at the top of my list to improve on.

I’d like to also define as much Grafana configuration as possible using this. I provided a GUI walkthrough of how I set up the alerting, but it would be great to have Nix build this for us instead.

While not largely catalog related, we can extend on home-manager to allow us to be able to use the catalog to deploy Nix configs to non-NixOS nodes. When we do this, we will use the package manager component of Nix (nixpkgs) to manage the packages on a host.

The catalog has been largely beneficial as my source of truth; adding a service in here means that I automatically get an endpoint for it with forwarding, and have it monitored too - plus with easy extensibility to other use cases as they come. It’s also enabled configurations to be read across different hosts, without the use of a service discovery component. It’s one of the benefits of using/experimenting with NixOS of which I’m glad I’ve invested the time in and look forward to playing around with more in future.

Nix Cheat Sheet

Fri, 13 Jan 2023 00:00:00 +0000

The world of Nix has a bit of a steep learning curve. As I write blog posts on the subject, I’ll refer to some of the tips and tricks from the cheat sheet here.

Documentation

A list of Nix documentation I refer to, along with some getting started pages:

Resources

Here are a list of resources I use to search for Nix options, services, and any Nix built-in functions.

Functions (and Lambdas)

In some languages you typically have a formal declaration of a function (or lambda), such as for Python:

def double_num(num_to_double)
  return num_to_double * 2

Nixlang is a functional language, and a typical behaviour of these is to define lambdas and assign them to variables.

double_num = num_to_double: num_to_double * 2

The lambda is defined when we create parameters for that lambda, which are defined by having a variable name prefixed by a colon. From the example above, the parameter for the double_num lambda is num_to_double.

Once all the parameters are defined we move onto the body, in which the final statement is returned, similarly to the implicit return in Ruby. Here we’re just using the maths expression num_to_double * 2 to have the intended outcome.

Check if attribute is in attrset shorthand

You can use the built in function hasAttr to determine if an attribute exists within an attrset.

$ attrSet = { key1 = "key1"; };

$ builtins.hasAttr "key1" attrSet
true

$ builtins.hasAttr "key2" attrSet
false

You can use the ? operator as an alternate way of doing the same.

$ attrSet ? "key1"
true

$ attrSet ? "key2"
false

Merge two attrsets shorthand

You can use the // operator to merge the contents of the attrset on the right into the attrset on the left.

$ newAttrset = { key1 = "key1"; } // { key2 = "key2"; }

$ newAttrset
{ key1 = "key1"; key2 = "key2"; }

If the attribute already exists in the left attrset, then it is overwritten.

$ attrSet = { key1 = "key1"; }

$ newAttrSet = attrSet // { key1 = "newKey1"; }

$ newAttrSet
{ key1 = "newKey1"; }

Add an attribute to an attrset

Given an attrset with this structure:

$ items = {
  keyName = {
    value1 = "value";
    value2 = "value";
  }
}

We can use the map function to loop over each attrName (attribute name (aka key)) and its value in items to create a new attrset with its attrName added to it:

$ newItems = map
    (key: items."${key}" // { name = key; })
    (builtins.attrNames items);

The map function takes two parameters:

a lambda that should be performed against each element
- (key: items."${key}" // { name = key; })
  - This lambda takes a parameter key, and returns the attribute value at items.key, and merges it with the attrset { name = key; } using the [[#Merge two attrsets shorthand]].
a list of elements
- (builtins.attrNames items)
- builtins.attrNames is a function that takes a attrset and returns all the attribute names (keys) for it

So newItems will equal the below:

$ newItems
[
  {
    name = "keyName";
    value1 = "value";
    value2 = "value";
  }
];

The map function is also useful for converting an attrset to a list, since a list is what is returns by default.

$ items_list = map
    (key: items."${key}")
    (builtins.attrNames items);

$ items_list
[
  {
    value1 = "value";
    value2 = "value";
  }
]

Converting a list to an attribute set

When we [[#Add an attribute to an attrset|add an attribute to an attset]], the map function returns a list, when we may want to have an attrset back instead.

We can use the builtins.listToAttrs function to convert it to one. It takes one argument, a list of attrsets that contain two attributes; name and value - the former becomes the attrName and the latter its value.

$ list = [
    {
        name = "keyName";
        value = {
            value1 = "value";
            value2 = "value";
        };
    }
    {
        name = "keyName2";
        value = {
            value3 = "value";
        };
    }
]

$ attrSet = builtins.listToAttrs list

attrSet will then equal the below:

attrSet = {
    keyName = {
        value1 = "value";
        value2 = "value";
    };
    keyName2 = {
        value3 = "value";
    };
}

Converting to the Church of Nix

Mon, 17 Oct 2022 00:00:00 +0000

I’ve written in the past about some services I host at home which were running on a Raspberry Pi 3 with Raspbian. Making changes to the host was done ad-hoc with no configuration management tool like Ansible, so should anything happen to the host and it dies, I won’t have a way to reproduce what I had working before.

Naturally last December, the Pi SD card got corrupted and lost everything that was on there.

Rather than rebuilding things how they were in the same way as before, I wanted to look into a better way to ensure I could reproduce services installed and manage configurations on a host to mitigate this in the future.

NixOS came to mind as it solves these problems and more, I had tried to get it working on the Pi and on my Dell XPS 9360 laptop in the past with unsuccessful results. This time round there is much greater support for NixOS on Pi, and coupled with some more motivation, I decided to make the switch.

In this series I’ll be writing about my setup, how I deploy modules, onboarding flakes, and problems I came across along the way. The series will be more NixOS config management orientated, while being referred back to in other posts where I’ll talk about some of the services I’m running via NixOS. This post kicks off with a quick introduction with some more in-depth posts to come.

Why NixOS

For a quick 411, there are 3 components that fall under the “Nix” umbrella:

Nix / Nixlang
- The functional, declarative language that is used to build, manipulate, packages, files, configurations, within the ecosystem
Nixpkgs
- A package manager that can be installed on a Linux machine, or on macOS via nix-darwin
- Think an alternative to apt, brew, etc.
NixOS
- An operating system that uses Nixpkgs as its package manager, but also can configure OS-level configurations, hardware settings, services, via Nixlang

Nixpkgs are designed to be easily reproducible by defining explicit versions and hashes of software and its dependencies - and the same goes for NixOS which takes this same philisophy and applies it beyond software. Nix builds this software and stores it in the Nix store (appropriately named /nix/store), where a previous deployment of a configuration can be rolled back easily with a single command. This makes it useful for experiments or to rollback when shit hits the fan.

There are plenty of talented people who have written about it before, so I’ll link them here:

So the 3 components together allow for some pretty powerful stuff. Your NixOS configurations are written to a .nix suffixed file by default located at /etc/nixos/configuration.nix, which can contain minimal config:

{ config, pkgs, ... }:

{
  networking.hostName = "my-hostname"; # Define your hostname.

  # Set your time zone.
  time.timeZone = "Europe/London";

  # Define a user account. Don't forget to set a password with ‘passwd’.
  users.users.my-user = {
    isNormalUser = true;
    home = "/home/my-user";
    extraGroups = [ "wheel" "networkmanager" ]; # Enable ‘sudo’ for the user.
    password = "hunter2";
  };

  environment.systemPackages = with pkgs; [
    vim
  ];
}

Whenever we want to deploy the configuration to make it live, we can use the command nixos-rebuild switch, which will read this file and build the packages and configurations requested.

Should we want to have Firefox installed? We can just define it:

environment.systemPackages = with pkgs; [
  vim
  # add in the below
  firefox
];

If we want to enable SSH access to our hosts, it’s easy to do that:

  # Enable the OpenSSH daemon.
  services.openssh = {
    enable = true;
    permitRootLogin = "no";
    passwordAuthentication = true;
  };

Services are usually defined with the syntax services.SERVICE_NAME.enable, however as shown above we can see permitRootLogin and passwordAuthentication attributes are defined. These inputs are read by the service in nixpkgs to then be output as configuration for the service.

One last example, what if the service we just deployed requires a particular port opened? NixOS can do that too!

networking.firewall.allowedTCPPorts = [ 4040 ];

When you compare that against traditional package managers, you would first have to install the software and then configure it, probably by hand. On top of that, you would have to manage the versioning of the configuration using another tool. NixOS does all of that for you - provided that you’re checking in your NixOS configuration files!

Just to cap off - if something broke during the nixos-rebuild switch command and you want to go back to the previous state, then we can do so with nixos-rebuild switch --rollback.

Layout

Before going into the details of the setup I’m running I think it helps to have some context of where I was before the migration.

Legacy

Prior to starting this, I had this setup across my network:

dee

My starting point for playing with self-hosting, it was a Raspberry Pi 3.

Service	Purpose
Caddy	Reverse proxy for all services in the network
NFS server	Network wide storage, backed up to cloud storage
PiHole	For adblocking and local DNS resolution
UniFi Controller	Control plane for my UniFi devices at home

frank

A Ubuntu VM on a Proxmox hypervisor, built so that I could play with Proxmox and docker containers.

Service	Purpose
Heimdall	Dashboard for services
Huginn	Experiment with automation (such as NHS vaccine alerts)
Portainer	UI for docker containers

Today

Host frank is unchanged from above today, while here’s what I have running elsewhere.

dee

Upgraded to a Raspberry Pi 4 (NixOS is RAM hungry!) in an Argon One case.

Service	Purpose
AdGuardHome	For adblocking and local DNS resolution (replaces PiHole)
Caddy	Reverse proxy for services local to the host
Healthchecks	Monitor cron jobs (primarily the backup jobs for now)
Minio	S3 compatible storage
NFS server	Network wide storage, backed up to cloud storage
Plex	Music and video player
UniFi Controller	Control plane for UniFi devices

dennis

New VM running NixOS on the Proxmox HV. I built this as I had some RAM issues on dee, which I subsequently fixed but ended up putting more services on there to experiment with multiple hosts.

Service	Purpose
Caddy	Reverse proxy for services local to the host
Dashy	Replaces Heimdall as a dashboard for services
Grafana	Metric and log visualisation
Loki	Log collections
Prometheus with Thanos	Metric scraping and long term storage
Victoria Metrics	Metric scraping (just testing as a potential Prometheus replacement)

Additional features

Besides NixOS itself, there are some extra features which I’m using in the stack:

nix flakes
- Version pin dependencies
- Allows for an additional layer of reproducibility
agenix
- Encryption using age
- Allows for secrets to be safely checked in to Git repos so that they can be used in configurations
deploy-rs
- Deploy NixOS configurations to hosts remotely
home-manager
- Allows configuration of user-level programs and settings
- Typically used as a dotfile manager, which can replace tools such as chezmoi
- Can be applied to non-NixOS machines
nixos-hardware
- Crowdsourced configuration best practices to get NixOS running smoothly on particular hardware

Configurations

Since NixOS can be used to build packages and services, it can also be used to build configurations. I have defined a service “catalog” which acts as a source of truth for services, so that dependent services can refer back to this to build their own config. Some examples of this are generating configs used for:

DNS rewrites to forward DNS names to the host hosting the service
Reverse proxy the service to the port
Dashy for the home dashboard
Prometheus monitoring service endpoints

The inspiration for having a centralised location for service catalog came from jhillyerd - where they achieve the same thing.

My NixOS configurations are stored in GitHub, so you can have a browse through there. Subsequent posts will dive into what each of these is doing in more detail, but here’s a high-level of the top-level directories.

common/

Settings that are applied to all hosts

home-manager/

User-level programs and settings

hosts/

Configurations for individual hosts, along with their hardware configs via nixos-hardware

modules/

Where services and their dependencies are defined
Rarely do I install software via just services.SERVICE_NAME.enable = true, there is usually extra properties to come with it - so having wrapping them in a custom module is usually easiest for me to maintain

overlays/

Nix overlays allow you to overwrite packages and configurations pulled from a dependency
In my case here, I’m using it to fix some bugs in one program, and add functionality to another

secrets/

Where agenix secrets are stored

catalog.nix

Contains information about the various services and where

flake.nix

The main entry point into the configurations when deploying

Conclusion

This is only a quick introduction into the series which introduces NixOS, what I’m using it for, and a high-level look into the configurations.

Alerting on NHS Coronavirus Vaccine Updates With Huginn

Thu, 03 Jun 2021 00:00:00 +0000

UPDATE 2021-06-06

A few days after publishing this I came across a bug where agent jobs would be stuck in pending state. I’ve since fixed this and documented some additional changes I’ve made at the end of the post.

The UK’s coronavirus vaccine strategy has been to target those most vulnerable first, and then trickle down towards the healthier population. Since that age is creeping down toward my age group, I wanted to see if I could alert myself when I would be eligible for the vax.

My local GP would send out an SMS text message informing me when I’m eligible, however I’ve heard that this text can come days after you’re eligible. Knowing that the latest guidance is maintained on the NHS Coronavirus Vaccine site, I can use Huginn to alert me when the page updates with the latest eligibility.

tl;dr

Huginn is an automation tool with a number of different agents
It can be configured to monitor a property (or properties) on a web page and trigger an action
That action can take the form of an email alert
This can be used to monitor the latest age group eligible for a vaccine

Deploying Huggin

Huginn is a self-hosted automation kit that allows you to create agents and workflows in response to events, sort of like your own IFTTT (If This Then That).

Being self-hosted it can be deployed out in a number of ways. I already have Portainer (a GUI for Docker) running in a virtual machine - so to deploy it out I can follow the instructions for Docker container deployment. I created a docker-compose file so that it can be easily replicated for yourselves in Docker, or even as a Portainer Stack.

You’ll notice there are some environment variables for SMTP here; the values for these will differ for your SMTP setup. I talk more about how I set this up with my Gmail account later in the post.

Also big thanks to zblesk for their blog post on Huginn which helped me iron out some of the environment variables!

version: "3"
services:
  huginn:
    command:
      - /scripts/init
    container_name: huginn
    environment:
      - SMTP_PORT=587
      - SMTP_SERVER=<SMTP_SERVER>
      - SMTP_PASSWORD=<SMTP_PASSWORD>
      - SMTP_USER_NAME=<SMTP_USER_NAME>
      - SMTP_ENABLE_STARTTLS_AUTO=true
      - SMTP_AUTHENTICATION=plain
      - SMTP_DOMAIN=<SMTP_DOMAIN>
      - DATABASE_POOL=30
      - TIMEZONE=London
      - IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false
    image: huginn/huginn:latest
    ports:
      - 3000:3000/tcp
    restart: unless-stopped
    user: "1000"
    volumes:
      - mysql:/var/lib/mysql
    working_dir: /app

volumes:
  mysql:
    driver: local

Be sure to change TIMEZONE to your timezone, I found that having an incorrectly set timezone caused Huginn jobs to be backed up in a pending state. I tried Europe/London first but that caused the process to crash on boot; so I ultimately got it working with just London.

The config above will expose Huginn on the Docker host at port 3000. I like to give my services a nice domain name to access them at using Caddy, which I’ve written about before. Here’s a condensed version of what my Caddy config file looks like for Huginn.

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "192.168.2.15:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": ["huginn.joannet.casa"]
                }
              ],
              "terminal": true
            }
            // ... others removed for brevity
          ],
          "tls_connection_policies": [
            {
              "match": {
                "sni": [
                  "huginn.joannet.casa"
                  // ... others removed for brevity
                ]
              }
            },
            {}
          ]
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuer": {
              "challenges": {
                "dns": {
                  "provider": {
                    "api_token": "CLOUDFLARE_API_TOKEN",
                    "name": "cloudflare"
                  }
                }
              },
              "module": "acme"
            },
            "subjects": [
              "huginn.joannet.casa"
              // ... others removed for brevity
            ]
          }
        ]
      }
    }
  }
}

I’ll need to also add in a DNS entry in PiHole to route HTTP calls on my network for https://huginn.joannet.casa to the IP address of my Caddy server.

Deployed and ready for set up!

Setting up agents

The workflow we need to set up here is pretty simple whereby we only need two agents; a Website Agent and an Email Agent. The website agent will perform the scraping of the NHS website on a regular basis, and if the component on the web page has changed, then it will invoke it’s downstream notifier - the Email Agent.

Website Agent    ------ invokes ------>    Email Agent

Before we do that we’ll need to create an account for it - this is all local to your deployment and is not external. The invitation code you’ll need to enter is try-huginn.

When you get through to the main page, I suggest disabling or removing the agents that are there already to prevent some problems later on. If you set IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false then you should not see any there.

Website agent

From the Huginn home page, create a new agent, where the type will be Website Agent.

There will be a bunch of fields that appear, the only ones you need to fill in are:

Name
- e.g. NHSScrape
Schedule
- for me, checking once an hour is good enough
Options

The other fields serve a purpose that’s beyond the scope of this post.

Within Options comes the configuration used to define the website agent. The documentation for the agent config appears on the right hand side, so you can read through that for reference.

In order for us to determine how to configure this we first need to decide what part of the web page we want to be alerted on in event of a update. Refer to the screenshot below of the NHS Coronavirus Vaccine page and you’ll see a number of bullet points listed out of the vaccine criteria. The text we want to be alerted on is “people aged 30 and over”.

The website agent supports an xpath syntax as a config option, which is an expression syntax used to retrieve objects from XML documents - including HTML. But how do we find out what the xpath for this text field is?

Enter SelectorGadget. It’s a bookmarklet tool that helps with just that - I recommend taking a look at the short tutorial on how to use it.

Since I know I only want to select the first bullet point in this particular list, I start off by first clicking that property which turns that node green.

Clicking on the property I want to target highlights it green

You can see the xpath for this is a <li> node, which returns 76 results on the page we’re scraping. These are represented by the boxes that are highlighted yellow.

What I want to do is whittle it down by filtering out all the other <li> nodes I’m not interested in, so that I only have 1 result left. As I’ve already made my first selection, any subsequent clicks will now filter those out. So now it’s just a case of playing whack-a-mole until all yellow highlighted fields are gone.

Clicking the second bullet point indicated that I don’t care about any other <li> elements in this range.

<li> properties also make up headings at the top of the page - so these appear highlighted in yellow too.

Clicking the Home element filters that out.

Scrolling down further on the page, there’s another <li> range which needs to be filtered.

Boom - filtering that means we only have 1 element being targeted.

We now only have 1 selection, so now we can click the XPath button in the tool to retrieve the config we need.

//ul[(((count(preceding-sibling::*) + 1) = 6) and parent::*)]//li[(((count(preceding-sibling::*) + 1) = 1) and parent::*)]

N.B. since writing this it looks like the xpath changed - I’ve updated the config below with the same. I retrieved it using the same method as described above.

As a future task it’d be great to see if we could be alerted on when the working status of a job fails - but it seems that feature is missing.

Going back to the agent config, this xpath syntax then goes into the xpath key.

{
  "expected_update_period_in_days": "2",
  "url": "https://www.nhs.uk/conditions/coronavirus-covid-19/coronavirus-vaccination/coronavirus-vaccine/",
  "type": "html",
  "mode": "on_change",
  "extract": {
    "title": {
      "xpath": "//ul[(((count(preceding-sibling::*) + 1) = 4) and parent::*)]//li[(((count(preceding-sibling::*) + 1) = 1) and parent::*)]",
      "value": "normalize-space(.)"
    }
  }
}

Once it’s configured then you can click on Dry Run at the bottom to see the text it extracts.

Email agent

Now that the website agent is set up we need to set up our alert destination; this takes the form of an email agent. At minimum, we only need to configure these fields:

Name
- e.g. NHSEmail
Sources
- select the website agent you created beforehand
Options

Note that an agent such as email can have multiple sources - so if you wanted to be alerted on multiple web pages then you only need one email agent.

The options field is not as complex as the website agent - mine is configured with this:

{
  "subject": "NHS Coronavirus Page update",
  "headline": "Vaccine age updated",
  "expected_receive_period_in_days": "2"
}

The real complexity lies in configuring Huginn to send email…

Configure Huginn for sending email over Gmail SMTP

I have a Gmail account which opens up SMTP access to allows applications to send email programatically, where Huginn has support for this. You’ll notice earlier when we deployed Huginn that I had a number of environment variables configured for SMTP. It took a bit of trial-and-error and searching through GitHub issues to get it right, but that config works for me.

Since my Gmail account is set up with 2FA, I cannot use my actual Gmail password in the SMTP_PASSWORD field. Instead what I have to do is set up an application-specific password that only Huginn is configured for. This restriction known in Google as less secure app access.

As mentioned, we’ll need to set up an app password. For this I navigated to the app password page for my account and selected Other (Custom name) from the app dropdown, to then enter the name of the app. The name doesn’t matter here - it’s for your reference.

When you click on Generate it will display the password assigned. It is this value that goes into the SMTP_PASSWORD env var for Huginn to pick up.

As a recap, the environment variables that I needed to set in order to get emails to be sent were:

SMTP_DOMAIN=gmail.com
SMTP_SERVER=smtp.gmail.com
SMTP_PORT=587
SMTP_AUTHENTICATION=plain
SMTP_USER_NAME=$EMAIL_ADDRESS
SMTP_PASSWORD=$APP_PASSWORD
SMTP_ENABLE_STARTTLS_AUTO=true
DATABASE_POOL=30
- While not directly related to SMTP, it helps ensure there are enough threads to process database requests

Note if you did not start up Huginn with these environment variables already set, then you’ll need to restart the service so that it can pick them up.

Testing the flow

We can test the whole flow by making a modification to the website agent we created. Currently the mode we have it set to is on_change which is the desired end mode, and will only trigger its receivers if there has been a change in the property being selected. If we change the mode to be all then it will always invoke the receivers.

Couple this with setting a frequent schedule (i.e. every 5m) then we should be receiving an email with the current value every 5 minutes.

Once the flow is tested, we can set the mode on the website agent back to on_change

Now we just play the waiting time until getting vaxx’ed up as Marc Rebillet says… 💉

Fixing agent jobs stuck in pending state

Using the docker compose file above caused an issue for me where Huginn would get jobs stuck in a pending state - where rebooting the container was the only way to unblock them… no good for an alerting app!

Huginn by default includes a mysql daemon as the datastore if none is provided in the environment variables. I decided to have mysql running in a separate container to see if that fixed it… and it did!

My new docker compose file looks like this:

services:
  huginn:
    command:
      - /scripts/init
    container_name: huginn_huginn
    environment:
      - SMTP_PORT=587
      - SMTP_SERVER=<SMTP_SERVER>
      - SMTP_PASSWORD=<SMTP_PASSWORD>
      - SMTP_USER_NAME=<SMTP_USER_NAME>
      - SMTP_ENABLE_STARTTLS_AUTO=true
      - SMTP_AUTHENTICATION=plain
      - SMTP_DOMAIN=<SMTP_DOMAIN>
      - TIMEZONE=London
      - DATABASE_POOL=30
      - DATABASE_NAME=huginn
      - DATABASE_USERNAME=huginn
      - DATABASE_PASSWORD=<MYSQL_PASSWORD>
      - DATABASE_HOST=huginn_mysql
      - DATABASE_PORT=3306
      - START_MYSQL=false
      - DATABASE_ENCODING=utf8mb4
      - IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS=false
      - DOMAIN=huginn.joannet.casa
      - INVITATION_CODE=<INVITATION_CODE>
    image: huginn/huginn:latest
    ports:
      - 3000:3000/tcp
    restart: unless-stopped
    user: "1000"
    working_dir: /app
    depends_on:
      - mysql
  mysql:
    image: mysql
    container_name: huginn_mysql
    restart: always
    ports:
      - "3306:3306"
    environment:
      - MYSQL_ROOT_PASSWORD=<MYSQL_ROOT_PASSWORD>
      - MYSQL_DATABASE=huginn
      - MYSQL_USER=huginn
      - MYSQL_PASSWORD=<MYSQL_PASSWORD>
    volumes:
      - mysql:/var/lib/mysql

volumes:
  mysql:
    driver: local

The mysql container is pretty standard so I won’t cover that here. There are some env vars I had to add to the huginn container:

DATABASE_HOST - the database hostname to connect to, we can use the container name here
DATABASE_PORT - the port to which to connect to the database
DATABASE_NAME - the name of the database to use
DATABASE_USERNAME - who we should connect to the database as
DATABASE_PASSWORD - authentication for the user
DATABASE_ENCODING - a requirement when using a newer version of MySQL as defined in the documentation
START_MYSQL - whether to use a local mysql daemon or not

Some additional env vars I added unrelated to the new database:

IMPORT_DEFAULT_SCENARIO_FOR_ALL_USERS - I don’t care able the agents that are added by default
INVITATION_CODE - Lock down Huginn by requiring this code for new user sign ups
DOMAIN - the endpoint that Huginn is available at

I’ve also added in a depends_on on the database container to assist with orchestration. On first boot however it takes some time for mysql to initialise the database, so Huginn may fail as the database is not yet ready to be connected to. Once the initialisation is done then reboot the Huginn container and it should be able to bootstrap the database fine.

Backup to Backblaze B2 using restic and rclone

Tue, 04 May 2021 00:00:00 +0000

Over the last UK lockdown I spent some time making sure I had backups of my music collection after I had organised them using beets. I used this as a good opportunity to ensure I had other backups in place for some other critical files at home too.

Backup tools

Restic is a backup snapshot tool that given a set of directories, it will split the data into chunks and de-duplicate these chunks to a specified backup repository. You can specify your backup policy to a repository, where it will manage what daily, weekly, or monthly snapshots to keep.

Rclone is a cloud file transfer tool that allows you to synchronise, copy, etc., any data across a huge library of cloud backends.

Both of these are open source tools that are free to use. Getting set up on them is beyond the scope of this post.

Backups should be automated without us having to think of them (at least until you need to restore!), so for this I’ll be using systemd unit services with timers.

Restic is storing the snapshots in repositories on my local NFS server. I’m then using Backblaze B2 to store these in the cloud. At the time of writing they charge $0.005 GB/month for storage, and $0.01 per GB downloaded. So it costs more to download from B2, but since this is a disaster recovery backup I don’t anticipate downloading all that much. Being able to store it cheap over a long term is most important.

There are a load of other storage services too; here’s a list below taken from Backblaze’s website. The other one I was contending with was Wasabi; they’re only marginally more expensive than B2 for storage ($0.0059 GB/month), but they don’t charge for downloads.

Cloud storage cost comparisons

Backup contents and policy

I have restic set up to backup to two respositories:

small-files
- PiHole configuration
- UniFi backups
- Logitech Media Server (LMS) backups
media
- all music files
- beets databases

Reason being why these are split into two repositories is so that I can define a separate backup policy for each of them.

For example, small-files contains… small files. You can see from the list above that these are just configuration files or even backups of files themselves. Since config files are important, I’d like the option to go back to a particular setting from weeks or maybe months ago. So the snapshots I want to keep for this repository is 30 daily snapshots, 5 weekly snapshots, 12 monthly snapshots, and 3 yearly snapshots. It might be a bit overkill, but I can always reduce that number and restic will remove them.

Whereas for media, these files tend to be much larger which will be reflected in the repository size. Especially since music files don’t tend to change over time - all I’m looking for is a way of reverting back to a state I had recently in case I did some reorganising with beets. Based on this I only need to keep 30 daily snapshots.

Also in the media repository are the beets databases. I’ll expand on how I have this set up in a future post, but for now think of it as a database containing the metadata for your music collection. It’s useful to include this with my music files so that I can reflect on the state of my collection at a particular point in time.

Backup procedure

I have a 2TB USB hard drive connected to a Pi (dee) which is configured to be the NFS server for my network, along with PiHole for DNS, and a UniFi controller too. The drive holds my music collection and the restic local repositories.

LMS sits on a different machine to dee so we’ll need to retrieve the files first before we perform a snapshot.

Once restic has taken snapshots and stored them on the USB drive, we’ll need to upload the backups to B2 with rclone.

N.B. the configuration to automate restic with systemd was largely inspired from a post on Fedora Magazine.

You can view the code for all these scripts and systemd unit files at my GitHub repository.

Retrieving backups from remote servers

I’ll have the restic backup script call another script, pull-backups.sh to retrieve remote files that I want to back up.

For now I only need to retrieve LMS backups from PiCorePlayer - for which there is a handy command to help with that: pcp bu. Don’t get too invested in the file path locations, these are specific to PiCorePlayer.

# pull-backups.sh

#!/usr/bin/env bash

# Invoke and pull backups from remote servers and place them on USB

function pcp() {

    local fpath="/mnt/mmcblk0p2/tce"
    local fname="mydata"
    local ext="tgz"
    local now=$(date -u +"%Y-%m-%dT%H%M%S")
    local dstPath="/mnt/usb/Backup/lms/*.${ext}"

    echo "removing previous backups..."
    rm -rf $dstPath

    local sourceFile="${fpath}/${fname}.${ext}"
    local dstFile="/mnt/usb/Backup/lms/${fname}_${now}.${ext}"

    echo "starting pcp backup"
    ssh -i /home/jdheyburn/.ssh/pcp tc@pcp.joannet.casa -C 'pcp bu'

    echo "copying $sourceFile on remote to $dstFile"
    scp -i /home/jdheyburn/.ssh/pcp "tc@pcp.joannet.casa:${sourceFile}" $dstFile
}

function main() {
    echo "entered $0"
    pcp
}

main $@

This takes the backup created on the remote at /mnt/mmcblk0p2/tce/mydata.tgz and places it locally at /mnt/usb/Backup/lms/mydata_DATETIME.tgz. This lms directory can then be used as a backup path for restic.

Should I need to retrieve backups from any other servers, I’ll write a new function here and append it to main.

Restic backup

# restic-all.sh

#!/usr/bin/env bash

# Master script for backing up anything to do with restic

function do_restic() {
    local mode=$1
    local target=$2

    if [ $mode == "backup" ]; then
        echo "Backing up $target"
        restic backup --verbose --tag systemd.timer $BACKUP_EXCLUDES $BACKUP_PATHS
        echo "Forgetting old $target"
        restic forget --verbose --tag systemd.timer --group-by "paths,tags" --keep-daily $RETENTION_DAYS --keep-weekly $RETENTION_WEEKS --keep-monthly $RETENTION_MONTHS --keep-yearly $RETENTION_YEARS
    elif [ $mode == "prune" ]; then
        echo "pruning $target"
        restic --verbose prune
    fi
}

function small_files() {
    export BACKUP_PATHS="/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms"
    export BACKUP_EXCLUDES=""
    export RETENTION_DAYS="7"
    export RETENTION_WEEKS="5"
    export RETENTION_MONTHS="12"
    export RETENTION_YEARS="3"
    export RESTIC_REPOSITORY="/mnt/usb/Backup/restic/small-files"
    export RESTIC_PASSWORD_FILE="/home/restic/.resticpw"

    local mode=$1

    do_restic $mode "small files"
}

function media() {
    export BACKUP_PATHS="/mnt/usb/Backup/media/beets-db /mnt/usb/Backup/media/lossless /mnt/usb/Backup/media/music /mnt/usb/Backup/media/vinyl"
    export BACKUP_EXCLUDES=""
    export RETENTION_DAYS="30"
    export RETENTION_WEEKS="0"
    export RETENTION_MONTHS="0"
    export RETENTION_YEARS="0"
    export RESTIC_REPOSITORY="/mnt/usb/Backup/restic/media"
    export RESTIC_PASSWORD_FILE="/home/restic/.resticmediapw"

    local mode=$1

    do_restic $mode "media"
}

function main() {

    mode=$1

    if [ $mode == "backup" ]; then
        /home/jdheyburn/dotfiles/restic/pull-backups.sh
    fi

    small_files $mode
    media $mode
}

main $@

This script is a bit messy, but it gets the job done. It takes in an argument which can be either backup or prune, depending on what task needs to run. If the mode is backup then it will invoke the pull-backups.sh prior to snapshotting so that it has the latest files to backup. I cover the prune function later.

Both restic repositories will be kept under /mnt/usb/Backup/restic.

Once the script is defined we can have systemd invoke it.

# /etc/systemd/system/restic-all.service

[Unit]
Description=Restic backup everything service
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/restic-all.sh backup

[Install]
WantedBy=multi-user.target

You can see that we’re passing in the mode as an argument at ExecStart.

We can perform a test to make sure everything is defined correctly by running systemctl start restic-all.service, and viewing the logs back at journalctl -u restic-all.service -f.

In order to then have this systemd unit invoked on a regular occurrence, we need to define a timer for this unit.

# /etc/systemd/system/restic-all.timer

[Unit]
Description=Backup with restic daily

[Timer]
OnCalendar=*-*-* 2:00:00
Persistent=true

[Install]
WantedBy=timers.target

This will invoke the service at 2am every day once we enable it with systemctl enable restic-all.timer.

We can have a look at the resulting snapshots with the below command.

$ restic -r /mnt/usb/Backup/restic/small-files/ snapshots
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
ID        Time                 Host        Tags           Paths
------------------------------------------------------------------------------------------
d93573f4  2020-06-30 02:00:01  dee         systemd.timer  /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

96ecf97e  2020-07-31 02:00:35  dee         systemd.timer  /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

d2b0a78e  2020-08-31 02:00:21  dee         systemd.timer  /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

... # removed for brevity

ec6fd77e  2021-05-03 02:00:34  dee         systemd.timer  /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup

722fcbbf  2021-05-04 02:00:34  dee         systemd.timer  /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup
------------------------------------------------------------------------------------------
39 snapshots

Syncing to B2 with rclone

Restic has now created snapshots and stored them in their respective repositories on the USB drive - but this isn’t really a safe place to keep backups since the USB drive could crap out at any moment. We can use rclone to offload the repositories to B2.

For this I’m going to use the same pattern as before for restic; a shell script invoked by systemd.

# rclone-all.sh

#!/usr/bin/env bash

# Master script for backing up all rclone stuff to various clouds

RCLONE_CONFIG=/home/jdheyburn/.config/rclone/rclone.conf

function main() {

    echo "rcloning beets-db -> gdrive:media/beets-db"
    rclone -v sync /mnt/usb/Backup/media/beets-db gdrive:media/beets-db --config=${RCLONE_CONFIG}

    echo "rcloning music -> gdrive:media/music"
    rclone -v sync /mnt/usb/Backup/media/music gdrive:media/music --config=${RCLONE_CONFIG}

    echo "rcloning lossless -> gdrive:media/lossless"
    rclone -v sync /mnt/usb/Backup/media/lossless gdrive:media/lossless --config=${RCLONE_CONFIG}

    echo "rcloning vinyl -> gdrive:media/vinyl"
    rclone -v sync /mnt/usb/Backup/media/vinyl gdrive:media/vinyl --config=${RCLONE_CONFIG}

    echo "rcloning restic -> b2:restic"
    rclone -v sync /mnt/usb/Backup/restic b2:iifu8Noi-backups/restic/ --config=${RCLONE_CONFIG}

    echo "Done rcloning"
}

main $@

In addition to the restic repository directory, I’m also backing up the beets databases and music files. These are going to my Google Drive storage in case I want to hook it up to some other apps that can pull from there.

Like restic-all.sh above, this script is also invoked by systemd.

# /etc/systemd/system/rclone-all.service

# This should be invoked after restic has done doing the daily backup

[Unit]
Description=Rclone backup everything service
After=restic-all.service
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/rclone-all.sh

[Install]
WantedBy=restic-all.service

We can test it with systemctl start rclone-all.service.

Since we want to have rclone invoked after restic has done its thing, we need to specify After=restic-all.service and WantedBy=restic-all.service. Note that this’ll run even if restic-all.service failed - but that’s not really an issue for me since rclone is uploading more than just restic respositories.

We don’t need to specify a systemd .timer file for this unit file since we’re using the completion of restic-all.service as our invocation point, so we can start that service in order to test it’ll run afterward.

systemctl start restic-all.service

Handling service failures

N.B. big thanks to Laeffe for their excellent guide on this.

Automated backups are very useful for setting and forgetting, but when something goes wrong and backups haven’t occurred, we need to be made aware of it.

We can configure each of the systemd files to invoke a script on failure which can then send us an email when this happens.

You’ll notice that each of the systemd unit files have OnFailure=unit-status-mail@%n.service defined in them. This is an additional unit file where if the scripts fail, this service will be invoked.

# /etc/systemd/system/unit-status-mail@.service

[Unit]
Description=Unit Status Mailer Service
After=network.target

[Service]
Type=simple
ExecStart=/home/jdheyburn/dotfiles/restic/systemd/unit-status-mail.sh %I "Hostname: %H" "Machine ID: %m" "Boot ID: %b"

The @ symbol in the filename is a special case within systemd that enables special functionality. In our case when we invoked it at OnFailure=unit-status-mail@%n.service the calling unit file will pass its name via %n to allow the receiving unit file to access it at %I. So when the restic-all service fails, this value ends up in unit-status-mail@.service at %I.

This %I variable is then being passed as an argument to the script unit-status-mail.sh. The contents of the script are below:

# unit-status-mail.sh

#!/bin/bash

# From https://northernlightlabs.se/2014-07-05/systemd-status-mail-on-unit-failure.html

MAILTO="ADD_EMAIL_HERE"
MAILFROM="unit-status-mailer"
UNIT=$1

EXTRA=""
for e in "${@:2}"; do
  EXTRA+="$e"$'\n'
done

UNITSTATUS=$(systemctl status $UNIT)

sendmail $MAILTO <<EOF
From:$MAILFROM
To:$MAILTO
Subject:Status mail for unit: $UNIT

Status report for unit: $UNIT
$EXTRA

$UNITSTATUS
EOF

echo -e "Status mail sent to: $MAILTO for unit: $UNIT"

The name of the calling unit file is passed to UNIT where the status of it is received, and the output is sent in an email to MAILTO.

You’ll need to make sure you have sendmail installed on the machine to do this.

sudo apt install sendmail -y

For me, the emails landed in the spam folder - but once you configure a rule on your email client to always forward them to your inbox, you’ll always be notified if there’s an error.

The resulting email

Removing aged restic snapshots

One last area to look at with restic is pruning, this is where restic will remove old data that has been “forgotten”.

Back in restic-all.sh we are calling restic forget after each backup - this tells restic to clean up any old snapshots not required by our defined backup policy. The script has been written to accommodate for both backup and prune functionality, so in order to execute that portion of the script we need to set up another systemd unit file to invoke it.

# /etc/systemd/system/restic-prune.service

[Unit]
Description=Restic backup service (data pruning)
OnFailure=unit-status-mail@%n.service

[Service]
Type=oneshot
ExecStart=/home/jdheyburn/dotfiles/restic/restic-all.sh prune

Since it is resource intensive to perform prune against your repository, its best to run this at a different interval to your backups. From the below unit file OnCalendar=*-*-1 10:00:00 corresponds to the first day of the month at 10am.

[Unit]
Description=Prune data from the restic repository monthly

[Timer]
OnCalendar=*-*-1 10:00:00
Persistent=true

[Install]
WantedBy=timers.target

Restoring a backup

Now the most important part - how to restore a restic snapshot. Let’s start with how to restore from a local repository.

Restore from local restic repository

Firstly we need to find the snapshot ID that we want to restore to - in this example I want the latest snapshot.

$ restic -r /mnt/usb/Backup/restic/small-files/ snapshots --last
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
ID        Time                 Host        Tags           Paths
------------------------------------------------------------------------------------------
f7ef9c33  2021-04-17 02:00:51  dee         systemd.timer  /etc/pihole
                                                          /mnt/usb/Backup/media/beets-db
                                                          /var/lib/unifi/backup/autobackup

e2a73c00  2021-04-21 02:00:21  dee         systemd.timer  /etc/pihole
                                                          /var/lib/unifi/backup/autobackup

722fcbbf  2021-05-04 02:00:34  dee         systemd.timer  /etc/pihole
                                                          /mnt/usb/Backup/lms
                                                          /var/lib/unifi/backup/autobackup
------------------------------------------------------------------------------------------
3 snapshots

So the ID is 722fcbbf, let’s browse the contents to see if the file we want is in there.

$ restic -r /mnt/usb/Backup/restic/small-files/ ls 722fcbbf
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
snapshot 722fcbbf of [/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms] filtered by [] at 2021-05-04 02:00:34.383403601 +0100 BST):
/etc
/etc/pihole
/etc/pihole/GitHubVersions
/etc/pihole/adlists.list
# ... removed for brevity

Let’s say we want to restore /etc/pihole/adlists.list, we can use the --include argument to specify just that. If we didn’t use a filter argument then restic would default to restoring the entire contents of the snapshot.

$ restic -r /mnt/usb/Backup/restic/small-files/ restore 722fcbbf --target /tmp/restic-restore --include /etc/pihole/adlists.list
repository 79dbc9b6 opened successfully, password is correct
found 1 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
restoring <Snapshot 722fcbbf of [/var/lib/unifi/backup/autobackup /etc/pihole /mnt/usb/Backup/lms] at 2021-05-04 02:00:34.383403601 +0100 BST by root@dee> to /tmp/restic-restore

$ tree /tmp/restic-restore/
/tmp/restic-restore/
└── etc
    └── pihole
        └── adlists.list

Restore from B2 restic repository

We’re storing the repositories on B2 too, and restic has B2 integration built into it. So in the scenario where the NFS server had died and we needed to restore a snapshot stored on B2, we can hit it directly without having to use rclone to pull down the entire repository for us to restore from. This’ll be a cheaper approach as restic is only pulling down the files it needs to restore from, lowering B2 download costs.

We just need to configure some variables to permit restic to hit B2. If you’re using rclone you can use the same ID and key here.

export B2_ACCOUNT_ID="ACCCOUNT_ID"
export B2_ACCOUNT_KEY="ACCCOUNT_KEY"
export RESTIC_PASSWORD_FILE="/path/to/passwordfile
restic -r b2:BUCKET_NAME:restic/small-files snapshots --last

From here you can then use the same commands as in the local repository to traverse the snapshots and restore.

Conclusion

The process above is probably more complex than what it needs to be; having a separate process for backing up and restoring. Since I want to back up to the NFS locally first followed by B2, this approach made the most sense as it allows me to minimise download costs from B2 through targeting the local repository first.

What’s most important is that backups are being made, and that I can restore from them.

How to automate zero downtime maintenance with AWS SSM & ALBs

Tue, 19 Jan 2021 00:00:00 +0000

Welcome to the last post in this series where we’ve been exploring SSM Documents, so far we’ve covered:

How Command Documents can help to execute commands on EC2 Instances
Automating these Command Documents through Maintenance Windows
Safely chaining Command Documents through Automation Documents, and aborting for any failures

This post will now look into how we can use Automation Documents to perform maintenance on EC2 instances without impacting user experience.

tl;dr

With the introduction of load balancers to front your services, you can control which instances should be receiving traffic
This enables you to proactively remove instances from rotation so that you can perform maintenance on the backends to minimalise user disruption
SSM automation documents can enable us to execute pre-maintenance steps such as removing an instance from a load balancer, as well as adding them back after
- See the document produced in this post highlighting this, and where I explain how it works, and how to deploy it using Terraform

Prerequisites

If you’re just joining in from this post then I recommend reading through the previous posts to gain an understanding of how we got here; or if you know what you’re looking for the tl;dr provides a summary.

As always, the code for this post can be found on GitHub.

A basic understanding and knowledge of Application Load Balancers (ALB), and its components (e.g. target groups) is required.

Introducing load balancers

Load balancers are a key component in software architecture that distribute traffic and requests across backend services in a variety of algorithms, such as:

Round-robin
- every backend serves the same number of requests
- the most commonly used algorithm
Weighted round-robin
- backends receive a fixed percentage of incoming requests
- useful if some backends are more beefy than others
- also used in canary deployments
Least outstanding requests
- the backend which is currently processing the least number of requests is forwarded the request

There are multiple benefits to having a load balancer sit in front of your services:

Protecting your infrastructure; user requests are proxied via the load balancer
Distribute traffic and requests however you like
Perform healthchecks on backends and don’t forward traffic to unhealthy nodes
Drain and remove backends to permit for rolling upgrades

While there are several different software-based load balancers out there such as nginx and HAProxy, AWS has its own managed load balancer service known as an Elastic Load Balancer (ELB).

Adding web services to our demo environment

In order for us to get the benefit of load balancers to front the EC2 instances in the architecture this series left off from last time, we will need to have a service running on our instances.

Let’s simulate a real web service by running a simple Hello World application across each of the instances. We can utilise EC2s user data to start a basic service up for us by giving it a script to run on instance provision.

This is the architecture we’ll be building out in this section, with an ALB fronting our EC2 instances

Let’s use Go to create a web service for us since it is easy to get set up quickly - we’ll have the server return a simple Hello, World! message when a request hits it. Let’s also have it return the name of the instance that was hit - this will be used later.

#!/bin/bash
cat <<'EOF' > /home/ec2-user/main.go
package main

import (
        "fmt"
        "net/http"
        "os"
)

func main() {
        http.HandleFunc("/", HelloServer)
        http.ListenAndServe(":8080", nil)
}

func HelloServer(w http.ResponseWriter, r *http.Request) {
        hostname, _ := os.Hostname()
        fmt.Fprintf(w, "Hello, World! From %v\n", hostname)
}
EOF
yum install golang -y
(crontab -l 2>/dev/null; echo "@reboot nohup go run /home/ec2-user/main.go") | crontab -
export GOCACHE=/tmp/go-cache
nohup go run /home/ec2-user/main.go

So on instance creation this will:

Create a new file called main.go and populate it with the lines between the EOF delimiters
Install Go
Create a crontab entry to run the service on subsequent boots
Run the Go application in the background immediately

While user data is for useful provisioning services, I don’t advise storing the source code of your application in there like I’ve done - this is just a hacky way to get something up and running.

We’ve been using Terraform to provision our nodes. So we need to save this script in a file (scripts/hello_world_user_data.sh), and then pass it into the user_data attribute of our EC2 module.

module "hello_world_ec2" {
  source         = "terraform-aws-modules/ec2-instance/aws"
  version        = "~> 2.0"

  # ... removed for brevity

  user_data = file("scripts/hello_world_user_data.sh")
}

Terraform will recreate any EC2 nodes with a change in user_data contents, so when you invoke terraform apply all instances will be recreated.

N.B. the AMI I picked up for my EC2 instances has an issue with SSM Agent. Ensure you execute the AWS-UpdateSSMAgent across your instances after they have provisioned, or you can use an SSM Association document to do that for you as shown here.

After they have all successfully deployed, you should be able to curl the public IP address of each instance from your machine to verify your setup is correct. If you are getting timeouts then make sure your instances have a security group rule permitting traffic from your IP address through port 8080.

$ curl http://54.229.209.60:8080
Hello, World! From ip-172-31-39-169.eu-west-1.compute.internal

$ curl http://3.250.160.209:8080
Hello, World! From ip-172-31-21-197.eu-west-1.compute.internal

$ curl http://34.254.238.146:8080
Hello, World! From ip-172-31-8-52.eu-west-1.compute.internal

Fronting instances with a load balancer

Now that we have a web service hosted on our instances, let’s now add a load balancer in front of it. This load balancer will now become the point of entry for our application instead of hitting the EC2 instances directly.

For this I am using a Terraform ALB module for provisioning all the components in the load balancer, and expanding on them is beyond the scope of this post.

You can navigate to the AWS ALB documentation to find out about the underlying components the module creates for us.

Security groups

Before we can provision the load balancer, we need to specify the security group (SG) and the rules that should be applied to it. You can view this on GitHub.

Since our application is written to serve request on port 8080, we need to permit both the new aws_security_group.hello_world_alb SG and the existing aws_security_group.vm_base SG to communicate between each other.

resource "aws_security_group" "hello_world_alb" {
  name   = "HelloWorldALB"
  vpc_id = data.aws_vpc.default.id
}

resource "aws_security_group_rule" "alb_egress_ec2" {
  security_group_id        = aws_security_group.hello_world_alb.id
  type                     = "egress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.vm_base.id
}

resource "aws_security_group_rule" "ec2_ingress_alb" {
  security_group_id        = aws_security_group.vm_base.id
  type                     = "ingress"
  from_port                = 8080
  to_port                  = 8080
  protocol                 = "tcp"
  source_security_group_id = aws_security_group.hello_world_alb.id
}

Now we need to open up the ALB to allow traffic to hit it. In our case it will just be us hitting it, but this will change depending on who the consumer of the service is. If it is to serve traffic from the Internet then cidr_blocks would be ["0.0.0.0/0"].

The ALB will be hosting the traffic on insecure HTTP (port 80).

resource "aws_security_group_rule" "alb_ingress_user" {
  security_group_id = aws_security_group.hello_world_alb.id
  type              = "ingress"
  from_port         = 80
  to_port           = 80
  protocol          = "tcp"
  cidr_blocks       = [local.ip_address]
}

ALB module

The module documentation will tell us how we need to structure it. Our requirements dictate we need the following:

Receive traffic on port 80
Forward traffic to backend targets on port 8080
Include health checks to ensure we do not forward requests to unhealthy instances

Translate these requirements into the context of the module and we have something like this:

module "hello_world_alb" {
  source  = "terraform-aws-modules/alb/aws"
  version = "~> 5.0"

  name = "HelloWorldALB"

  load_balancer_type = "application"

  vpc_id          = data.aws_vpc.default.id
  subnets         = tolist(data.aws_subnet_ids.all.ids)
  security_groups = [aws_security_group.hello_world_alb.id]

  target_groups = [
    {
      name_prefix      = "pref-"
      backend_protocol = "HTTP"
      backend_port     = 8080
      target_type      = "instance"
      health_check = {
        enabled             = true
        healthy_threshold   = 2
        unhealthy_threshold = 2
        interval            = 6
      }
    }
  ]

  http_tcp_listeners = [
    {
      port               = 80
      protocol           = "HTTP"
      target_group_index = 0
    }
  ]
}

In order to keep this post simple I am not fronting services over HTTPS (secure HTTP) - I would strongly advise against doing this for non-test scenarios.

Another step we will need is to hook up our EC2 instances with the target group created.

resource "aws_lb_target_group_attachment" "hello_world_tg_att" {
  count            = length(module.hello_world_ec2.id)
  target_group_arn = module.hello_world_alb.target_group_arns[0]
  target_id        = element(module.hello_world_ec2.id, count.index)
  port             = 8080
}

There’s some clever Terraform going on here. All we’re doing is looping over each of the created EC2 instances in the module and adding it to the target group, to receive traffic on port 8080.

Hitting the load balancer

Whereas earlier when we were testing the services by hitting the EC2 instances directly, we’ll now be hitting the ALB instead. You can grab the ALB DNS name from the console.

$ curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal

Nice - and we can see from here what instance we’ve hit in the backend, since we included the hostname in the response.

If we now hit the ALB one more time, we will get a different instance respond.

$ curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal

This is the load balancer rotating between the backends available to it. We can see the rotation by repeatedly hitting the endpoint.

$ while true; do curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80 ; sleep 0.5; done
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal
Hello, World! From ip-172-31-31-223.eu-west-1.compute.internal
Hello, World! From ip-172-31-0-10.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal
Hello, World! From ip-172-31-43-11.eu-west-1.compute.internal

So what does all this have to do with our maintenance document?

Now that we have a load balancer fronting our services, let’s review executing our automation document from the last post and the problem it brings.

If an instance were to be rebooted during the AWS-RunPatchBaseline stage of the automation document, then there is a chance that a request would have been forwarded to that instance before the health checks against it have failed.

To simulate this, let’s create a new automation document which simulates a reboot. We’ll just take our existing patching document and replace the patch step with a reboot command, following the AWS guidelines to do this. I’ve also modified the health check to check to see if the new service came up okay - this may differ for your environment.

You can view the document on GitHub.

---
schemaVersion: "0.3"
description: Executes a reboot on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
mainSteps:
  - name: InvokeReboot
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters:
        commands: |
          flag_location=/home/ec2-user/REBOOT_STARTED
          if [ ! -f $flag_location ]; then
            echo "Creating flag file at $flag_location"
            touch $flag_location
            echo "Reboot initiated"
            exit 194
          fi
          echo "Reboot finished, removing flag file at $flag_location"
          rm $flag_location          
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunShellScript
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters:
        commands: |
          sleep 60
          if ! curl http://localhost:8080/; then
            exit 1
          fi

Then the Terraform code to create this document will look like this:

resource "aws_ssm_document" "reboot_with_healthcheck" {
  name            = "RebootWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/reboot_with_healthcheck_template.yml",
    {
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

Testing for failure

After this has been applied in our environment let’s get our test set up. In a terminal window from your machine have this script running in the background to simulate load.

$ while true;
do
  resp=$(curl http://HelloWorldALB-128172928.eu-west-1.elb.amazonaws.com:80 2>/dev/null)
  if echo $resp | grep -q html; then
    error=$(echo $resp | grep -oPm1 "(?<=<title>)[^<]+")
    echo "Error - $error"
  else
    echo $resp;
  fi
  sleep 0.5
done

Then we need to invoke the new reboot document against an instance (see here for how we achieved this last time). Once it is running let’s monitor the output of the command in your terminal.

Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Error - 502 Bad Gateway
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-14-19.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal
Hello, World! From ip-172-31-18-158.eu-west-1.compute.internal

Not good! 😧

These error messages are coming from the load balancer, indicating that the underlying backend wasn’t able to complete the request. This is happening when the instance gets rebooted as specified in our document.

The load balancer hasn’t had enough time to determine whether the instance is unhealthy or not - as dictated from our health check policy (2 failed checks with 6 seconds between them) - and so still forwards traffic to it even though it cannot respond.

We can actually view this disruption in CloudWatch Metrics too. ALBs expose metrics for us which we can monitor against, including:

RequestCount
- Informs us how many incoming requests the ALB is receiving
HTTPCode_ELB_5XX_Count
- How many HTTP 5XX error codes are being returned by the ALB

We can chart them together for visualisation.

HTTPCode_ELB_5XX_Count only reports on failures - if the metric is missing data points then no errors occurred at that time

While we are the only users hitting this, had this been a production box hit by 1,000s of users, each one of them would experience an issue with your application, and equals lost customers! 😡

What we need is a means of removing the node from the load balancer rotation so that we can safely perform maintenance on it.

Removing instances from load balancer rotation

Load balancer target groups have an API endpoint that allow you to drain connections from backends - where the load balancer stops any new requests being forwarded to that backend, and allows existing requests to complete. This can be done via - you guessed it - Automation Documents!

Graceful load balancer document

You can see the document in its entirety here. The steps that it performs are:

Check that the target group is healthy
- We want to ensure we’re not fuelling a dumpster fire
Check that the instance we’re targeting is in the target group we’re modifying
- Otherwise what’s the point? :upside_down:
Remove the instance from the target group and wait for it to be removed
Execute our maintenance document
Register the instance back and wait for it to be added back

Document metadata

As this is an automation document, the schemaVersion should be 0.3. We’re using two parameters here to run the document:

TargetGroupArn - the ARN of the target group we are making modifications too
InstanceId - the instance that is undergoing maintenance

schemaVersion: "0.3"
description: Gracefully reboot instance with healthchecks
parameters:
  InstanceId:
    type: String
    description: The instance to target
  TargetGroupArn:
    type: String
    description: The target group ARN for the instance

Sanity checks

The first two steps of the document are sanity checking the target group to ensure preconditions are met before we introduce change. We’re using the aws:assertAwsResourceProperty action to allow us to query against the AWS ELBv2 API (Elastic Load Balancer v2) and verify a response is what we expect it to be.

You can see all the API endpoints available for ELBv2 at this location.

The response of the DescribeTargetHealth endpoint returns an object that is structured like this.

{
  "TargetHealthDescriptions": [
    {
      "Target": {
        "Id": "i-083c8ca9c9b74e1cd",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    },
    {
      "Target": {
        "Id": "i-08a63b118a0b2a6b7",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    },
    {
      "Target": {
        "Id": "i-08c656b7160dd6729",
        "Port": 8080
      },
      "HealthCheckPort": "8080",
      "TargetHealth": {
        "State": "healthy"
      }
    }
  ]
}

This represents all the backends that are configured against the target group. The property selector $.TargetHealthDescriptions..TargetHealth.State specified in the document will check against all state fields to see if they are healthy. If any of them aren’t then the document will be aborted. As mentioned before, this check is performed to ensure we’re not causing more problems for ourselves if any of the nodes are unhealthy.

- name: AssertTargetGroupHealthBefore
  description: Assert the target group is healthy before we bounce Tomcat
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 3
  timeoutSeconds: 60

In the next sanity check we’re ensuring that the instance is definitely in the target group we want to remove it from. This is a slightly different query to the last step where we’re specifically requesting for state health on that one instance.

- name: AssertInstanceIsInTargetGroup
  description: Assert the instance is a healthy target of the target group
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"
  maxAttempts: 3
  timeoutSeconds: 60

Remove the instance from rotation

Now our preconditions have been met we can remove the instance using the aws:executeAwsApi action. This action is similar to aws:assertAwsResourceProperty in that it calls an AWS API endpoint, but we’re not checking the response of it - in fact the DeregisterTargets endpoint doesn’t return anything for us to check against.

- name: DeregisterInstanceFromTargetGroup
    description: Proactively remove the instance from the target group
    action: aws:executeAwsApi
    inputs:
      Service: elbv2
      Api: DeregisterTargets
      TargetGroupArn: "{{ TargetGroupArn }}"
      Targets:
        - Id: "{{ InstanceId }}"

Once we’ve done that we need to verify the instance has definitely been removed. Remember that the target group allows for existing connections to complete their requests when it is draining, so deregistering the instance doesn’t happen instantaneously - this is where the aws:waitForAwsResourceProperty helps us.

- name: WaitForDeregisteredTarget
  description: Wait for the instance to drain connections
  action: aws:waitForAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - unused
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"
  maxAttempts: 1
  timeoutSeconds: 600
- name: AssertTargetIsDeregistered
  description: Assert the instance is no longer a target
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions[0].TargetHealth.State
    DesiredValues:
      - unused
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"

Execute maintenance

At this point we’re 100% sure that the instance is now removed from the target group and is no longer receiving requests, so let’s go ahead and use aws:executeAutomation to invoke the maintenance document from earlier. Remember it takes in the InstanceIds as a parameter to execute on, so we’ll need to pass it there too.

We’re specifying an onFailure too, this tells the document should the step fail then move onto this step instead of the default action which is to abort the rest of the document.

step:RegisterTarget is actually the next step after this one, which adds the instance back to the target group. Since it performs health checks for us and won’t actually forward to an unhealthy instance, we’ll let the target group make the call if this instance can receive traffic or not.

- name: RebootWithHealthcheck
  description: Reboot the instance with a healthcheck afterward
  action: aws:executeAutomation
  inputs:
    DocumentName: RebootWithHealthcheck
    RuntimeParameters:
      InstanceIds:
        - "{{ InstanceId }}"
  maxAttempts: 1
  onFailure: step:RegisterTarget

Add instance back to target group

In a similar vein to DeregisterTargets, this action will register the instance back to the target group.

- name: RegisterTarget
  description: Add the instance back as a target
  action: aws:executeAwsApi
  inputs:
    Service: elbv2
    Api: RegisterTargets
    TargetGroupArn: "{{ TargetGroupArn }}"
    Targets:
      - Id: "{{ InstanceId }}"

Registering the instance happens instantaneously, but we will have to wait for the target group to perform initial health checks against the instance. Once we’ve asserted that it’s healthy, then the document is complete!

- name: WaitForHealthyTargetGroup
  description: Wait for the target group to become healthy again
  action: aws:waitForAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 1
- name: AssertTargetGroupHealthAfter
  description: Assert the target group is healthy after activity
  action: aws:assertAwsResourceProperty
  inputs:
    Service: elbv2
    Api: DescribeTargetHealth
    PropertySelector: $.TargetHealthDescriptions..TargetHealth.State
    DesiredValues:
      - healthy
    TargetGroupArn: "{{ TargetGroupArn }}"
  maxAttempts: 3
  timeoutSeconds: 60
  isEnd: true

Remember that this document only handles one instance at a time, it will typically be up to the caller (i.e. a maintenance window) to rate limit the execution of multiple instances one at a time. We explored this in the previous post.

Terraform additions and updates

The above document can be represented in Terraform to provision it.

resource "aws_ssm_document" "graceful_reboot_instance" {
  name            = "RebootInstanceGraceful"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/graceful_patch_instance.yml",
    {
      reboot_with_healthcheck_document_arn = aws_ssm_document.reboot_with_healthcheck.arn,
    }
  )
}

We’ll also need to update our maintenance window task to correctly reflect this new document, along with the new parameters it takes.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "AUTOMATION"
  task_arn         = aws_ssm_document.graceful_reboot_instance.arn
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors      = "0"

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name = "InstanceId"
        values = ["{{ TARGET_ID }}"]
      }

      parameter {
        name = "TargetGroupArn"
        values = [module.hello_world_alb.target_group_arns[0]]
      }
    }
  }
}

And lastly, we’ll need to update the IAM role permissions for aws_iam_role.patch_mw_role.arn as it will be invoking more actions.

data "aws_iam_policy_document" "mw_role_additional" {
  statement {
    sid    = "AllowSSM"
    effect = "Allow"

    actions = [
      "ssm:DescribeInstanceInformation",
      "ssm:ListCommandInvocations",
    ]

    resources = ["*"]
  }

  statement {
    sid    = "AllowElBRead"
    effect = "Allow"

    actions = [
      "elasticloadbalancing:DescribeTargetHealth",
    ]

    resources = ["*"]

  }

  statement {
    sid = "AllowELBWrite"
    effect = "Allow"

    actions = [
      "elasticloadbalancing:DeregisterTargets",
      "elasticloadbalancing:RegisterTargets",
    ]

    resources = [module.hello_world_alb.target_group_arns[0]]
  }
}

Testing the new document

You can test the automation document by following the same process as before, else you can test the whole stack via changing the execution time of the maintenance window. I’ll be following along with the latter.

While the document is running you can re-use the same command to hit the ALB endpoint from earlier to see how traffic is distributed amongst the instances. You’ll first see that it will only execute on one instance at a time, which was the enhancement we introduced in the last post.

When we drill down into each invocation we can see the automation steps doing their magic.

We can have a look back at the ALB metrics again to see if we received any errors.

No error - no problem!

Conclusion

Each post prior to this one in the series has been leading up to where we are now - a means of achieving automated zero downtime maintenance for anything that sits behind an AWS ALB.

SSM is very much a bit of a beast and I hope that this series has helped clear the fog and given yourselves an idea of what you can do with SSM to automate a variety of tasks in your AWS estate.

Happy automating! 💪 🙌

Who Goes Blogging 7: Hugo Minify RSS Code Indentation Fix

Mon, 14 Dec 2020 00:00:00 +0000

UPDATE 2021-12-15

The minify config was incorrect at time of publication, and so it’s been corrected so that minification does happen while not impacting XML files.

Thanks to Andrea for pointing this out to me!

It’s certainly been a while since the previous post in this series, which has become the home of any updates I make to my Hugo website.

This post is a quick one but it’s something I’ve been meaning to fix for a while. The RSS feeds that are generated will also include code blocks as defined through code fences (```) or through {{< highlight >}} shortcodes in your post content.

However for some reason the code blocks generated in my RSS feeds were losing their indentation. Take this code snippet example from my latest post.

resource "aws_ssm_document" "patch_with_healthcheck" {
name = "PatchWithHealthcheck"
document_type = "Automation"
document_format = "YAML"
content = templatefile(
"documents/patch_with_healthcheck_template.yml",
{
healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
output_s3_bucket_name = aws_s3_bucket.script_bucket.id,
output_s3_key_prefix = "ssm_output/",
}
)
}

Whereas it should be rendering as:

resource "aws_ssm_document" "patch_with_healthcheck" {
  name            = "PatchWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/patch_with_healthcheck_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

This is happening during the GitHub Action that builds the website - I had previously been using the --minify flag which follows this configuration by default.

What’s happening is the RSS XML that Hugo generates for us is then being minified to remove the whitespace generated in that file. This includes the whitespace that’s used to indent the code in the RSS feeds. Not. Good.

On original publication I thought setting minify.tdewolff.xml.keepWhitespace = true would be enough. On closer reading, it does not have the desired effect.

The Fix

In order to fix this we need to add in the below to the site’s config. Then we can remove the --minify flag from our build script, since minification is enabled via our config now.

[minify]
  disableXML = true
  minifyOutput = true

I’m using toml for my config, so make sure you change it to what your config is defined in (yaml / json)

Hope this help you with fixing your RSS feeds too!

Automate Instance Hygiene with AWS SSM: Automation Documents

Fri, 04 Dec 2020 00:00:00 +0000

In part two of this series we looked at how we can automate SSM command documents using SSM Maintenance Windows.

This part will now explore another type of SSM Document; Automation.

tl;dr

Automation documents can allow us to combine command documents together
With this, we can utilise maintenance window error thresholds to stop further invocations
Dynamic invocation of command documents can also be achieved with automation documents

Prerequisites

All the code for this post can be found on GitHub.

You’ll notice that we have changed some things around in this post, so if you’ve been using terraform apply in other posts to deploy to your AWS environment, you will notice some destructions.

Instead of having 1 Windows and 1 Linux EC2 instance, we’re now using 3 Linux EC2 instances - to emulate an application running across multiple instances for redundancy. You don’t actually need anything to be running on these instances, just have them visible in the Managed Instances console.

Note that the EC2 instances are tagged with the key App and value HelloWorld - we’ll be using this to specify our automation document targets.

Automation Documents

Back in part one I gave a brief intro to automation documents. To save the click:

Automation document can call and orchestrate AWS API endpoints on your behalf, including executing Command documents on instances

Essentially we can combine two command documents into one with an automation document. But why would we want to do this?

Introducing proactive healthchecks

Well in the last post, we set up a maintenance window with two tasks; one for invoking AWS-RunPatchBaseline and another for PerformHealthcheckS3 (our healthcheck SSM Document) - both of these are command documents. Say if we had a policy that wanted to ensure that after AWS-RunPatchBaseline was invoked, we would always want the PerformHealthcheckS3 invoked afterward… the Automation Document would help us get there.

Not only that, the way that our maintenance window is currently structured is it will invoke AWS-RunPatchBaseline across all instances in scope at the same time. Once they are all done then it will invoke PerformHealthcheckS3 across all instances at the same time. This looks like this:

Given we have 2 instances; i-111, i-222

T+0: Invoke AWS-RunPatchBaseline on i-111, i-222
T+1: AWS-RunPatchBaseline finishes: i-111, i-222
T+2: Invoke PerformHealthcheckS3 on i-111, i-222
T+3: PerformHealthcheckS3 finishes: i-111, i-222

Say if you wanted to limit the rate of patching across these instances so that only one instance at a time was patched, and any healthcheck failures aborted the rest of patching, then simply changing max_concurrency from 100% to 1 for each maintenance window task will not achieve this.

Maintenance windows complete one task across all instances in scope before moving onto the next task. If we have Task 1 for Patching and Task 2 for Healthchecking (is that even a word?), then the maintenance window is going to patch all instances first before it performs healthchecks on the instances. There is no way to execute the tasks synchronously on one instance at a time.

This means that if a bad patch were to be installed in your estate, you could have this order of events:

Given we have 2 instances; i-111, i-222

T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke AWS-RunPatchBaseline on i-222
T+3: AWS-RunPatchBaseline finishes: i-222
T+4: Invoke PerformHealthcheckS3 on i-111
T+5: PerformHealthcheckS3 FAILS: i-111
T+6: Invoke PerformHealthcheckS3 on i-222
T+7: PerformHealthcheckS3 FAILS: i-222

Now the healthcheck has failed for i-222 as well because the bad patch landed on both instances, and production now has an outage.

Solution

Thankfully, Automation Documents help us avoid that - by combining the two command documents (AWS-RunPatchBaseline and PerformHealthcheckS3), we can mark this new automation document as a solo maintenance window task and have it invoked one at a time on instances, and have it abort further invocations if any sub-documents failed within in:

Given we have 2 instances; i-111, i-222

T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke PerformHealthcheckS3 on i-111
T+3: PerformHealthcheckS3 FAILS: i-111
T+4: Abort invoking AWS-RunPatchBaseline on i-222
T+5: Abort invoking PerformHealthcheckS3 on i-222

Notice how the failed healthcheck on the first instance caused the rest of task invocations to abort? Here is the order of events for a happy path.

Given we have 2 instances; i-111, i-222

T+0: Invoke AWS-RunPatchBaseline on i-111
T+1: AWS-RunPatchBaseline finishes: i-111
T+2: Invoke PerformHealthcheckS3 on i-111
T+3: PerformHealthcheckS3 succeeds: i-111
T+4: Invoke AWS-RunPatchBaseline on i-222
T+5: AWS-RunPatchBaseline finishes: i-222
T+6: Invoke PerformHealthcheckS3 on i-222
T+7: PerformHealthcheckS3 succeeds: i-222

So we know that automation documents allow us to combine command documents together - this is done using the aws:runCommand action, though there are many more actions available to you, some of which we’ll explore later.

Combining command docs into automation

We combine command documents as below in YAML; like command documents they can also be defined in JSON.

For the first time we are defining a parameter called InstanceIds, which takes in a list of instance IDs to then pass down to the command documents, as they will need to know what instances to invoke the commands on. The value assigned to this parameter is retrieved back with the notation "{{ InstanceIds }}", which you can see being passed into the inputs of the sub-documents.

InstanceIds is a special parameter name that AWS recognises and so will provide an instance picker in the GUI of Execute Automation, as shown below.

The instance picker can be helpful if you only want to target a subset of instances in your estate.

We’re also following logging best practices by having the command output logged to S3.

Other than that, there’s not really a whole lot of difference between this and a command document, so far!

It’s important to note the schemaVersion must be 0.3 for automation documents.

---
schemaVersion: "0.3"
description: Executes a patching event on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
mainSteps:
  - name: InvokePatchEvent
    action: aws:runCommand
    inputs:
      DocumentName: AWS-RunPatchBaseline
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: jdheyburn-scripts
      OutputS3KeyPrefix: ssm_output/
      Parameters:
        Operation: Scan
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: PerformHealthcheckS3
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: jdheyburn-scripts
      OutputS3KeyPrefix: ssm_output/

Terraform automation documents

We can deploy these to AWS using Terraform once again. Note that the document_type is Automation and that we’re using templating to set the variables in the document, such as referencing the PerformHealthcheckS3 command document ARN.

You can see the templated version of the document in GitHub.

resource "aws_ssm_document" "patch_with_healthcheck" {
  name            = "PatchWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/patch_with_healthcheck_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

View the above resource in GitHub.

Testing automation documents

Now let’s go and test this by manually invoking it. This can be done by navigating to Automation within Systems Manager and clicking Execute Automation.

For a shortcut of invoking the commands, you can use the below command to invoke the below CLI command. Then you may skip to the results.

aws ssm start-automation-execution \
  --document-name "PatchWithHealthcheck" \
  --document-version "\$DEFAULT" \
  --target-parameter-name InstanceIds \
  --max-errors "0" \
  --max-concurrency "1" \
  --region eu-west-1

Setup

Navigate to the Owned by me tab and select the name of your created document, PatchWithHealthcheck, then click Next.

Because we want to test this document executing one at a time on an instance, we’ll need to select the Rate control option.

We’ll need to select what instances are our targets. The instances in scope for this document are tagged with the key App with the value HelloWorld, so let’s use that as our criteria. Note this is the most scalable solution for targeting instances.

Then we need to specify how the rate of invocation should be controlled. Our criteria for this is:

Execute on 1 instance at a time
Abort further invocations if any produce a error

Therefore we need to set the concurrency to 1 and the error threshold to 0.

Once all done then click Execute.

Results

As the execution progresses you’ll notice it invokes the document on one instance at a time. As it completes you’ll have a screen that looks like the below as you click onto the execution detail page. Notice that the start and end times of each instance invocation do not overlap with one another.

Step name is the same as the instance ID in this case

We can dive into each step invocation (the blue text in the above screenshot) to view the commands that were invoked.

At this detail we can see the individual aws:runCommand actions performed on the instance.

Failure testing

Okay so we’ve confirmed the document now only invokes synchronously. Let’s now test to see if further invocations are aborted when there is a failure.

To simulate the failure, we can borrow a trick from the first post in this series by flipping the healthcheck script from > to <. Once that change is deployed we can re-run the document using the same method as above.

We can see the failure, and that the automation did not invoke on more instances! We can drill down into the invocation to see why it failed.

From this page you can then continue to drill down to the run command output to determine the cause of the failure.

Configuring automation tasks for maintenance windows

Now that we’ve tested the automation document and we’re happy to have it automated, let’s get this added to a maintenance window task. We can reuse the same maintenance window as we created last time, but with some differences.

Maintenance window task

Since we are targeting an automation document, there are some differences we need to account for when compared to command tasks.

Specify the task_type as AUTOMATION
Update the task_arn to our new document accordingly
Ensure our new combined document is only invoked on one instance at a time, so max_concurrency is set to 1
Within task_invocation_parameters we use automation_parameters as opposed to run_command_parameters.
- document_version allows us to target a specific document version
- any parameters required by the document are defined within parameter

Remember that our document takes in InstanceIds as a parameter? Well you’ll notice that the value is set to "{{ TARGET_ID }}". This is known in AWS as a pseudo parameter, whereby the instance ID returned by WindowTargetIds will be passed into the automation document.

Depending on what resource_type your aws_ssm_maintenance_window_target is set up as will result in a different value to {{ TARGET_ID }} - in our case ours is INSTANCE, so this becomes the instance ID.

See the AWS docs for a full breakdown.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "AUTOMATION"
  task_arn         = aws_ssm_document.patch_with_healthcheck.arn
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors      = "0"

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name   = "InstanceIds"
        values = ["{{ TARGET_ID }}"]
      }
    }
  }
}

You can view the above resource in GitHub.

Maintenance window target

We need to update the target to use tag lookups against the instances - this mimics how we tested our automation document earlier on.

resource "aws_ssm_maintenance_window_target" "patch_with_healthcheck_target" {
  window_id     = aws_ssm_maintenance_window.patch_with_healthcheck.id
  name          = "PatchWithHealthcheckTargets"
  description   = "All instances that should be patched with a healthcheck after"
  resource_type = "INSTANCE"

  targets {
    key    = "tag:App"
    values = ["HelloWorld"]
  }
}

You can view the above resource in GitHub.

Additional IAM policies

We’ll need to also attach some new permissions to the IAM role for the maintenance window, aws_iam_role.patch_mw_role.arn, to allow the automation document to perform a lookup on instances by their tag as defined in the updated aws_ssm_maintenance_window_target.patch_with_healthcheck_target resource.

data "aws_iam_policy_document" "mw_role_additional" {
  statement {
    sid    = "AllowSSM"
    effect = "Allow"

    actions = [
      "ssm:DescribeInstanceInformation",
      "ssm:ListCommandInvocations",
    ]

    resources = ["*"]
  }
}

resource "aws_iam_policy" "mw_role_add" {
  name        = "MwRoleAdd"
  description = "Additonal permissions needed for MW"

  policy = data.aws_iam_policy_document.mw_role_additional.json
}

resource "aws_iam_role_policy_attachment" "mw_role_add" {
  role       = aws_iam_role.patch_mw_role.name
  policy_arn = aws_iam_policy.mw_role_add.arn
}

You can view the above resources in GitHub.

Simply put, we’re creating an new IAM policy with anything additional that the maintenance window requires for it to operate. We’ll use this policy to add any new actions if we need to in the future.

Testing automation documents in maintenance windows

Once you’ve got the config above applied you’ll need to run a test. Just like last time, you can do this by changing the maintenance window execution time to something relatively close to your current time.

Once the execution is complete (and hopefully it was successful, if not then check back at the configuration), you’ll see that there will be 3 task invocations, one for each instance, and that none of them overlapped one another.

Notice how the start and end times don’t overlap

Just like for command documents, you can view the detail for each invocation made on the instance. Here we can see the two steps that make up our newly created command document.

But the whole point of this exercise was to proactively handle errors right? So let’s introduce some by doing what we’ve done before and change the healthcheck script to fail.

Once the new “broken” script is applied we can rerun another test of the maintenance window.

This time round we can see that there was a failure in one task invocation, which then aborted further invocations on the remaining instances! Just as before, we can do a deep dive into the invocation to determine why it had failed.

From here you can follow the command invoked by the step to troubleshoot the failure

Bonus: Using automation to dynamically invoke command documents

In this post we’ve looked at a common maintenance task in the form of an SSM command document called AWS-RunPatchBaseline, and created an automation document that will always invoke our healthcheck script after this invocation.

You may have more command documents that perform some form of maintenance on instances, for which you would also want the healthcheck script to execute after as well.

Instead of copying and pasting these automation documents, we can create just one automation document which takes in the command document ARN as a parameter, dynamically invoke it, and then have a hardcoded step afterward for executing a healthcheck!

In it’s raw YAML form, we would get a document that looks like this:

---
schemaVersion: "0.3"
description: Executes a maintenance event on the instance followed by a healthcheck
parameters:
  InstanceIds:
    type: StringList
    description: The instance to target
  DocumentArn:
    type: String
    description: The document arn to invoke
  InputParameters:
    type: StringMap
    description: Parameters that should be passed to the document specified in DocumentArn
    default: {}
mainSteps:
  - name: InvokeMaintenanceEvent
    action: aws:runCommand
    inputs:
      DocumentName: "{{ DocumentArn }}"
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}
      Parameters: "{{ InputParameters }}"
  - name: ExecuteHealthcheck
    action: aws:runCommand
    inputs:
      DocumentName: ${healthcheck_document_arn}
      InstanceIds: "{{ InstanceIds }}"
      OutputS3BucketName: ${output_s3_bucket_name}
      OutputS3KeyPrefix: ${output_s3_key_prefix}

Just like what we’ve done for InstanceIds, we’re taking in the DocumentArn as a parameter and providing it as the input for the aws:runCommand step. Some documents will also take parameters, so we can allow the caller to specify them using InputParameters, which is defined as a StringMap type - allowing it to then be used in as parameters for the document we are invoking.

When we create this document in the console and then execute it, we can then dynamically add in the document we want to invoke.

Then just like any other document we can invoke it.

We can see that the input from the previous screen was passed to the step. Let’s keep going until we hit the command invocation.

The document name was dynamically passed to the command execution

Terraforming dynamic command documents

So now we’ve confirmed that documents can be dynamically invoked, let’s get this Terraformed. You can view this in GitHub.

resource "aws_ssm_document" "maintenance_wrapper" {
  name            = "MaintenanceWithHealthcheck"
  document_type   = "Automation"
  document_format = "YAML"

  content = templatefile(
    "documents/maintenance_wrapper_template.yml",
    {
      healthcheck_document_arn = aws_ssm_document.perform_healthcheck_s3.arn,
      output_s3_bucket_name    = aws_s3_bucket.script_bucket.id,
      output_s3_key_prefix     = "ssm_output/",
    }
  )
}

Not a lot has really changed in this when we compare it to our one from earlier, but the difference is in the parameters field for the document.

And then you can include it as a maintenance window task as below; I’m reusing the same maintenance window task as before.

resource "aws_ssm_maintenance_window_task" "patch_with_healthcheck" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "AUTOMATION"
  task_arn         = aws_ssm_document.maintenance_wrapper.arn
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "1"
  max_errors      = "0"

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    automation_parameters {
      document_version = "$LATEST"

      parameter {
        name   = "InstanceIds"
        values = ["{{ TARGET_ID }}"]
      }

      parameter {
        name   = "DocumentArn"
        values = ["AWS-RunPatchBaseline"]
      }

      parameter {
        name   = "InputParameters"
        values = [jsonencode({ Operation = "Scan" })]
      }
    }
  }
}

You can then copy and paste the same task resource, only changing the values for the InputParameters and DocumentArn parameters accordingly. If the document you are calling doesn’t take any parameters, then you can just omit that block.

You’ll have to ensure that the IAM role the maintenance window task is assuming has the correct IAM permissions as required by the document you’re calling.

Conclusion

What we’ve done in this post is taken our rudimentary command document, prone to introducing errors into our estate, and converted it to an automation document. With the right SSM maintenance window settings, you can ensure that any maintenance tasks you need to perform on your EC2 instances are done so in a manner that reduces the risk of errors in your environment.

Next time, we’ll be taking this a step further to proactively remove EC2 instances from circulation when behind a load balancer for maintenance activities.

Automate Instance Hygiene with AWS SSM: Maintenance Windows

Mon, 16 Nov 2020 00:00:00 +0000

Last time we looked at writing our own SSM Command Document for the purpose of executing a healthcheck script on a set of EC2 instances across multiple platforms.

In this post we’ll be exploring how we can automate this using maintenance windows - also within the SSM suite. This is something I’ve covered before, but want to extend on that to show how its done.

tl;dr

We can use SSM Maintenance Windows to automate our newly created command documents on a schedule
Multiple command documents can be combined in a maintenance window, such as a patching event followed by a healthcheck
This provides us with a means of viewing historical invocations on whatever workflow we’ve automated
By storing command outputs to S3, we can ensure we can recover logs that are too large to display in the console
Using S3 Lifecycle Rules we can remove aged logs

Once again all the Terraform code for this post is available on GitHub. It is split into two parts:

aws-ssm-automation-1-barebones is for the barebones walkthrough
aws-ssm-automation-1-logging is for the logging enhancement

Intro to Maintenance Windows

Maintenance Windows, are a means of executing some automation workflow in your AWS estate on a schedule. Got an SSM Document you’ve written and want it automated? What about a Lambda you want invoked at a regular schedule? Or maybe it’s a Step Function? Whatever the use case, Maintenance Windows are for you - just don’t be fooled by the name - they don’t necessarily have to be just for maintenance!

Similarities to EventBridge Rules

“But wait”, I hear you ask, “don’t CloudWatch/EventBridge Rules also allow you to invoke events on a schedule too?”

Yes they do - both Maintenance Windows and EventBridge Rules (the bigger sibling of CloudWatch Rules) use cron expressions to define the schedule they should run on. The primary difference between the two is that Maintenance Windows allow you to specify the timezone that the cron expression adheres to, whereas EventBridge is tied to UTC.

No timezone, no party

So using maintenance windows can be handy if you’re in a non-UTC timezone and you don’t have to constantly convert your local timezone to UTC to schedule events. More importantly, maintenance windows will respect daylight savings time (DST) if your timezone observes it, so you can be sure your automation will be invoked at the same time in the specified timezone throughout the year.

On the other hand, EventBridge Rules are fixed to UTC; meaning if your timezone does observe DST, then you’ll find your automation could be off by an hour for some portion of the year (unless you change it of course - but who wants to be changing automation twice a year??).

I don’t think I’ve ever met a software engineer that’s a fan of DST!

Notably as well, you can view the execution history of maintenance windows as they’ve occurred in the past, allowing you to quickly see whether a particular invocation was successful or not - and drill down into any failures.

I’m not bashing EventBridge Rules, in fact, it is easier to set up than Maintenance Windows. But there’s always the right tool for the job.

For the rest of this post, we’re going to be exploring how to automate the command document we created last time. Later on in the series we’ll be looking at using maintenance windows to automate automation documents.

Automating command documents with maintenance windows

So what’s the purpose of creating the command document we achieved last time? Well we’re not going to be manually invoking it like what we have been doing so far - as engineers we need to be automating as many repetitive tasks as possible.

To summarise where we are now, we’ve produced a Command document which when executed, automates the following:

Downloads a healthcheck script from S3
Executes the healthcheck script, failing the command invocation if healthcheck does not pass

Healthchecks are important to run both continuously in our environment, as a means of monitoring and verifying the estate is working as intended, before your users notice. They are also necessary to run after a change has been introduced to the environment, such as a new code deployment, or even a patching event via the AWS-RunPatchBaseline document.

Barebones maintenance window with AWS-RunPatchBaseline

We’re going to use Terraform again to build out a minimal maintenance window. You can view the code for it here. You’ll notice the repository this file sits in is identical to the one from the first post, except I’ve included the new resources that will enable us to accomplish the requirement.

Here is a breakdown of each of the resources we’re going to create.

Maintenance Window

This creates the maintenance window resource, which is then referred to in the subsequent resources we create.

It’s nothing more than that cron expression I mentioned earlier, along with the timezone it should execute in
We specify how long the window lasts for, and the cutoff; both of which are specified in hours
- The cutoff indicates how long before the end of the window should AWS not schedule any new tasks in that window

resource "aws_ssm_maintenance_window" "patch_with_healthcheck" {
  name              = "PatchWithHealthcheck"
  description       = "Daily patch event with a healthcheck afterward"
  schedule          = "cron(0 9 ? * * *)" # Everyday at 9am UK time
  schedule_timezone = "Europe/London"
  duration          = 3
  cutoff            = 1
}

Maintenance Window Target

We need a means of telling the window what instances to target, and the aws_ssm_maintenance_window_target resource is how you do it. Below I’m demonstrating two methods of doing this:

Specify the instance IDs directly
- Handy if you have a fixed list of instances you only want to be included in the maintenance window
Target instances by their tags
- This is much for scalable, and means you don’t have to keep adding instance IDs to the list
- When the maintenance window executes, it will filter instances with this tag key and value combo for what to target

An optimum tag to use would be Patch Group described here - which I have mentioned previously. For the sake of this demo, we will keep it simple by targeting instance IDs.

resource "aws_ssm_maintenance_window_target" "patch_with_healthcheck_target" {
  window_id     = aws_ssm_maintenance_window.patch_with_healthcheck.id
  name          = "PatchWithHealthcheckTargets"
  description   = "All instances that should be patched with a healthcheck after"
  resource_type = "INSTANCE"

  targets {
    key = "InstanceIds"
    values = concat(
      module.windows_ec2.id,
      module.linux_ec2.id
    )
  }

  # Using tags is more scalable
  #   targets {
  #     key    = "tag:Terraform"
  #     values = ["true"]
  #   }
}

Maintenance Window Tasks

Patching task

Now for the tasks… remember that we want to execute our healthcheck SSM document after a patch event right? We need to build a task for executing the AWS-RunPatchBaseline document.

resource "aws_ssm_maintenance_window_task" "patch_task" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "RUN_COMMAND"
  task_arn         = "AWS-RunPatchBaseline"
  priority         = 10
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "100%"
  max_errors      = 0

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    run_command_parameters {
      timeout_seconds  = 3600

      parameter {
        name = "Operation"
        values = ["Scan"]
      }
    }
  }
}

Let’s run through the main attributes:

window_id - the maintenance window to associate this task with
task_type - what kind of task this is
- since we’re executing a command document, the value here is RUN_COMMAND
task_arn - the ARN of the document you wish to run
- note the document name can also be used here, as demonstrated above
priority - defines in what order should tasks be executed in, whereby the lower the number given, the earlier the task is executed in the window
- e.g. a task priority of 1 gets executed before one with 10
- tasks with the same priority get executed in parallel
service_role_arn - tells which IAM role should be assumed to execute this task as
- we’ll get an explanation of this later
max_concurrency - specifies how many instances this task should be invoked on simultaenously
- it can take a percentage as a value, only applying to that percentage of target instances at a time
  - i.e. 50% indicates only half of targeted instances will have the task executed at a time
- or it can take a fixed number such as 1; indicating that only one instance should have this task executed at a time
- 100% indicates this task will be invoked on all targets at the same time
max_errors - indicates how many errors should be thrown before we abort further invocations
- 0 indicates any error will abort the maintenance window and set its result to failed

The targets block allows us to define what instances to target this on - we’re referencing the aws_ssm_maintenance_window_target resource we created previously.

Lastly the task_invocation_parameters allows us to customise how the document should be ran via the parameter setting - which is passed to the document. For this example we’re only performing the Scan operation on the document, for testing purposes.

Scan will only check for missing patches - it won’t actually install them.

A full list of available commands can be found in the AWS-RunPatchBaseline command document.

In the console GUI we can see the available parameters for the AWS-RunPatchBaseline document

Healthcheck task

Next we want to define the healthcheck task.

resource "aws_ssm_maintenance_window_task" "healthcheck_task" {
  window_id        = aws_ssm_maintenance_window.patch_with_healthcheck.id
  task_type        = "RUN_COMMAND"
  task_arn         = aws_ssm_document.perform_healthcheck_s3.arn
  priority         = 20
  service_role_arn = aws_iam_role.patch_mw_role.arn

  max_concurrency = "100%"
  max_errors      = 0

  targets {
    key    = "WindowTargetIds"
    values = [aws_ssm_maintenance_window_target.patch_with_healthcheck_target.id]
  }

  task_invocation_parameters {
    run_command_parameters {
      timeout_seconds  = 600
    }
  }
}

There’s not a whole lot of difference here compared to the patching task; our healthcheck script takes no parameters so we can leave them out (although if yours does, you’ll need to add it here!), and the task_arn points to the command document we created last time.

Probably the most significant change though is the priority. Remember that the priority number indicates the ordering of tasks to be invoked? Our patching task had a priority of 10, whereby our healthcheck task is 20. This means the patch task will be invoked before the healthcheck one.

I could have set the priority of the patching and healthcheck tasks to 1 and 2 respectively to achieve the same thing.

However, giving some distance between them means you can programmatically add new tasks before/after each other.

Want a post-patch, pre-healthcheck task? Attach a new task with priority 15!

IAM role for maintenance window

We’ve been referencing aws_iam_role.patch_mw_role.arn as our task service_role_arn. You can view the code for it here - but let’s run through them quickly.

All we’re doing is creating an IAM role, allowing the EC2 and SSM AWS services to assume said role, and applying the predefined AWS policy AmazonSSMMaintenanceWindowRole to that role. This policy gives some basic permissions to the role which allow it to execute commands and more on the instances.

data "aws_iam_policy_document" "patch_mw_role_assume" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type = "Service"

      identifiers = [
        "ec2.amazonaws.com",
        "ssm.amazonaws.com",
      ]
    }
  }
}

resource "aws_iam_role" "patch_mw_role" {
  name               = "PatchingMaintWindow"
  assume_role_policy = data.aws_iam_policy_document.patch_mw_role_assume.json
}

data "aws_iam_policy" "ssm_maintenance_window" {
  arn = "arn:aws:iam::aws:policy/service-role/AmazonSSMMaintenanceWindowRole"
}

resource "aws_iam_role_policy_attachment" "patch_mw_role_attach" {
  role       = aws_iam_role.patch_mw_role.name
  policy_arn = data.aws_iam_policy.ssm_maintenance_window.arn
}

Testing the barebones maintenance window

Once we’ve ran terraform apply on all the above, we can test the maintenance window out. Currently we have it set to run at 9am UK time, which may or may not be a long time away - so change it manually in the console to a time not far away from your time now.

Once it’s done executing, you can navigate to the history tab to view the execution.

Let’s hope you have more luck on your first attempts running this than I did!

You can select any execution and drill down into it with the View details button.

You can see that the tasks got executed in the order we defined them in

We can go deeper in the execution details and pull out the result of individual commands by selecting View details on the task invocation.

The maintenance window has redirected us to the same page as when we manually invoked the command documents in the last post

Clicking on one of the instance IDs in the above screenshot will take us to the command output for that instance.

Logging command output to S3

Maintenance windows by default only capture the first 2500 characters of a command output, if your command outputs more than this then it gets truncated. This can be a problem if you have a task failure and need to examine the output for the reason why it failed.

Take the AWS-RunPatchBaseline output on a Linux instance for example. It’s pretty hefty, and so we lose a lot of context on what actually happened:

Method

To combat this, maintenance windows allow you to dump command output to an S3 bucket, so that you can retrieve it later. In the last post we created an S3 bucket to store our SSM scripts (aws_s3_bucket.script_bucket.arn), we can reuse that bucket to store our command logs too.

In order to do this there are some steps we need to take:

The S3 bucket policy needs to permit the EC2 instance role aws_iam_role.vm_base to s3:PutObject on "${aws_s3_bucket.script_bucket.arn}/ssm_output/*"
- ssm_output/ is the directory/prefix in the S3 bucket where we will store the logs
The KMS key used to encrypt objects in the target S3 bucket needs to permit instance role aws_iam_role.vm_base to kms:GenerateDataKey
The instance role aws_iam_role.vm_base needs permissions to do the above on its respective side

You can view the changes required in GitHub:

Once that is done we’ll need to add some new attributes to the maintenance window tasks, telling it where to dump the command output to.

resource "aws_ssm_maintenance_window_task" "task_name" {

  # ... other attributes hidden

  task_invocation_parameters {
    run_command_parameters {
      output_s3_bucket     = aws_s3_bucket.script_bucket.id
      output_s3_key_prefix = "ssm_output/"

      # ... other attributes hidden
    }
  }
}

When you have the new config written, then you can terraform apply and run another test on the maintenance window.

We now get a button that can redirect us to where the logs are stored in S3

If we click on this button, we can view the logs being stored in S3. Follow the path in S3 until you reach the S3 object containing the logs.

Now you can open this using the Object actions button in the top-right hand corner to view the entire logs!

Logging problems

It’s worth noting that SSM does not raise an error if an instance cannot push logs to S3 - the Amazon S3 button will redirect you to an object in S3 that does not exist. So if your logs are not appearing in S3 then ensure you’ve followed the steps above.

An example showing if logs were not successfully uploaded to S3 - if you get this then double check you’ve set everything up right!

Removing old command logs

Now that we have maintenance windows storing our logs in S3, we should ensure we’re maintaining a good level of hygiene by removing old logs - otherwise our S3 bucket is going to store more and more logs, costing us more money.

S3 has a feature called Lifecycle Rules which tells S3 how to handle objects throughout their lifecycle. We can tell it to move old files to a cheaper storage class, archive them to S3 Glacier (AWS’s long-term storage service), or just simply delete them!

Given we’re not exactly sentimental with logs, we can define a policy that will remove any logs older than 3 months (90 days).

This is very easy for us to add, we simply need to make the addition below to our aws_s3_bucket resource.

resource "aws_s3_bucket" "script_bucket" {
  # ... other attributes hidden

  # Remove old SSM command output logs
  lifecycle_rule {
    id      = "RemoveOldSSMOutputLogs"
    enabled = true

    prefix = "ssm_output/"

    expiration {
      days = 90
    }
  }
}

You can add more rules if you like, such as a transition block to move it to cold storage before deleting if you wish. Take a look at the Terraform documentation for the resource for example of this.

Logs older than 90 days certainly do not spark joy…!

Conclusion

With the first two posts of this series, you have enough to be able to create your own automated series of commands that can be executed on your EC2 instances.

You can also use SSM documents to retrieve files from instances - such as log files. Or even use them to update third-party software on the instances.

The next post and thereafter we’ll be exploring the Command Documents sibling; Automation Documents, and exploring how these can further enhance automation to other AWS services beyond EC2 instances.

Automate Instance Hygiene with AWS SSM: Command Documents

Thu, 29 Oct 2020 00:00:00 +0000

It’s been a while since my last post… which could be down to me trying to salvage something out of summer! 😅

In this post I want to talk a little bit more about AWS SSM. This was something I touched on when discussing patch baselines in a previous post, and within there is another service known as Automation. On first glance it may look pretty dull, but once you scratch the surface there are a number of capabilities it can unlock for you. I found a distinct lack of resources on how to write these documents, so this post will aim to help you get started on doing so!

This will be part one of a four part series about SSM Automation. The outline of the posts will be:

Re-introduction to SSM Documents in general, specifically the Command document by writing our own healthcheck document
How to automate command documents with SSM Maintenance Windows
Looking into Automation documents, and integrating them with maintenance windows
Advanced use case for Automation documents

tl;dr

To summarise the key parts of this post:

SSM can be used to automate several parts of your estate
Command documents are a means of executing logically indifferent commands across multiple platforms
How to write a basic command document in Terraform
How to write a verbose command document in Terraform

Prerequisites

This series assumes you have some knowledge of:

Terraform
These AWS services:
- EC2
- IAM
- S3

All source code for this post is available on GitHub, I’ll be referencing it throughout.

SSM Re-primer

I’ve mentioned SSM Documents before; to save you a click:

An SSM Document is essentially an automation script that you can perform on one or more instances at a time, with conditions to apply different sets of scripts depending on the operating system (OS) platform (i.e. Windows / Linux).

In that post I talk about how AWS-RunPatchBaseline is an example of a COMMAND document. Another type of document is known as AUTOMATION. AWS describes them as:

Command
- Uses command documents to run commands
- State Manager uses command documents to apply a configuration
- These actions can be run on one or more targets at any point during the lifecycle of an instance
- Maintenance Windows uses command documents to apply a configuration based on the specified schedule
Automation
- Use automation documents when performing common maintenance and deployment tasks such as creating or updating an Amazon Machine Image (AMI)
- State Manager uses automation documents to apply a configuration
- These actions can be run on one or more targets at any point during the lifecycle of an instance
- Maintenance Windows uses automation documents to perform common maintenance and deployment tasks based on the specified schedule

I like to use the below to differentiate between the two:

Command documents execute scripts on instances
Automation document can call and orchestrate AWS API endpoints on your behalf, including executing Command documents on instances

There are also other types too which are beyond the scope of this series.

SSM Managed Instances

In order to have command documents executed on your instances, they will need to become managed instances. This requires having the below set up correctly on your instances:

the SSM Agent installed on your instance
- done so by default on all Amazon Linux AMIs and Windows AMIs
connectivity from your instances to the following endpoints:
- https://ssm.REGION.amazonaws.com
- https://ssmmessages.REGION.amazonaws.com
- https://ec2messages.REGION.amazonaws.com
the correct IAM permissions applied on your EC2 instance profile
- these are all provided by the AWS IAM policy AmazonSSMManagedInstanceCore
- the Terraform example for this post shows the policy being attached to the EC2 IAM role
Optional: additional policies that may be required based on your use case

If your instances are appearing in the managed instances console then everything is set up correctly; if not then follow the troubleshooting guide.

Command documents

Documents are defined in either JSON or, for better or for worse, YAML. You can also define what OS each command should be executed on. This could be helpful if you wanted a healthcheck script to be executed across all (or a subset of) your instances in one action. As an example, my healthcheck script could be to check to see if the CPU is overloaded.

Constructing a healthcheck script

I could use the below to perform a healthcheck on a Windows box:

$Avg = (Get-WmiObject Win32_Processor | Measure-Object -Property LoadPercentage -Average | Select Average).Average
If ($Avg -gt 90) {
  Throw "Instance is unhealthy - Windows"
}
Write-Output "Instance is healthy - Windows"

And the equivalent for Linux would be:

# Sources:
# https://stackoverflow.com/a/9229580
# https://bits.mdminhazulhaque.io/linux/round-number-in-bash-script.html
avg_cpu=$(grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print int(usage)+1}')
if (( avg_cpu > 90 )); then
  echo "Instance is unhealthy - Linux"
  exit 1
fi
echo "Instance is healthy - Linux"

You’ll notice the scripts output the OS they are running on - this will serve as an explanation for when we run the scripts as a command document later on.

Note these scripts are just rudimentary examples of healthcheck scripts. Your healthcheck script may be checking that services are running ok, whether a task can be performed, etc. For the sake of this example, I’ve decided to do a simple check against the CPU load for demonstration.

Testing in AWS SSM Console

Let’s now test them in the AWS SSM Console. For this example I’ve spun up two EC2 instances, one Linux and one Windows, using Terraform. To run the commands we can navigate to Run Command in the console, and run AWS-RunPowerShellScript and AWS-RunShellScript for both Windows and Linux EC2s respectively. We don’t care about logging the output of the scripts just yet. Make sure when running the script you manually select what instance to run it on.

We can see they have executed fine. Let’s flip the condition in the healthcheck script so we can test them failing (done by changing > to <).

This is cool - in the script we can define what constitutes as a failure and have this propagate up to AWS - notice how the status was Failed. This will come in use later on in the series.

Terraforming command documents

Now let’s get these scripts Terraformed so we can reap the benefits of infrastructure-as-code. First we need to define the document in the YAML format.

# documents/perform_healthcheck.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - name: PerformHealthCheckWindows
    action: aws:runPowerShellScript
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - "$Avg = (Get-WmiObject Win32_Processor | Measure-Object -Property LoadPercentage -Average | Select Average).Average"
        - "If ($Avg -gt 90) {"
        - '  Throw "Instance is unhealthy- Linux"'
        - "}"
        - 'Write-Output "Instance is healthy - Windows"'
  - name: PerformHealthCheckLinux
    action: aws:runShellScript
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - "avg_cpu=$(grep 'cpu ' /proc/stat | awk '{usage=($2+$4)*100/($2+$4+$5)} END {print int(usage)+1}')"
        - "if (( avg_cpu > 90 )); then"
        - '  echo "Instance is unhealthy - Linux"'
        - "  exit 1"
        - "fi"
        - 'echo "Instance is healthy - Linux"'

GitHub URL for the above

Since a document is meant to perform an action (or group of actions) on a set of instances regardless of their operating system (OS), only the steps that apply to the OS platform for the instance they are being executed on will be invoked.

Therefore when we target this document to run on two types of EC2 instances, one Windows and one Linux, the step PerformHealthCheckWindows will be executed on the Windows box and vice versa for PerformHealthCheckLinux. This is because of the precondition key which filters on the platformType for the instance. Notably as well, we are targeting the appropriate actions for each platform; aws:runPowerShellScript for Windows and aws:runShellScript for Linux.

See here for a full list of what actions you can perform in a command document.

Using Terraform you can deploy out the document to your environment like so.

resource "aws_ssm_document" "perform_healthcheck" {
  name            = "PerformHealthcheck"
  document_type   = "Command"
  document_format = "YAML"

  content = file("documents/perform_healthcheck.yml")
}

GitHub URL for the above

Once deployed, we can navigate to System Manager in the AWS Console to invoke the document on our estate. This is done in the same manner as when we ran AWS-RunShellScript above, except now we are targeting PerformHealthcheck. Since our document has commands for both Linux and Windows, we can have it invoked across both platform types and only the scripts written for their platform will be invoked.

We can see that both executed successfully! You can view the output of each command invocation on each instance like previously. For the screenshot below of the Linux instance, only the Linux step was executed, whereas the Windows step was skipped. You’ll notice our message from earlier “ - Linux” is there, reassuring us that only the Linux script was executed.

Let’s dive into some more intermediate documents.

Verbose command documents

The example above was a very basic example of such a document where we only had a few lines of code to execute. The healthcheck script you write for your service may have several more lines of code to execute, and trying to read lines of code in amongst the document markup format is not easy on the eyes. Here is a snippet of the AWS-RunPatchBaseline document as an example of what I mean. It is written in JSON and has over 100 lines in the runCommand section.

{
  // ...
  "mainSteps": [
    {
      "precondition": {
        "StringEquals": ["platformType", "Windows"]
      },
      "action": "aws:runPowerShellScript",
      "name": "PatchWindows",
      "inputs": {
        "timeoutSeconds": 7200,
        "runCommand": [
          "# Check the OS version",
          "if ([Environment]::OSVersion.Version.Major -le 5) {",
          "    Write-Error 'This command is not supported on Windows 2003 or lower.'",
          "    exit -1",
          "} elseif ([Environment]::OSVersion.Version.Major -ge 10) {",
          "    $sku = (Get-CimInstance -ClassName Win32_OperatingSystem).OperatingSystemSKU",
          "    if ($sku -eq 143 -or $sku -eq 144) {",
          "        Write-Host 'This command is not supported on Windows 2016 Nano Server.'",
          "        exit -1",
          "    }",
          "}",
          "# Check the SSM agent version",
          "$ssmAgentService = Get-ItemProperty 'HKLM:SYSTEM\\CurrentControlSet\\Services\\AmazonSSMAgent\\'",
          "if (-not $ssmAgentService -or $ssmAgentService.Version -lt '2.0.533.0') {",
          "    Write-Host 'This command is not supported with SSM Agent version less than 2.0.533.0.'",
          "    exit -1",
          "}",
          ""
          // ...
        ]
      }
    }
  ]
}

You can see the whole thing on AWS.

Even from this snippet it is hard to distinguish what is going on. Losing out on syntax highlighting means the code isn’t readable by any means. And since this is a JSON file format, we lose type hinting for the language we are writing the script in (PowerShell in this case). If the script was saved in as a PowerShell file (.ps1) then we’d get all those benefits.

We can fix all these issues by adopting a common pattern when composing command documents. We can have S3 store the script, then have the command document perform these actions:

Download the script in question from S3
Execute the script from the download location

Such a command document would have a composition as below, whereas this is performing the same healthcheck script previously.

# documents/perform_healthcheck_s3.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - action: aws:downloadContent
    name: DownloadScriptWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/jdheyburn-scripts/ssm_scripts/PerformHealthcheck.ps1"}'
  - action: aws:runPowerShellScript
    name: ExecutePerformHealthCheckWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - ..\downloads\PerformHealthcheck.ps1
  - action: aws:downloadContent
    name: DownloadScriptLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/jdheyburn-scripts/ssm_scripts/perform_healthcheck.sh"}'
  - action: aws:runShellScript
    name: ExecutePerformHealthCheckLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - ../downloads/perform_healthcheck.sh

GitHub URL for the above

Note the new action called aws:downloadContent - which you can view the documentation for here. Again we’re using the precondition key to ensure each platforms downloads their respective script. We’re also using the inputs key to instruct the action where it can download the script from; S3 in this case, at the given S3 bucket location. There is an optional destinationPath field which allows you to change where it downloads to.

Once the script is downloaded to the instance, we will need to have it executed. aws:downloadContent saves the script to a temporary directory for executing SSM command invocations on instances, so we need to reference it in the downloads directory for it; indicated by the ../downloads/perform_healthcheck.sh command.

Terraforming verbose command documents

This involves having your script first uploaded to S3. Thankfully, through the power of infrastructure-as-code, you can have this automatically deployed to your environment once the module is written.

Check the GitHub repository for the full example, for now let me break down what each bit is doing.

KMS

I want my S3 bucket to encrypt objects that are stored there, so we’ll need to create a KMS key and give it an IAM policy permitting any users of the account to decrypt using the key. You can make this more secure by only targeting the IAM roles that require it, instead of the whole account, adopting the principle of least privilege best practice 🔒

You can read up more about IAM policies here.

data "aws_iam_policy_document" "kms_allow_decrypt" {
  statement {
    sid    = "AllowKMSAdministration"
    effect = "Allow"

    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion"
    ]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:user/jdheyburn"]
    }

    resources = ["*"]
  }

  statement {
    sid    = "AllowDecrypt"
    effect = "Allow"

    actions = ["kms:Decrypt"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
      # TIP: For increased security only give decrypt permissions to roles that need it
      # identifiers = [aws_iam_role.vm_base.arn]
    }

    resources = ["*"]
  }
}

resource "aws_kms_key" "script_bucket_key" {
  description = "This key is used to encrypt bucket objects"
  policy      = data.aws_iam_policy_document.kms_allow_decrypt.json
}

S3

Next we’re creating the S3 bucket to store the scripts; we’re encrypting it with the KMS key we created, aws_kms_key.script_bucket_key. We’re also applying an IAM policy on the bucket permitting any users of the account to download anything from the ssm_scripts/ prefix in the bucket (this prefix is where the scripts will be stored).

resource "aws_s3_bucket" "script_bucket" {
  bucket = "jdheyburn-scripts"

  # Encrypt objects stored in S3
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = aws_kms_key.script_bucket_key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

data "aws_caller_identity" "current" {}

data "aws_iam_policy_document" "s3_allow_script_download" {
  statement {
    sid    = "AllowAccountAccess"
    effect = "Allow"

    actions = ["s3:GetObject"]

    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
      # TIP: For increased security only give decrypt permissions to roles that need it
      # identifiers = [aws_iam_role.vm_base.arn]
    }

    resources = ["${aws_s3_bucket.script_bucket.arn}/ssm_scripts/*"]
  }
}

resource "aws_s3_bucket_policy" "script_bucket_policy" {
  bucket = aws_s3_bucket.script_bucket.id
  policy = data.aws_iam_policy_document.s3_allow_script_download.json
}

Store Scripts in S3

Now we are uploading the scripts from their location in the repository to S3, at the prefix where we defined an IAM policy to pull them from.

locals {
  perform_healthcheck_script_fname_windows = "PerformHealthcheck.ps1"
  perform_healthcheck_script_fname_linux   = "perform_healthcheck.sh"
}

resource "aws_s3_bucket_object" "perform_healthcheck_windows" {
  bucket  = aws_s3_bucket.script_bucket.id
  key     = "ssm_scripts/${local.perform_healthcheck_script_fname_windows}"
  content = file("scripts/${local.perform_healthcheck_script_fname_windows}")
}

resource "aws_s3_bucket_object" "perform_healthcheck_linux" {
  bucket  = aws_s3_bucket.script_bucket.id
  key     = "ssm_scripts/${local.perform_healthcheck_script_fname_linux}"
  content = file("scripts/${local.perform_healthcheck_script_fname_linux}")
}

SSM Document

resource "aws_ssm_document" "perform_healthcheck_s3" {
  name            = "PerformHealthcheckS3"
  document_type   = "Command"
  document_format = "YAML"

  content = templatefile(
    "documents/perform_healthcheck_s3_template.yml",
    {
      bucket_name   = aws_s3_bucket.script_bucket.id,
      linux_fname   = local.perform_healthcheck_script_fname_linux,
      linux_key     = aws_s3_bucket_object.perform_healthcheck_linux.id,
      windows_fname = local.perform_healthcheck_script_fname_windows,
      windows_key   = aws_s3_bucket_object.perform_healthcheck_windows.id,
    }
  )
}

Here we’re creating an aws_ssm_document as we had done before, but the document file we’re targeting this time is a template file.

Notice how we have the presence of ${bucket_name} and others? These are template variables. With the use of the Terraform function templatefile(), we can insert Terraform variables into the config to have the template name replaced with the value we’re passing in. In this case, ${bucket_name} will get replaced with the output of aws_s3_bucket.script_bucket.id, which is jdheyburn-scripts, and the same for the remaining variables.

# documents/perform_healthcheck_s3_template.yml
---
schemaVersion: "2.2"
description: Perform a healthcheck on the target instance
mainSteps:
  - action: aws:downloadContent
    name: DownloadScriptWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/${bucket_name}/${windows_key}"}'
  - action: aws:runPowerShellScript
    name: ExecutePerformHealthCheckWindows
    precondition:
      StringEquals:
        - platformType
        - Windows
    inputs:
      runCommand:
        - ..\downloads\${windows_fname}
  - action: aws:downloadContent
    name: DownloadScriptLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      sourceType: S3
      sourceInfo: '{"path":"https://s3.amazonaws.com/${bucket_name}/${linux_key}"}'
  - action: aws:runShellScript
    name: ExecutePerformHealthCheckLinux
    precondition:
      StringEquals:
        - platformType
        - Linux
    inputs:
      runCommand:
        - ../downloads/${linux_fname}

GitHub URL for the above

EC2 IAM Permissions

For the pattern to work, you’ll need to attach an IAM policy to the instance to allow it to pull the scripts from the S3 bucket. If we don’t do this then the aws:downloadContent action will fail. From the Terraform above we’ve already applied the corresponding permissions on the S3 bucket, allowing all users and roles in the account to perform s3:GetObject on the scripts. We’ve also allowed done the same for performing kms:Decrypt on the KMS key that encrypts the S3 objects.

If you’ve been following the GitHub example, these required permissions have already been defined - for a recap:

data "aws_iam_policy_document" "ssm_scripts" {
  statement {
    sid    = "AllowS3"
    effect = "Allow"

    actions = ["s3:GetObject"]

    resources = [
      "${aws_s3_bucket.script_bucket.arn}/ssm_scripts/*",
    ]
  }

  statement {
    sid    = "AllowKMS"
    effect = "Allow"

    actions = ["kms:Decrypt"]

    resources = [aws_kms_key.script_bucket_key.arn]
  }
}

resource "aws_iam_policy" "ssm_scripts" {
  name        = "PullSSMScripts"
  description = "Enables instances to download SSM scripts from S3"

  policy = data.aws_iam_policy_document.ssm_scripts.json
}

resource "aws_iam_role_policy_attachment" "instance_download_scripts" {
  # Change this to point to the role(s) for your instances
  role       = aws_iam_role.vm_base.name
  policy_arn = aws_iam_policy.ssm_scripts.arn
}

This is an important concept to remember about setting IAM policies - you will need to ensure that the correct permissions are applied on both the source of the requestor, as well as the destination.

Testing verbose documents

Now, let’s give this command a spin in the console. We’re going to execute it the same way we did for other documents earlier, except now targeting PerformHealthcheckS3.

If you got any failures, make sure to dive into the failed invocation and see why it failed. Did it fail because of your healthcheck command? Then it is working as intended! Although if it failed on aws:downloadContent, check to make sure your instances are running the latest version of SSM agent. You can do this with the AWS-UpdateSSMAgent SSM document. Don’t be like me and spend hours troubleshooting against an out-of-date SSM agent! 😂

Trying to write my next blog post about AWS SSM, and had a load of hours wiped out trying to troubleshoot an issue with SSM agents... turns out AWS AMIs install these bad agent versions by default 😭https://t.co/mgFMVyB5fL
— Joseph D. Heyburn (@jdheyburn) October 24, 2020

Let’s now dive into the output of the Linux instance.

Just like with the previous document with the script embedded PerformHealthcheck, we can see the steps conditioned for Windows have been skipped (steps 1-2). Step 3 is where the document is doing real work, downloading the Linux script from the S3 location into the temp directory for SSM, and then executing it in step 4.

Conclusion

I think this has been my longest post thus far, and it’s only part one of this series. SSM is a bit of a beast and is often shrugged off as being a pain to set up and troubleshoot. I put that down to it encapsulating several other services, and a lack of tried and tested documented methods - which I hope this series resolves.

We covered in this post:

What SSM Documents are; and the differences between Command and Automation documents
How to create our own command documents to execute the same business logic on differing instance platforms
Adopting best practices by storing scripts in S3 and have a command document orchestrate the download and execution of said scripts
… all while written in Terraform!

If any mistakes were made in this post then please contact me - thanks for reading! 😃

Assertions in gotests Test Generation

Mon, 20 Jul 2020 00:00:00 +0000

I’ve been doing some programming in Go for a side project again, and I’m back using gotests to generate unit tests for functions. For this I’ve been referencing a post I’ve previously written in order to help me get them set up. If you’d like more context on the background I’d recommend reading there first.

Today I’ll be talking about a small enhancement to how the tests are generated to make use of the assert package within testify. Go already comes with enough for you to write tests, but assert provides me with more options for comparison in a natural language form. I’ll also be adding support for when a test case returns an unexpected error.

Don’t care about the waffle? Jump straight to it here.

Recap

From the last time we visited this, our test code took the format below.

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

Although this works nicely, there is one issue - we’re not capturing unexpected errors. Or rather, if an error is returned to err, and tt.wantErr is set to nil, then the test will not fail.

Okay, so we still have the if got != tt.want condition to fail the test if needed. Although we still could have this condition pass, we want to make sure we capture the error. The test suite is doing currently is assuming we don’t care about the outcome of err, just because we didn’t want one as described by tt.wantErr.

I seem to remember assumptions being the ~~brother~~ mother of something…

Enhancing for Unexpected Errors

In order to enhance what we have already from the original modified function.tmpl, we could have it output something like this to capture unexpected errors.

// ... removed for brevity
for _, tt := range tests {
    t.Run(tt.name, func(t *testing.T) {
        got, err := validateDogName(tt.args.name)
        if err != nil && tt.wantErr == nil {
            t.Errorf("validateDogName() unexpected error = %v", err)
            return
        }
        if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
            t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
            return
        }
        if got != tt.want {
            t.Errorf("validateDogName() = %v, want %v", got, tt.want)
        }
    })
}

The highlighted lines show the new addition. This is something we could implement fairly easily. On the other hand, the assert library gives us a lot more to play with. It essentially is doing the same as the above under the hood, albeit in a cleaner fashion… and I’m all for better code readability!

Rewriting for Assert

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
        })
    }
}

The above code block is what we get when we take the test code above that is using the t.Errorf function call to record a test failure, and rewrite it to use assert.

What we now need to do is have gotests generate it for us.

Customising Gotests Generated Test v2

We’ll be following a process similar to when I last did this. I’m still using VSCode, so you’ll need to find the correct plugin for your editor.

Check out gotests and copy the templates directory to a place of your choosing
- git clone https://github.com/cweill/gotests.git
- cp -R gotests/internal/render/templates ~/.vscode/gotests/templates
In order for us to achieve the generated test using assert, this time we’re going to need to edit two files; function.tmpl and results.tmpl
- Overwrite the contents of function.tmpl with the contents of this Gist
- Overwrite the contents of results.tmpl with the contents of this Gist
Add the following setting to VSCode’s settings.json
- "go.generateTestsFlags": ["--template_dir=~/.vscode/gotests/templates"]

Now we can generate tests of functions that return the following:

only returns an error
a value, and an error
multiple values, and an error

I have the Go plugin for VSCode, so I just need to right click over a function to have the dropdown menu appear with an option to generate tests.

Only returns an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
            }
        })
    }
}

Returns a value, and an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
        })
    }
}

Returns multiple values, and an error

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        want1   bool
        wantErr error
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, got1, err := validateDogName(tt.args.name)
            if err != nil && tt.wantErr == nil {
                assert.Fail(t, fmt.Sprintf(
                    "Error not expected but got one:\n"+
                        "error: %q", err),
                )
                return
            }
            if tt.wantErr != nil {
                assert.EqualError(t, err, tt.wantErr.Error())
                return
            }
            assert.Equal(t, tt.want, got)
            assert.Equal(t, tt.want1, got1)
        })
    }
}

Bonus: Only returns values

The below example is for a function that doesn’t produce any errors, but I’m including it for the sake of completeness.

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name string
        args args
        want bool
    }{
        // TODO: Add test cases.
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got := validateDogName(tt.args.name)
            assert.Equal(t, tt.want, got)
        })
    }
}

That’s It!

This is just a short update to an enhancement I previously made to gotests. The assert library is awesome for test cases and it’s great to have it autogenerated in my tests too.

Using Terraform to Manage AWS Patch Baselines at Enterprise Scale

Thu, 02 Jul 2020 00:00:00 +0000

If you’re new to AWS and patching principles then continue reading, else you can skip to juicy stuff below.

AWS Primer ☁️

EC2

Amazon Web Services (AWS) provides Cloud resources to those that require it, for a cost mind you. One resource in particular is a box-standard server known on AWS as EC2 Instances which you can secure shell (SSH) into and do whatever you like with it - the world is now officially your oyster! 🦪

Just like with your personal laptop, computer, mobile phone, etc., you need to patch your servers with the latest security updates to ensure that you are protected from any adversary gaining access to your devices.

If they were to gain unauthorised access, they can execute whatever they like on there; whether that be sniffing around your network for some sensitive files, or mining cryptocurrency on your paid-for resources. Just like you earlier - your world is now officially their oyster!

EC2 instances are operated by you, and as per the AWS Shared Responsibility Model, you are responsible for ensuring the software is kept secured.

SSM & Patch Manager

AWS have a service used to help with the administration of EC2 instances called Systems Manager (SSM*). SSM is a bit of a beast and covers a lot of different functionalitie, going deep on SSM is beyond the scope of this article - so you can read more about it on the AWS documentation.

* SSM formerly was an acronym for Simple Systems Manager - I guess it got more complex than they thought!

To help you keep your instances patched, AWS provided the Patch Manager - within there it contains a number of tools to help with this. These are:

Patch Baselines
- a policy which filters available patches for your instances to what should be installed on it
- filters which you can apply to a baseline include
  - how many days since the patch was released
  - the severity and classification of the patch
  - or even explicitly deny individual patches if you know they introduce a bug
- e.g. only install security classified patches marked as critical severity which have been released more than 7 days ago
Patch Groups
- a label which joins together patch baselines, to what instances they should be applied to
- they apply to instances as a tag at a key named Patch Group

To then perform a patch event on an instance, you will need to execute an SSM Document known as AWS-RunPatchBaseline on your target instances.

An SSM Document is essentially an automation script that you can perform on one or more instances at a time, with conditions to apply different sets of scripts depending on the operating system (OS) platform (i.e. Windows / Linux).

AWS-RunPatchBaseline is one such example of a document that has both Windows and Linux stages, and it will check and install patches against the patch baseline that has been applied to your instance, via a patch group.

However if an EC2 instance is not assigned to a patch group, then AWS will pick the default baseline for that instances operating system

Say you had a security policy of ensuring instances must check for patches once a week - there’s no need to manually execute the SSM Document yourself. Maintenance Windows can help with that. They are essentially a cron expression that you define to then execute a task. In this case we can create a maintenance window to execute the AWS-RunPatchBaseline document on a weekly basis.

The resulting relationship of all the above looks like this:

Maintenance Windows ──┬── invokes ───> SSM Document ─── queries ───> Patch Baseline
                      |
                      |
                      └── targets ───> Patch Groups ─── assigned to ───> EC2 Instances

Infrastructure as Code Primer

Creating your resources through a web UI such as AWS Console is okay for learning in. But how do you ensure your infrastructure is repeatable across several environments? Not only that, how do you easily keep a history of the state of your application infrastructure?

This is where infrastructure as code (IAC) comes in. IAC is where all your infrastructure resources are defined in code, and published onto a source code repository of your choosing (GitHub, GitLab, etc.).

That way you can ensure the code that defines your infrastructure can be repeated across your different application environments, and be able to view history of code changes.

It also acts as a single source of truth that everyone in your team can depend on for what the application state looks like.

There are several IAC tools that you can use:

CloudFormation is AWS’s home-grown solution exclusively for AWS services
There are platform-agnostic tools such as Terraform
- written in its own proprietary language HashiCorp Configuration Language (HCL)
- I’ve talked about Terraform before in a previous post
Or for a language-agnostic, platform-agnostic tool - Pulumi is the one for you.

In the case of Terraform, it keeps the current state of your infrastructure stack in a file stored in a shared remote location. This is known as the state file, and it is used by Terraform to understand what resources Terraform is aware of in your platform.

Patch Baselines in an Enterprise Environment

When managing Terraform in an enterprise environment, it is a best practice to split up the infrastructure on the structure of the teams working on them, known as workspaces. This is something advised by Terraform themselves. For example, you would have a workspace for billing, and one for networking. That way you can ensure teams can work effectively without treading on each others’ toes.

The same principle can be applied to a security team that defines security policies, who will also write up patching policies. These policies will be followed by teams working in different workspaces to understand at minimum what patches should be installed on a server, and how quickly to install them.

Check out the University of Exeter’s own patching policy for an example.

For example, an enterprise may have a patching policy that states all patches with a severity marked as ‘Critical’ or ‘Important’ must be installed within 14 days of its release. If an out-of-band (OOB) patch is released, then it must be installed within 3 days.

These folk are the same lot who will also conjure up the Patch Baselines we learned about earlier.

They may deploy these patch baselines in a different Terraform workspace from yours. If that were the case, then if you wanted to refer back to these in your Terraform workspace, then the remote_state resource is something you might need.

# This example references a subnet_id created in another workspace
data "terraform_remote_state" "vpc" {
  backend = "remote"

  config = {
    organization = "hashicorp"
    workspaces = {
      name = "vpc-prod"
    }
  }
}

resource "aws_instance" "foo" {
  # ...
  subnet_id = data.terraform_remote_state.vpc.outputs.subnet_id
}

So the above works if all parties are using Terraform workspaces. But what if you’re using Terraform, and the security team (who are managing patch baselines) are using something different like the mentioned CloudFormation or Pulumi.

If you are creating patch groups in Terraform - you won’t be able to reference the patch baselines they’ve created, because they are in a different state file.

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = aws_ssm_patch_baseline.front_end_servers.id # Terraform does not know about this resource!
  patch_group = "front_end_servers"
}

You could of course replicate the patch baseline they’ve created in your Terraform code, but then that does not scale in an enterprise.

resource "aws_ssm_patch_baseline" "front_end_servers" {
  name = "front-end-servers"

  ... # variables removed for brevity
}

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = aws_ssm_patch_baseline.front_end_servers.id # Not good!
  patch_group = "front_end_servers"
}

This is what we want to avoid.

Data Sources For Patch Baselines

Data sources in Terraform allow you to pull resources in your platform that may exist in a separate state file to yours, or even a completely different IAC tool to Terraform, such as CloudFormation. What we need is a data source component for the aws_ssm_patch_baseline resource.

With a pull request I made to the terraform-aws-provider project - I added the ability to pull in patch baseline resources that exist in the AWS account you are targeting, meaning if the enterprise security team had deployed a patch baseline via a different stack, then you can reference that in your patch group.

# This resource is defined outside of the current working Terraform state
# So we are making a call to retrieve the ID of the resource in AWS
data "aws_ssm_patch_baseline" "front_end_servers" {
  owner            = "Self"
  name_prefix      = "FrontEndServers"
  operating_system = "CENTOS"
}

resource "aws_ssm_patch_group" "front_end_servers" {
  baseline_id = data.aws_ssm_patch_baseline.front_end_servers.id
  patch_group = "front_end_servers"
}

resource "aws_instance" "front_end_server" {
  ... # variables removed for brevity

  tags = {
    "Patch Group" = aws_ssm_patch_group.front_end_servers.id
  }
}

The documentation for its usage can be found here.

Another scenario might be where you want to reuse the patch baselines that AWS has created in your account. One such baseline is AWS-WindowsPredefinedPatchBaseline-OS-Applications, which patches both the Windows OS, and selected Microsoft applications installed on the Windows server.

Currently this is not the default baseline for Windows. So if you want to have this baseline assigned to a patch group, you could do something like this:

data "aws_ssm_patch_baseline" "windows_predefined_os_and_apps" {
  owner            = "AWS"
  name_prefix      = "AWS-WindowsPredefinedPatchBaseline-OS-Applications"
  operating_system = "WINDOWS"
}

resource "aws_ssm_patch_group" "active_directory" {
  baseline_id = data.aws_ssm_patch_baseline.windows_predefined_os_and_apps.id
  patch_group = "active_directory"
}

resource "aws_instance" "active_directory" {
  ... # variables removed for brevity

  tags = {
    "Patch Group" = aws_ssm_patch_group.active_directory.id
  }
}

Conclusion

This is a relatively short post - but it was just a quick introduction to AWS Patch Manager. I hope you find it helpful in applying your enterprise patching strategy to your product teams!

Reverse Proxy Multiple Domains Using Caddy 2

Tue, 09 Jun 2020 00:00:00 +0000

During lockdown, I’ve spent a bit of time improving our home network. The bigger picture of which I’ll write about in a future post. But for now, I came across some challenges with running Caddy 2 as a reverse proxy for multiple domains used internally.

If you’ve stumbled across this looking for the end config file for Caddy, then you can skip there.

Background

A few months back I kitted out my home with some Ubiquiti UniFi gear to fix our crappy Wifi at home, following inspiration from Troy Hunt and Scott Helme.

In order to administrate UniFi devices, you’ll need the UniFi Cloud Key which runs the Controller software to do just that. Although if you have a spare Raspberry Pi lying around, you can download the software for free and run it on there - this is what I did.

I’ve also wanted to protect my home network with a self-hosted DNS server, such as PiHole. I won’t go into depth about how that was done, but you can follow Scott Helme’s guide on how you can set the same up.

Both of these services can be accessed through web browsers at the IP address and ports where they are being hosted, such as http://192.168.1.10:8093/admin/ in the case of PiHole. Having to remember the IP address and the port can be a pain. We can front these services with a rememberable domain name which points to these services - of which I’ve written about in a previous post.

Securing with HTTPS

The web is evolving, and there is no reason why we should access services via insecure HTTP, that includes services that are only running on an internal network such as a home network. Web browsers nowadays give you a warning when you are connecting to website over an unencrypted connection.

Simply accessing over HTTP is not an option, when browsers present us with a huge warning message

Caddy is a web server similar to Apache, nginx, et al., but it is different in that it enables HTTPS by default and upgrades requests from HTTP to HTTPS. Managing certificates for HTTPS is a pain - so Caddy does that too, so long as you can prove you own the domain you are hosting requests at. We can use Caddy in a reverse proxy mode, allowing us to access services at endpoints such as https://pihole.domain.local in our browsers and forward them to the corresponding IP address hosting the service.

A reverse proxy is a service that simply forwards client requests onto the server on the clients behalf.

Proving Domain Ownership

Caddy uses Let’s Encrypt (LE) to provide certificates for domains. Since domains can be exposed publicly, we will have to prove ownership of the domain to have LE issue certificates on our behalf - so we’ll have to purchase the domain from a registrar. I talked about how to do this for this website in the past.

LE supports several challenge methods in order to prove you own the domain. This helps mitigates attacks by adversaries by claiming they own a domain such as natwest.co.uk - allowing them to create phishing attacks and steal banking information.

Since my network is only visible internally for the moment (i.e. the domain will only resolve to an IP address on my network) - I cannot use HTTP or TLS since these require the domain to resolve to a public IP address to a web server hosting a challenge file requested by LE. Therefore the only option I have is DNS challenge, where a randomly string generated by LE is placed into the TXT record of a DNS record to confirm ownership.

Building Our Caddy

For this exercise I’ll be using the latest version, Caddy 2, which allows for plugins to be built into the binary depending on your use case - including DNS challenge. This plugin isn’t included by default, so we’ll need to build our own Caddy binary. The tool to do this is called xcaddy.

UPDATE 2020-09-30

Looks like Caddy now comes with a nice web interface for downloading a Caddy binary with whatever plugins you desire. I just tested out the Linux arm 7 platform with just the github.com/caddy-dns/cloudflare plugin, and it was able to run my Caddy configuration below perfectly!

Once you’ve got the binary downloaded, copy it to the Pi then skip to Caddy Configuration.

To build using xcaddy, you need to make sure you have Go installed on your machine.

Note that I am building Caddy on my laptop, but running it on a Pi, so I will have to specify the architecture that Pi is running on so that Go can correctly build it.

# Download xcaddy
go get -u github.com/caddyserver/xcaddy/cmd/xcaddy

# Build custom Caddy binary for Raspberry Pi
GOOS=linux GOARCH=arm GOARM=7 xcaddy build --with github.com/caddy-dns/cloudflare

# Copy the new binary across to the Pi
scp caddy pi:/home/pi/caddy/

Caddy Configuration

The configuration I’m using can be seen below. Some things to note:

I’m using Cloudflare as the DNS name servers for the domain, even though I purchased my domain from namecheap
- This repeats an exercise I’ve done previously
- I’ve done this for two reasons:
  - Caddy at the time of writing does not have a namecheap DNS challenge plugin
  - It’s a proven method I know already
A CLOUDFLARE_API_TOKEN is required to have Caddy set the TXT record DNS challenge received from LE
- Guide for the same
Caddy is reverse proxying traffic to services running locally on the Pi
Caddy is not verifying the certificate being hosted by the UniFi Controller (insecure_skip_verify = true)
- The controller self-signs a certificate, and the reverse proxy has no means of establishing a chain of trust to verify the certificate
- It’s not a best practice to not verify the chain of trust, however I’m happy to accept the risk for now

Click here to see documentation on Caddy JSON config files.

Updating DNS Records

Remember that the domain names aren’t actually publicly accessible. At a basic level we can update the /etc/hosts file of the machine we’re running on to add a record telling our machine how to resolve the domain.

sudo sh -c "echo \"192.168.1.10 pihole.joannet.casa\n192.168.1.10 unifi.joannet.casa\" >> /etc/hosts"

However, we’re already using PiHole as our own DNS server right? We can add the records there instead.

PiHole let’s you specify where local domain names should resolve to

The IP addresses you see above are pointing to the host running Caddy, the Raspberry Pi.

Verifying Caddy

Once the config file is built, you can perform a test run to confirm everything is working by executing this command.

sudo ./caddy run --config config.json

We need to execute using sudo so that we can expose the service to restricted ports 80 and 443 (HTTP and HTTPS respectively).

Now we have a memorable domain name fronting the service, and Firefox is happy that we’re encrypting the connection too. The certificate being produced in seen below.

Enabling Caddy Service

Since we’re not using the standard Caddy installation method, we will need to specify a service unit file so that Caddy starts up at the same time as the host - which is what PiHole and UniFi are doing currently.

First check to see if there is a stale service there already.

$ ls -la /etc/systemd/system/caddy.service
lrwxrwxrwx 1 root root 9 Jun  4 09:14 /etc/systemd/system/caddy.service -> /dev/null

If you get the above then remove the symlink so that we can create a file there.

rm /etc/systemd/system/caddy.service

Then populate the same file with the below, remembering the change the location of the Caddy config file to where it exists on your machine.

[Unit]
Description=Caddy Reverse Proxy
Wants=network-online.target
After=network.target network-online.target
 
[Service]
ExecStart=/usr/local/bin/caddy run --config /home/jdheyburn/homelab/caddy/config.json
Restart=on-abort
 
[Install]
WantedBy=multi-user.target

Finalise the new service with the two commands, enabling it on host startup and starting the service right now.

sudo systemctl enable caddy.service
sudo systemctl start caddy.service

Conclusion

For now I have all the above running bare-metal on one Pi instance, which produces a huge single point of failure in my network. In the future I’d like to see how converting these to Docker containers and having them distributed on multiple Pis would increase the resiliency of these services.

Until then, these basic but essential services are being hosted at easy to remember domains, transported over an encrypted connection, for me to easily administer the network for when it gets more complex over time.

Who Goes Blogging 6: Three Steps to Improve Hugo's RSS Feeds

Thu, 14 May 2020 00:00:00 +0000

I “fixed” the default RSS template used by Hugo to show the full article content, along with images, and also talk about how to have social media cards appear in the RSS items too.

I uploaded my completed RSS file to GitHub Gist 🚀

What is RSS?

RSS is a great way to “subscribe” to websites to ensure that you don’t miss content from them. The standard for how it is generated has been the same for decades now - however its still the best supported and most accepted way to receive updates. It is simply a standardised XML file that allows consumers such as RSS aggregators to parse the content of the post - for which there are many to choose from (RIP Google Reader).

My RSS aggregator of choice currently is inoreader - I’ll be using this in my examples

Hugo generates RSS XML files from a template that will loop over your content and expose this at an endpoint for RSS aggregators to subscribe to and periodically check for updates against. Hugo generates a bunch of RSS XML files at different sections of the website, allowing consumers to subscribe to the section that interests them most - you can even subscribe to tags if that XML is being generated!

Usually there is site top-level available at /index.xml - in the case of my site that would be https://jdheyburn.co.uk/index.xml. The source code for the template file Hugo uses to generate this is embedded in Hugo at this location.

The Problem

Some (but not all) websites only include the first paragraph of their post in an RSS update (aka summary), along with a message after that paragraph requesting the reader to visit the site in a browser to view the rest of the content.

This is a common feature of WordPress blogs - and it is done to have you load the full website and along with it, all the code for providing the owner with user analytics, and advertising - if they have it. So really they’re getting the benefit of being able to publish updates via a standardised approach (RSS), to then lure you onto the website.

Hugo kind of does this through the highlighted line in the templated RSS XML file:

<description>{{ .Summary | html }}</description>

.Summary will do what was just described, print out the first paragraph of your post. However unlike WordPress, it doesn’t actually say if there is more content. So users may not load your full website, instead thinking you’ve produced a very short article!

I’m not against the practice of previewing the post, if a blog provides revenue for the author(s) then you will need to force readers to view the full website for analytics and advertising (businesses gotta make money!).

Although I’m sure the initial purpose of RSS was only meant to be used to provide subscribers a pure text version of your content. At the end of the day, I believe the author should give the reader choice on mediums of consumption - and they accept the limitations that come from RSS.

So now, I’ll be making several changes to the Hugo template RSS XML file to produce the full site content, along with some other enhancements.

Rendering Full Site Content

Before we start editing files, we need to produce our own copy of the template RSS XML file. This is embedded directly within Hugo itself and not packaged in with your theme.

There’s every chance your theme is generating one for you. Run the below command to see if there is anything for you to copy from.
find themes/ -type f -name '*.xml'

We’ve discussed template lookup orders before in previous posts - we can overwrite the default of any file used in generating the website by placing it in your local project root.

We can start to make modifications to the rss.xml file by copying the default into the directory where Hugo will read it from.

# Assuming you are in project root
wget https://raw.githubusercontent.com/gohugoio/hugo/master/tpl/tplimpl/embedded/templates/_default/rss.xml -O layouts/_default/rss.xml

We know that line 35 containing .Summary is causing the issue here. All we need to do is change it to .Content - then Hugo will print out the entire content of your post. See the highlighted line below for the change.

    <item>
      <title>{{ .Title }}</title>
      <link>{{ .Permalink }}</link>
      <pubDate>{{ .Date.Format "Mon, 02 Jan 2006 15:04:05 -0700" | safeHTML }}</pubDate>
      {{ with .Site.Author.email }}<author>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}</author>{{end}}
      <guid>{{ .Permalink }}</guid>
      <description>{{ .Content | html }}</description>
    </item>
    {{ end }}
  </channel>
</rss>

Let’s now view the changes in the RSS reader.

We can now see much more content is being produced - and with proper HTML this time!

UPDATE 2020-12-14

I came across an issue whereby code indentation wasn’t rendering properly in RSS feeds if you used the --minify parameter - this post resolves it.

Rendering images in RSS posts

Depending on how you are retrieving images - you may find that they are not displaying, and that the image alterative (alt) text is being displayed instead.

Note the highlighted alt texts of images

If you’re not already providing an alt field to your <img> blocks then you absolutely need to.

They help boost accessibility to your website should the user have a text-to-voice application reading your website.

They also provide a helpful description to your image in case it cannot load - which happens more often than you think!

Why it is breaking 😤

In part 4 of this series, we looked at placing resources for a post in the same directory as the content markdown file (also known as leaf bundles).

That means for a given directory tree structure:

$ tree content/blog
content/blog
└── some-blog-post
    ├── image-file-name.png
    └── index.md

We would use the following shortcode to generate the <img> block in index.md.

{{< figure src="image-file-name.png" alt="Here is an image!" >}}

This produces the block:

<figure>
    <img src="image-file-name.png" alt="Here is an image!">  
</figure>

Since this contains no preceding forward-slash / in src, this becomes a relative path. This means the webserver will look for the image at the path relative to the current page.

So when we navigate to blog/some-blog-post/ in our web browser (as dictated from the above tree structure), your browser will request the image at blog/some-blog-post/image-file-name.png for you.

Why is this relevant? This same <img> block is being rendered in the description of your RSS XML file, including the relative image location. RSS is very simple and renders only complete URLs (e.g. https://jdheyburn.co.uk/blog/some-blog-post/image-file-name.png) - so when we just specify the relative path, it cannot find the image and thus prints out the alt text.

Note: if you specify images from a complete path such as /images/image-file-name.png under static/images, then this is not an issue for you.

Fix image rendering

The fix for this involves some Go magic. We need to prepend the permalink for the current post that’s being printed out to anything that contains a relative path <img> block.

We need to go back to the same line where we added .Content and change it to:

<description>{{ replaceRE "img src=\"(.*?)\"" (printf "%s%s%s" "img src=\"" .Permalink "$1\"") .Content | html }}</description>

This calls the Hugo function replaceRE which enables us to perform a find-and-replace using regular expressions (regex), and it takes three inputs:

PATTERN is the regex pattern used to find the text we want to remove
- "img src=\"(.*?)\""
REPLACEMENT is the text that we want to replace the PATTERN with
- (printf "%s%s%s" "img src=\"" .Permalink "$1\"")
INPUT is the string we want to perform the find-and-replace on
- .Content

In the example above, we’re placing a capturing group in the PATTERN so that it will capture the contents inside src and save them so that we can refer to it in the REPLACEMENT, while removing img src="".

REPLACEMENT is calling another Hugo function called printf to construct the replacement text. We’re pretty much just injecting the page .Permalink prior to the captured text from the PATTERN - so that RSS can then display the complete URL. While adding back the img src="" that was removed.

INPUT will be the page .Content, what we were printing out before.

Once we’ve made the change, images are now rendered properly!

UPDATE 2020-05-16

Upon posting this, I noticed I needed to do the same thing for hyperlinks referencing headings with the same post. For instance the markdown below which renders a hyperlink to a heading…
[images to display in RSS content](#rendering-images-in-rss-posts)
Will render this in RSS:
<a href="https://jdheyburn.co.uk/#rendering-images-in-rss-posts">
Which does not take the reader to the right location at all. What we need to do is repeat the regex used above in rss.xml to catch these and replace them with the complete URL. See below for the snippet.
{{- $content := replaceRE "a href=\"(#.*?)\"" (printf "%s%s%s" "a href=\"" .Permalink "$1\"") .Content -}}
{{- $content = replaceRE "img src=\"(.*?)\"" (printf "%s%s%s" "img src=\"" .Permalink "$1\"") $content -}}
<description>{{ $content | html }}</description>

Adding cards for posts

Not dissimilar to social media cards, you can also have cards appear for your RSS posts in order to make them more attractive for readers to click on!

While in card view for inoreader, we can see featured images being displayed

You can do these in RSS via the enclosure tag…

<enclosure url="image-location.png" type="image/jpg"></enclosure>

If you have a look at our layouts/_default/rss.xml file, we don’t have this defined. We could go and add it in now, but I would like a unified social media image to be used across all platforms. This will enable me to focus my effort on generating a single social media card image that is shared across all methods of consumption.

Based on the above, my theme (hugo-coder), renders the tags required for social media cards through the internal template for twitter_cards. A snippet of the logic that determines this:

{{- with $.Params.images -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ index . 0 | absURL }}"/>
{{ else -}}
{{- $images := $.Resources.ByType "image" -}}
{{- $featured := $images.GetMatch "*feature*" -}}
{{- if not $featured }}{{ $featured = $images.GetMatch "{*cover*,*thumbnail*}" }}{{ end -}}
{{- with $featured -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ $featured.Permalink }}"/>
{{- else -}}
{{- with $.Site.Params.images -}}
<meta name="twitter:card" content="summary_large_image"/>
<meta name="twitter:image" content="{{ index . 0 | absURL }}"/>
{{ else -}}
...

The hierarchy of images used to render the social media card goes like this:

The first entry in the list of images found in the post front matter
Any image in the post leaf bundle containing feature in its name
Any image in the post leaf bundle containing either cover or thumbnail in its name
The default image defined at the site level params

Adding enclosure tags

We can effectively copy and paste the code being used for generating Twitter cards and modify it for enclosure tags. The snippet of code ultimately ends up looking like this 👇

{{- $pagePermalink := .Permalink -}}
{{- with .Params.images -}}
{{- $img := index . 0 -}}
<enclosure url="{{ printf "%s%s" $pagePermalink $img }}" type="image/jpg"></enclosure>
{{ else -}}
{{- $images := .Resources.ByType "image" -}}
{{- $featured := $images.GetMatch "*feature*" -}}
{{- if not $featured }}{{ $featured = $images.GetMatch "{*cover*,*thumbnail*}" }}{{ end -}}
{{- with $featured -}}
<enclosure url="{{ $featured.Permalink }}" type="image/jpg"></enclosure>
{{- else -}}
{{- with .Site.Params.images -}}
<enclosure url="{{ index . 0 | absURL }}" type="image/jpg"></enclosure>
{{- end -}}
{{- end -}}
{{- end }}

Walking through what this is doing:

Lines 1-4 will use the image defined in the images in the post front matter, if it exists
- I had to define {{-/* $pagePermalink := .Permalink */-}} on line 1 to make .Permalink available inside the with block at line 4
Line 7 will find an image containing feature in its name, at the post leaf bundle section, if an image is not found previously
Line 8 will revert to an image containing either cover or thumbnail, if an image is not found previously
Lines 12-13 will default to the image defined at the site level params

With lessons learnt trying to get images to display in RSS content, I am rendering the full URL of the image to ensure RSS is always able to retrieve it.

I’m going to follow lines 12-13 above and have a default card defined at the site level parameters. In my config.toml I’ll need to add the following…

[params]
    images = ["images/jdheyburn_co_uk_card.png"]

…while also ensuring that I’ve placed the image under static/images/jdheyburn_co_uk_card.png.

Once all that is done and deployed - we will have an RSS post card that looks like this 🙌

We can now advertise our RSS subscription URL to readers. My theme hugo-coder supports this as a social button on the home page. We just need to add the below block to the config.toml file.

[[params.social]]
    name = "RSS"
    icon = "fas fa-rss"
    weight = 6
    url = "https://jdheyburn.co.uk/index.xml"
    rel = "alternate"
    type = "application/rss+xml"

Once done - our homepage will have the button displayed.

Summarising changes to RSS XML

You can view the gist below to see my complete RSS XML file after all the above has been added. If you wish to use it, you’ll need to place it in layouts/_default/rss.xml in your project directory.

Click here for a git diff view on the changes I made from the default RSS XML file.

Thanks for reading! 💯

Who Goes Blogging 5: Updates Galore!

Sun, 19 Apr 2020 00:00:00 +0000

In this article we’re going to continue making some improvements to our Hugo website. Some of the things going to be covered are:

Upgrading our Hugo theme
Upgrading the Hugo version
Adding table of contents
Adding support for title emojis

So far since this website has launched we haven’t upgraded either the Hugo version used to build our website, alongside the theme being used. Hugo is a very active project with new features and bug fixes being added all the time. By not upgrading Hugo, we are missing out on enhancements that we can make to the site.

As for upgrading the theme, improvements in the styling or even making use of said new features in Hugo can be added as part of the theme. Remember that when you execute a hugo command, turning the website from markdown to rendered HTML files, that the theme chosen will be used to render those files.

Upgrading Hugo

This varies depending on what OS you are running on, but it is pretty much identical as the steps taken to install Hugo in the first place. Even the documentation for upgrading is a one-liner:

Upgrading Hugo is as easy as downloading and replacing the executable you’ve placed in your PATH, or running brew upgrade hugo if using Homebrew on macOS.

A quick execute of hugo version tells us what we’re currently running. We can then have a look at the Hugo release page to find the latest one - which at the time of writing is 0.68.3.

$ hugo version
Hugo Static Site Generator v0.58.3-4AAC02D4/extended linux/amd64 BuildDate: 2019-09-19T15:34:54Z

I run Linux so I would need to replace the executable on my PATH. This can be done with a handy script to pull the archive and install it for us.

HUGO_VERSION="0.68.3"
wget "https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz"
tar -zxvf "hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz" hugo
sudo mv -v hugo /usr/local/bin
hugo version
rm "hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz"

Remember that the hugo-coder theme requires the extended version of Hugo in order for it to render files correctly.

You can see where hugo is currently installed on your system with the command which hugo, given it is in your PATH.

Executing hugo version one more time confirms if the upgrade took place.

$ hugo version
Hugo Static Site Generator v0.68.3-157669A0/extended linux/amd64 BuildDate: 2020-03-24T12:13:38Z

Testing For Any Issues

At this point I would advise you do some testing with the new version to determine if your site is not only rendering as expected but functioning too. You can do this with the good old hugo serve command as you would have used before when writing out articles, and fixing anything that looks out of place.

However, since I’m going to be upgrading the hugo theme too, I’ll do my testing in one hit later in this post.

Updating Hugo in CI/CD

We’ve only upgrading Hugo on our machines, but we also need to upgrade it in our CI/CD pipeline to ensure it deploys with the correct rendering too. We set this up in parts 3.1 and 3.2.

I’m using GitHub Actions for my CI/CD pipeline, so I need to head to .github/workflows/deploy.yml and change hugo-version: "0.58.3" to hugo-version: "0.68.3".

It’s as easy as that thanks to the pre-made GitHub Action!

Upgrading Hugo Theme

A git submodules recap

As mentioned before, I’m using hugo-coder as the theme for my Hugo website which is being pulled as a submodule in my git repository. I can interface with it via the command git submodules.

One of the cool things about git submodules is I don’t need to have a copy of the source code for the theme in my project. I can just reference the codebase at a particular point in time - or a commit, and git will pull in the resources for which then my Hugo website can utilise it.

We already defined this back in the first post of this series, in our root directory we created a file called .gitmodules with the following contents:

[submodule "themes/hugo-coder"]
	path = themes/hugo-coder
	url = https://github.com/luizdepra/hugo-coder.git

Line by line, we are giving the submodule a name, telling git where to check out the module to, and lastly where git can find the repository for the module when we checkout our project.

When we pull in the submodule for the first time, it creates a tree in our projects .git/modules filled with metadata about the submodule.

$ tree .git/modules/themes/hugo-coder -L 1
.git/modules/themes/hugo-coder
├── branches
├── config
├── description
├── HEAD
├── hooks
├── index
├── info
├── logs
├── objects
├── packed-refs
└── refs

The .git/ directory contains all the metadata about your repository so that git knows where to pull/push to/from the repository, amongst other things. It has a near identical directory structure to the one shown above.

One of the files in this directory is called HEAD, and it contains the commit of which git should checkout the submodule

$ cat .git/modules/themes/hugo-coder/HEAD
e66c1740611b15eee0069811d10b5a22b0dc4332

We can even view this commit in the repository at https://github.com/luizdepra/hugo-coder/commit/e66c1740611b15eee0069811d10b5a22b0dc4332.

Upgrading Submodules

To upgrade submodules in your project, you simply need to execute the command below:

git submodule update --recursive --remote

We can then check the value in .git/modules/themes/hugo-coder/HEAD and see what this is updated to.

$ cat .git/modules/themes/hugo-coder/HEAD
acb4bf6f2dcce51a27dc0e1f1008afb369d378d8

Nice - this has changed from the previous value. We can jump over to GitHub to verify this is indeed the latest version.

Verify at the GitHub repo page

Validation and Testing

Now comes the testing part, ensuring that everything is working as intended with both the new Hugo version and the latest hugo-coder submodule. Some of the issues I came across were:

`<center>` tags not rending correctly

I believe the new Hugo version broke this one, where images that were wrapped in <center> tags were simply disappearing. I had been wrapping {{< figure ... >}} shortcodes in <center> tags in order to horizontally centre images - but it looks like the new version causes these images to appear completely.

It looks like the figure shortcodes now support center as a class property. Therefore we go from…

<center>{{< figure src="image-name.png" >}}</center>

To…

{{< figure src="image-name.png" class="center" >}}

Lists not rendering as before

It turns out that my markdown for articles wasn’t as watertight as I thought; Hugo is now a whole lot more strict in ensuring your syntax is correct. Check these two before and after for comparison.

For this I just needed to fix the markdown by adding another indent to the list item.

I saw several cases of this, and including a markdown linter in the CI/CD pipeline would help to prevent this. This is something I’ll look to include in the near future.

Adding New Features

The whole reason to upgrade both Hugo and the theme was to explore some of the new features that come in both of them, so let’s get adding them.

Adding Series Links in Content Footer

The hugo-coder theme now has better support for series - these are articles that directly follow on from one another, such as this Who Goes Blogging one. Series are nothing new to Hugo and we could have done them with the version we upgraded from, only in this commit for hugo-coder has it added a layout for it to be rendered.

To create a series, we use the series front matter setting for articles. For the Who Goes Blogging articles, adding the front matter to them looks like this.

---
date: 2020-03-10
title: "Who Goes Blogging 3.2: Deployment Methods - GitHub Actions"
#...
series:
  - Who Goes Blogging
tags:
#...
---

Once this is done for all pages, hugo-coder will render the See also in … section at the article footer.

Adding Table of Contents

Okay this one isn’t exactly a new feature of the theme, but since we’re playing around we may as well make use of this opportunity. Table of contents (TOC) aren’t included by default in the hugo-coder theme, so we need to add it in ourselves.

There is some documentation on how to add TOC to your articles. Before we do that we need to revisit how layouts are organised as discussed briefly in a previous post.

Template Lookup Order Primer

The type of a markdown file (defined by the front matter setting type) tells Hugo where to look for in the layouts/ directory of your project. The Hugo theme you’re using will already have these all defined, hence why they are predefined themes! Before Hugo looks for these layout HTML files in your theme, it will check for them in your projects layouts/ directory first.

In other words, the order of precedence (or hierarchy) for HTML files is anything in the project layouts/ directory takes a higher priority than that in your theme - as described in Hugo’s documentation for template lookup order.

See below for a directory tree explaning this - a lot of content has been removed for brevity.

$ tree layouts
layouts
├── posts
│   └── single.html					# hugo will use this file to render your HTML
└── themes
    └── hugo-coder
	    └── layouts
			└── posts
			    └── single.html     # hugo will ignore this one in the themes directory

Implementation

The primer above gives a hint as to what file we need to modify. The single.html file is in charge of how articles are rendered. Since we want everything else to remain the same about this file, we can take a copy of the file as it currently stands and place it into our project root.

mkdir layouts/posts
cp -v themes/hugo-coder/layouts/posts/single.html layouts/posts

I won’t go into the details here, I’ll leave that to you to make changes to see how they affect the rendering of the page.

I want the TOC to appear before the main content, but after the featured image for the article (if there is any). Based on this the files is going to need the highlighted change below.

<div>
  {{ if .Params.featured_image }}
  <img src="{{ .Params.featured_image }}" alt="Featured image" />
  {{ end }}
  {{ partial "toc.html" . }}
  {{ .Content }}
</div>

We’re not done there - we’ve referenced a partial template file here. These allow us to reuse HTML files in multiple places, or to break up a large HTML file into smaller, more easier to manage chunks.

Partial templates are found under the layouts/partials directory and follow the same principle as defined in the primer for template lookups, and are referenced back as {{ partial "<partial_name>.html" . }}.

Let’s go ahead and create the layouts/partials/toc.html file and populate it as below.

{{ if and (gt .WordCount 400 ) (.Site.Params.toc) }}
<aside>
  {{.TableOfContents}}
</aside>
{{ end }}

This is similar to Hugo’s example but just tweaked a bit. I’m not looking to do anything too fancy with the styling just yet, but maybe later on.

Previewing it by hugo serve we should get something that appears as below.

Bugs Along The Way

As the heading suggests, there were a few extra things I needed to change before releasing it into the wild.

Fix Improper Headings

Per Table of Contents Usage documentation, Hugo will render from <h2> headings, which are defined as ## in markdown. So # Introduction would not be included in the TOC, but ## Introduction will.

This is a problem for some of my articles since I’ve been writing headings liberally at the wrong level.

Notice the Applying Cartography heading doesn’t appear in the TOC

The given example above shows what happens when we use # Heading Name instead of ## Heading Name. To fix this I just need to go through each heading and make sure top-level headings begin with ## and any subheadings have an additional #.

Much better!

Fix Emojis

Who doesn’t love the use of emojis ⁉️ Well I don’t when they render badly. Before this exercise I was using a custom shortcode to render them.

# layouts/shortcodes/emoji.html
{{ .Get 0 | emojify }}

Shortcodes are similar to partial templates in that they contain HTML, but they are usually for referring to one HTML element as opposed to multiple.

Then in my markdown I would refer back to them such as {{< emoji ":wave:" >}}. However in headings they rendered as below…

Hugo clearly finds this hysterical

It’s a weird one for sure - especially since they render fine in the headings themselves already…

There is a fix for these, and it’s called rendering emojis the correct way! 😅

Firstly in the site config.toml I added enableEmoji = true at the root level and then proceeded to change all occurrence of {{< emoji ":emoji_name:" >}} to :emoji_name:. With that I can then remove layouts/shortcodes/emoji.html since it’s not being referred back to anymore.

Nicely done

Markdown Not Rendering

I also noticed that markdown wasn’t rendering properly too. Given the markdown ## Spice Up Your ~~Life~~ Python :snake:, which renders as a heading in the screenshot below…

But in the table of contents it appears without the tilde strikethrough (~) rendering.

This looks to be a bug within Hugo itself, since I was able to get strong and emphasise to render fine - I raised an issue on the project - I can live with it for now.

Emojify Everything!

Continuing on the theme of emojis; we were able to get them to appear in the Table of Contents fine. But what about in the titles of the articles themselves? We can add them in the title front matter setting like below:

---
date: 2019-07-05
title: ":snake: Three Ways To Spice Up Your Python Code"
# ...
---

Having a look around the site we see these aren’t getting rendered correctly in a number of places…

(1) Post title

(2) Browser title

(3) Type list entry

This is all to do with how the theme is rendering these, they are not “emojifying” the text it is receiving. This is easy to fix but it involves us copying over the respective files where they are being rendered from in hugo-coder and overwriting thme in our project root (back to template ordering again!).

Fix Post & Browser Title

We can kill two birds with one stone since these are both found in the same file, the layouts/posts/single.html file - this is the main file used to render the article page. If you followed above, you’ll notice we already copied across that file to our project root in order to add a TOC.

We need to go back to that file and emojify both the main article heading, alongside the browser title to change any occurrence of {{ .Title }} to {{ .Title | emojify }}.

By doing this we will always emojify the titles even if in our config.toml we specified enableEmoji = false, or didn’t specify it at all. This will conflict with the default state of the setting which is false. We could do some conditional logic on the enableEmoji setting, but unfortunately this isn’t exposed to us to be able to perform something like {{ if .Site.enableEmoji }}. Because of this we cannot create a pull request (PR) to merge it in with the hugo-coder theme. I’ve raised a couple of issues against Hugo requesting these features.

Since this is just for my website I am okay with the overriding behaviour of emojiying the title.

See the Gist below for my completed single.html file, including the TOC addition earlier.

You’ll notice I’m emojifying (surely I’ve said this enough times to make it into the Oxford Dictionary?!) the site title too. This is incase myself (or even yourselves) wish to add emojis there too.

The end result for the title will look like this:

And the browser title is fixed accordingly:

Fix Type List Entry

We need to find the file in the theme that determines how to render a list entry for a page of type posts and make the modification there - layouts/posts/li.html. We’ll need to do the same here as we did for single.html and copy it to our project root.

cp -v themes/hugo-coder/layouts/posts/li.html layouts/posts

Likewise for the title in the previous section, we need to emojify the title of the article as it’s being listed out.

Once done, we can see it reflect nicely in the list:

Conclusion

There was a lot covered in this post, to recap we…

Upgraded the Hugo version used to render the website locally, as well as in the CI/CD pipeline
Also updated the version of the theme we’re using
- Made use of new features that came with it such as series links
Added a table of contents to our longer articles
Added support for emojis in post titles

You can view all the changes made in this post at this pull request.

Thanks for reading! 🌝

Who Goes Blogging 4: Content Structure & Refactoring

Tue, 31 Mar 2020 00:00:00 +0000

So far in this series we’ve been looking at the infrastructure behind the Hugo website, but I haven’t looked into improving the content layout of the site. Some of the improvements I want to make are:

Moving content articles from a flat structure to a leaf bundle
Rename content section to blog from posts
Upgrading hugo-coder theme and hugo version
Adding a table of contents to the pages
… and further improvements as I find them

This post will cover items 1-2, the others will be in a follow up post.

Content Article Reorganising

Hugo has a few different ways you can organise your content, known as Page Bundles. Up to now I’ve adopted Branch Bundles where I have all my articles in named markdown files under the content/posts directory - acting as the branch bundle, and storing all images under static/images - so the article at content/posts/blog-bootstrap.md will be available at /posts/blog-bootstrap/. As time has gone on, the images directory has gotten rather large and difficult to understand what images apply to what posts.

The alternative way to organise content is the Leaf Bundle. This is where at a branch level you have a directory with the name of the article you wish to publish, followed by an index.md file with the article content - such as content/posts/blog-bootstrap/index.md.

Any page resources defined in this directory can be referenced relatively in the markdown files within it, such that any images can be stored within content/posts/blog-bootstrap/ and referred to in the markdown as figure src="my-relative-img.png" ... as opposed to src="/images/my-not-relative-img.png" ....

This allows me to organise content much more effectively such that we will be going from this structure:

$ tree content/posts 
content/posts
├── blog-bootstrap.md
├── extending-gotests-for-strict-error-tests.md
├── on-becoming-an-open-source-software-contributor.md
├── three-ways-to-spice-up-your-python-code.md
├── who-goes-blogging-0-applying-cartography.md
├── who-goes-blogging-1-getting-started.md
├── who-goes-blogging-2-custom-domain.md
├── who-goes-blogging-3-1-deployment-methods-travisci.md
└── who-goes-blogging-3-2-deployment-methods-github-actions.md

0 directories, 9 files

$ tree static/images 
static/images
├── analytics_example.png
├── ...                     # Everything stored in a flat hierarchy
└── type_hinting.png

0 directories, 57 files

To this:

$ tree content/posts
content/posts
├── blog-bootstrap
│   └── index.md
├── extending-gotests-for-strict-error-tests
│   └── index.md
├── on-becoming-an-open-source-software-contributor
│   ├── card.png
│   ├── elephant-cartoon.png
│   └── index.md
├── ...                     
└── who-goes-blogging-3-2-deployment-methods-github-actions
    ├── gha-deploy-key.png
    ├── gha-secrets-key.png
    ├── github-actions-build.png
    ├── index.md
    └── travis-disable-build.png

9 directories, 45 files

One other benefit of grouping all content together in a bundle is it allows you to use just one command to add your article content to the repository - no more will images in another directory be forgotten about!

git add content/blog/$articleName

Transformation Script

As opposed to doing the transformation manually, I’ve written a script which will move and alter the files for us.

# in project root dir
posts=$(find content/posts -type f -name '*.md' -not -path '**/index.md' -not -path '**/_index.md')
echo "$posts" | while read p; do
    # Move to leaf bundle
    postFname=$(basename -- "$p")
    title="${postFname%.*}"
    leafBundleDir=content/posts/$title
    mkdir $leafBundleDir
    newPostLocation=$leafBundleDir/index.md
    git mv -v $p $newPostLocation

    # Move images referenced in the articles to new leaf bundle dir
    images=$(grep -oP 'images\/.*.(png|jpg|jpeg)' $newPostLocation | xargs -n1 | sort -u)
    echo "$images" | while read i; do
        [ -z "$i" ] && continue
        imgFname=$(basename -- "$i")
        git mv -v static/$i $leafBundleDir/$imgFname
    done

    # Change references to images in the article
    ## figure src="/images/namecheap-domain-purchase.png"
    sed -i 's/src="\/images\//src="/g' $newPostLocation
    ## images:
    ##   - images/namecheap_landing.png
    sed -i 's/- images\//- /g' $newPostLocation
    ## [blog_arch]: /images/blog-arch-cover.png
    sed -i 's/: \/images\//: /g' $newPostLocation
    git add $newPostLocation
done

Breaking down the script, firstly we are retrieving all posts that aren’t already converted to a leaf bundle and looping over them, where lines 4-9 performs the conversion itself.

Once that is done then we need to get all images that are referenced in that post and move them to the new leaf bundle for the post. This is done in lines 12-17.

Lastly, we then need to change the post content itself so that it references images in the location relative to the leaf bundle instead of under static/images - lines 21-26 accomplish this.

Hugo allows you to define img blocks in several ways, where I’m currently using 3 of these methods. I ~~probably~~ should replace these with one approach - for the time being 3 sed commands will do the trick for me.

I’m sure that a few lines could be cut out with some better regex expressions - but I’m not an expert at them so I’d rather crack on with what I know 😃

Renaming Content Section

A gripe I’ve had with my site is when you click on Blog in the site header it takes you to Posts. Similar to the previous section this is because of Branch Bundles which are any directory that sits underneath the content directory. The bundle name here is what forms the URL and the parent page for those articles as defined.

Based on this information, migrating should be as easy as…

$ git mv content/posts/ content/blog/

Despite doing this, we come across some problems…

We lose the correct rendering for the article
- Hugo doesn’t know how to render content with type blog
Existing URL links are broken
- All hyperlinks both internal and external are referring to articles at /posts/
Blog landing page is now Blogs?? (see below)

Let’s walk through them one by one.

Fixing Article Rendering

We just made our existing articles leaf bundles that sat under content/posts, but now this is renamed to content/blog:

$ tree contents/blog -L 1
content/blog
├── blog-bootstrap
├── extending-gotests-for-strict-error-tests
├── on-becoming-an-open-source-software-contributor
├── three-ways-to-spice-up-your-python-code
├── who-goes-blogging-0-applying-cartography
├── who-goes-blogging-1-getting-started
├── who-goes-blogging-2-custom-domain
├── who-goes-blogging-3-1-deployment-methods-travisci
└── who-goes-blogging-3-2-deployment-methods-github-actions

9 directories, 0 files

The section of which your article sits underneath also tells Hugo how it should render the page - known as Content Types. Take the article blog-bootstrap which is under posts; Hugo will look in the theme for layouts/posts for html files to render it against* - this is how the articles go from a markdown file to rendered HTML.

* Hugo will first look in your local project for any overriding files.

$ tree themes/hugo-coder/layouts/posts 
themes/hugo-coder/layouts/posts
├── li.html
├── list.html
└── single.html

0 directories, 3 files

When we change the section name the article sits in, Hugo will not know how to render it correctly and will resolve to the default rendering for the theme. This ends us with something like the below.

Correct rendering on the left - notice publish date, minutes to read, and tags are missing

One way of fixing this would be to also change the hugo-coder theme from layouts/posts to layouts/blog - however a much simpler solution exists. We can use the type front matter setting to tell Hugo what layout to use to render it.

---
date: 2020-03-10
title: "Who Goes Blogging 3.2: Deployment Methods - GitHub Actions"
description: Migrating to GitHub Actions as our CI tool
type: posts
# ...
---

Hugo front matter are content parameter settings you place into the heading of the content - an example of this is right above!

Having to define this in every new article can become repetitive. We can make this easier for ourselves by using archetypes; which are essentially templates that new content is templated from.

In your project root directory, create the archetypes/blog directory and in there a file called index.md, and populate it with the below:

---
draft: true
type: posts
# ...       Add in other defaults as you see fit
---

You can then create new content from this template like so:

$ hugo new --kind blog blog/blog-post 
/home/jdheyburn/projects/jdheyburn.co.uk/content/blog/blog-post created
 
$ tree content/blog/blog-post         
content/blog/blog-post
└── index.md

0 directories, 1 file

$ cat content/blog/blog-post/index.md 
---
draft: true
type: posts
# ...
---

Fixing Existing URLs

Changing the section the article is in affects the URL too; meaning articles will appear at https://jdheyburn.co.uk/blog/blog-bootstrap/ instead of https://jdheyburn.co.uk/posts/blog-bootstrap/. Therefore existing hyperlinks pointing to these articles will break.

There is another quick fix available for us through the front matter setting aliases.

---
date: 2020-03-10
title: "Who Goes Blogging 3.2: Deployment Methods - GitHub Actions"
description: Migrating to GitHub Actions as our CI tool
type: posts
aliases:
    - /posts/who-goes-blogging-3-2-deployment-methods-github-actions/
# ...
---

You can read more about how aliases does its operations here.

From Blogs to Blog

Earlier on, we came across this weird bug…

This is occurring because of the translation feature of Hugo (sourced from i18n) which converts singular word titles to plural on content section landing pages, since Hugo is listing them out. This made sense when our previous section name was posts, but now we’d rather it have say blog.

Remembering that content/blog is a branch bundle, we can override the inferred title with a _index.md file at that directory.

# content/blog/_index.md
---
title: Blog
type: posts
---

We have another type: posts setting here because similar to above, we want Hugo to render the landing page the same as we did for posts.

Heading to Blog

Lastly and most easily, the Blog shortcut at the top-right corner of the page still directs us to /posts - we need to change that in the top-level config.toml to point to /blog.

[[languages.en.menu.main]]
name = "Blog"
weight = 2
url = "/blog/"

Thanks for reading!

Who Goes Blogging 3.2: Deployment Methods - GitHub Actions

Tue, 10 Mar 2020 00:00:00 +0000

From TravisCI to GitHub Actions

In the previous post we looked at moving to a CI/CD model by moving from the deploy.sh script to TravisCI.

In this post we will look at how we can migrate from TravisCI to GitHub Actions, GitHub’s own CI/CD tool.

This post will also be useful if you are looking to onboard GitHub Actions as your CI/CD pipeline! 🚀

Benefits ✅

Let’s talk about why we want to migrate away from TravisCI in the first place.

Firstly and most importantly, there is a whole community of shared actions (a set of build instructions) which can save you a huge amount of time when it comes to piecing together a CI pipeline. If the TravisCI config seemed a bit intimidating, then these will be a whole lot more gentler to you.

Whereas in our Travis config we had to define the individual commands needed to set up our environment and then how to build it, there’s an action for that! Want to include some markdown linting? There’s an action for that!

I think you folks get the picture now. There’s an awesome-actions repository worth checking out for more actions.

Secondly, all your DevOps tools are in one place! I’m a big sucker for GitLab and while I don’t use it for my personal projects, I’ve used it in a past life and found its seamless integration with all other tools second-to-none. Not having to worry about integrating between multiple services can only increase your productivity - allowing you to focus more on the application you’re writing.

Lastly, all configuration is managed in the workflow configuration file. One enhancement in particular that we will be introducing can be achieved with an additional setting in the workflow config file; for us to achieve the same in Travis would have to be done via the GUI. I’m a big sucker for having configuration baked into code so this is a very good plus.

Pricing ⏰

However - one downsides to GitHub Actions is how many build minutes you get. Remember Travis allowed unlimited build minutes for a public repository? With Actions - you are limited to 2,000 minutes in their free plan.

If you’ve been building your project in Travis already, you’ll notice it has been building (in my case at least) in ~30 seconds. With a bit of maths we can then say we will have 4,000 builds in a month on GitHub Actions.

Given that this isn’t a huge project with multiple contributors working on it, I think it’s safe to say we won’t ever reach this limit - unless you’re churning out blog posts left right and centre!

Sound good? Let’s go.

Creating Our Workflow

Like all great services in the world, there is great documentation to go along with them. Take a look over there if you’d like the detailed version.

What I will be focusing on is the documentation for two sets of predefined actions; actions-hugo for building our website, and actions-gh-pages for deploying it to GitHub Pages.

Deployment Keys Setup

The very first thing we need to do is set up some keys that will allow our source repository (where the workflow will reside on) to push the built project to the GitHub Pages repo.

In your terminal, create those keys now and copy the contents of the public key to your clipboard.

$ ssh-keygen -t rsa -b 4096 -C "$(git config user.email)" -f ~/.ssh/gh-pages -N ""
Generating public/private rsa key pair.
Your identification has been saved in /home/jdheyburn/.ssh/gh-pages.
Your public key has been saved in /home/jdheyburn/.ssh/gh-pages.pub
# ...

$ pbcopy < ~/.ssh/gh-pages.pub

macOS has a handy terminal command to copy file contents to the clipboard called pbcopy. I’ve created an alias on my Linux laptop that does the same.

alias pbcopy="xclip -selection clipboard"

In GitHub load up your GitHub Pages repo and navigate to Settings and then Deploy keys. Give it an appropriate name, and paste in the public key. Make sure you check Allow write access.

Copy the contents of the private key you created earlier (perhaps using your new command?! 😏) and navigate to the source code repository’s Settings page, then Secrets. You’ll need to give it a sensible name as this then referred to later in the workflow configuration. Paste the private key in the value field.

No secrets here!

Workflow Configuration

In the root directory of your source code repo, create a directory called .github/workflows. In this directory is where GitHub Actions will look for jobs to do. Create a yml file in this directory to contain your build job definition. I went ahead and named mine deploy.yml, but you can name it whatever you like.

I used the example provided in the actions-gh-pages documentation as a base for my build definition.

This config is much simpler to understand than the Travis one - let’s break it down once more.

Build Metadata and Environment

name: Build and deploy to jdheyburn.github.io

on:
  push:
    branches:
      - master
  schedule:
    - cron:  "0 10 * * *"

jobs:
  deploy:
    runs-on: ubuntu-18.04

This is pretty box standard at this point. We’re simply giving the name to the workflow and saying to run it on every push to the master branch.

An enhancement that we’re adding is the on.schedule.cron setting. This tells Actions not only to build the project on every push to the master branch but on a timed schedule too - in my example this is at 10am everyday. We could have done the same via Travis, but that can only be configured via the GUI.

The benefit to this is the way in which Hugo generates content. Hugo will only build content pages where the date on the content is either today or in the past, and is not a draft.

Therefore if you had written a post due to be published in the future, you can define that date and have the daily Hugo build publish it when that date has been reached - which is exactly how this blog post was published!

jobs is the field that contains the work we want to run. We’re giving the job a name of deploy and telling it to run on ubuntu-18.04 - which is the equivalent to bionic in Ubuntu.

In order to keep our build the same as Travis’s, we could instruct the job to run on ubuntu-16.04, nonetheless I’m pretty confident it will run on the next LTS of Ubuntu.

Project Checkout

# ...
steps:
  - uses: actions/checkout@v2

  - name: Checkout submodules
    shell: bash
    run: |
      auth_header="$(git config --local --get http.https://github.com/.extraheader)"
      git submodule sync --recursive
      git -c "http.extraheader=$auth_header" -c protocol.version=2 submodule update --init --force --recursive --depth=1

Here we’re executing the GitHub checkout action which will pull the repo to the build server. It’s worth noting that this action version I’m using here doesn’t checkout git submodules too - which is a problem for us as that’s how we’re currently pulling the theme for our Hugo site. We can workaround it with the next step in the build - Checkout submodules.

This functionality was available in v1 of the action, so you can use that if you’d prefer:

steps:
  - uses: actions/checkout@v2
    with: 
      submodules: true

It’s worth noting that we didn’t have to do this step for Travis - since it will checkout the repository with submodules already.

This is because GitHub Actions can be used for many more things than just repository code manipulation where you may not necessarily need the repo checked out.

Build and Deploy Setup

- name: Setup Hugo
  uses: peaceiris/actions-hugo@v2
  with:
    hugo-version: "0.58.3"
    extended: true

As stated we’re going to be using the Action actions-hugo to set up hugo on our build server. It can take in a number of parameters to allow us to customise it; for us we’re only concerned with hardcoding the version of Hugo, and to use the Hugo Extended binary as required by our theme.

At the time of writing 0.58.3 is not the latest version of Hugo, whereas when I wrote .travis.yml it was. I want the GitHub Actions build to be as close as possible to the TravisCI build in order to make them as similar as possible to prevent any unexpected build errors.

Once I’ve achieved a successful build - I’ll look to upgrade to a newer version, and then iron out any issues from there.

- name: Build
  run: hugo --gc --minify

Once hugo is set up we can then build it easily enough, as self-documented in the code.

- name: Copy CNAME
  run: cp CNAME ./public/

Lastly, we can’t forget to copy the CNAME file we made in part 2.

Deployment to GitHub Pages

- name: Deploy
  uses: peaceiris/actions-gh-pages@v3
  with:
    deploy_key: ${{ secrets.DEPLOY_KEY }}
    external_repository: jdheyburn/jdheyburn.github.io
    publish_branch: master
    publish_dir: ./public
    commit_message: ${{ github.event.head_commit.message }}

For the deployment to GitHub Pages I’m using the action actions-gh-pages. Again it only requires a bare minimum of parameters to work; an explanation of what I’m using is:

deploy_key is the private key we set up earlier in this post in Settings -> Secrets
- If you didn’t name yours DEPLOY_KEY then you’ll need to change it here too.
external_repository tells the action where we want the built website to go to - we set this to our GitHub Pages repo
publish_branch is the branch of the repo we publish to
publish_dir is the directory on the build server that we want to push to the repo
- Remember that hugo builds the website to the public directory locally
commit_message allows us to specify a custom commit message to the target repo
- Here I am telling it to inherit the commit message used in the source repo

Bringing It All Together (Again)

If you’re migrating from a previous CI tool (perhaps Travis?) then you’ll need to disable the builds on there since you may cause a conflict either build process.

For Travis, you can do that by navigating to your source code repo settings on Travis (https://travis-ci.com/jdheyburn/jdheyburn.co.uk/settings for me) and disabling Build pushed branches.

Now that’s done, go ahead and check in your new GitHub Actions workflow file and then navigate to the Actions tab of your source code repo on GitHub.

git add --all
git commit -m 'Migrate to GitHub Actions'
git push

Great success!

Hopefully your build went to success! If it didn’t have a look through the logs and see what the issue was. It took me a few builds to determine my finalised workflow config. You can even see it at my source code repo Actions page.

Conclusion

Now that we’ve migrated across over to GitHub Actions, we can close out the permissions that TravisCI has on our projects, and demise any secret keys we gave it.

From the tone of my writing you can probably tell which one I favour. That’s not to say I do not like TravisCI - each service has its own pros and cons. For this particular project, I prefer the one platform approach for which I am used to in GitLab. The number of build minutes available for GA is a concern, but not one I will have to worry about for now.

Thanks for reading! 🌝

Who Goes Blogging 3.1: Deployment Methods - TravisCI

Tue, 25 Feb 2020 00:00:00 +0000

Recap

Since part 1, we have been using a simple bash script called deploy.sh to build our Hugo website and upload it to our GitHub Pages repo. In part 2 we modified it slightly to include the CNAME file post-build to ensure GitHub Pages uses the custom domain we set up in that same part.

For this part, I will tell you about how I migrated from deploying via a script, to a CI/CD tool - namely TravisCI. Then I will document how I migrated from this, to the new GitHub Actions; GitHub’s offering into the CI/CD space.

CI/CD is an acronym for Continuous Integration / Continuous Deployment which is a very important concept in the DevOps culture. If you would like to find out more about that and DevOps culture, check out these resources 👇

https://www.atlassian.com/devops

https://medium.com/faun/the-basics-of-continuous-integration-delivery-with-10-most-popular-tools-to-use-9514231533f0

https://www.redhat.com/en/topics/devops/what-is-ci-cd

https://www.atlassian.com/devops

https://www.amazon.co.uk/dp/B07B9F83WM

I had originally planned this to be one post, but it soon became far too long - and I’m a fan of taking in content in sizeable chunks.

So this post will focus on migrating to TravisCI from the deploy.sh script. Whereas the next post will focus on migrating to, and setting up GitHub Actions. If you’re only interested in using that as your CI tool then I’ll provide a link to that here when it is posted.

Moving Away From deploy.sh

There’s nothing necessarily wrong with using deploy.sh to push our code, however we want to get all the bells and whistles that Continuous Integration can provide to us such as running a series of tests and checks automatically against every commit to our repository. Once those tests and checks pass then we can automate the deployment of our website.

Now there are many CI tools out there with the most well known likely to be Jenkins, but there are also hosted solutions available which will take your code and perform your pipelines against them.

One of those hosted solutions is TravisCI, where they integrate quite nicely with GitHub repositories to attach webhooks against them. They have several pricing options available, but for public open source projects, it is completely free!

So it is a good idea to set your source code repository on GitHub to be public. TravisCI does include (at the time of writing) private projects in their free plan, but you are capped in some shape or form on how much the platform will do for you.

Free is definitely a thing you love to see

TravisCI Account Setup

The TravisCI account setup for TravisCI is very streamlined - instead of creating another account for you to manage, it integrates in with GitHub, so this is the account you use to sign-up with. Head over to https://travis-ci.com/ and click on Sign in with GitHub.

You can’t resist a big green button…

GitHub will ask you if you really want to share some of your GitHub data with Travis.

Another green button? Why not!

Once you’ve done that, you’ll be redirected to your new Travis Dashboard which… is looking rather lonely 😦 - let’s fix that!

All we’ve done so far is allowed Travis to reach GitHub for creating an account for us - we now need to activate GitHub Apps integration to permit it to read and write to our repositories. The https://travis-ci.com/account/repositories page is what you need for that - then click on the Activate button.

…More green buttons?!

Now on the next screen you may or may not want the default selection of All repositories which will give Travis read and write access to all your repos. I completely trust Travis if I were to select this, however it is a best practice to follow the principle of least privilege (POLP); not just for users but for services too.

For the scope of this effort we’re only wanting Travis to read and manipulate against two repos, jdheyburn.co.uk and jdheyburn.github.io - it also gives you a cleaner Travis dashboard too.

Back to GitHub

The next step is required to permit Travis to push the built project to our GitHub Pages repo. We need to generate a secret with the permissions that Travis requires and keep it aside for the Travis config file later.

Navigate to the GitHub Personal Access Tokens page and click on Generate new token.

You’ll come across a page asking for the name of the token being created. It doesn’t matter what you call it, but it may be useful to link it back to what it is being used for. You’re also going to want to select the repo checkbox as done so below.

After this you don’t need to provide any more permissions to the token. Scroll down to the end of the page and click Generate token.

The token’s secret will display on the next screen. Make sure you copy it and place it somewhere you can refer back to it later such as a text editor like Notepad - we’ll need it again in the next section.

TravisCI Configuration

Once we have our Travis account set up, we need to add in a configuration file that Travis will read from to determine what steps we’d like it to perform.

In our source code repository (jdheyburn.co.uk in my case) we want to create a file at the root directory and call it .travis.yml. See below for an example of how I configured mine.

Let’s break it down section by section.

Build Environment

These settings here all refer to the build environment that we’d like our project to build on.

dist: xenial

git:
  depth: false

env:
  global:
    - HUGO_VERSION="0.58.3"
  matrix:
    secure: REDACTED

dist specifies what platform Travis should build the project on. In this case xenial refers to Ubuntu 16.04, which is a Linux distribution. There are several others to choose from and more likely than not you’ll want the platform to be Linux. However if you had a Windows application written in .NET then you would likely want it built on a Windows Server since that is what supports it.

git.depth tells Travis how many commits of your project to check out. This is passed directory to the git parameter --depth (more info on that here). For our use case we’re not interested in this option so we set it to false to disable the flag being passed to git.

env.global allows us to define what variables should be set in the environment. This is done in the form of an array of strings in the format key=value. So given the example, HUGO_VERSION will be set to 0.58.3. We’ll come back to this later.

env.matrix is the encrypted value that gets passed to the GITHUB_TOKEN environment variable which is used to allow Travis to commit the built project to our GitHub Pages repo.

You’re going to want to take the personal access token generated from GitHub in the earlier step and encrypt it using this method, then add it back to this setting

External Dependencies

Once the build environment is defined, we can tell Travis to pull in some additional dependencies or files required for our project. Now remember this is a hugo project and we needed to install it on our local machines to run deploy.sh, we need to do the same for Travis too.

install:
- wget -q https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}/hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz
- tar xf hugo_extended_${HUGO_VERSION}_Linux-64bit.tar.gz
- mv hugo ~/bin/

Remember we defined HUGO_VERSION earlier? This is where it is called back again. In order, the steps we’re performing are:

Downloading the archive containing the specified hugo version
Extracting all contents of the archive
Moving the hugo binary to the ~/bin/ directory
- This directory is on the build servers PATH, which enables us to execute the binary using just the hugo command later

Previous Issues

In a previous version of hugo I used, there was an additional dependency I needed to include. The hugo-coder theme requires to be built with Hugo Extended since it requires Sass/SCSS support.

For this particular Hugo Extended version, it required a library which was not included in the build server distribution in the past. On the plus side - Travis allows us to define additional build steps to ensure all required libraries are on the build server beforehand.

before_install:
  # This workaround is required to avoid libstdc++ errors while running "extended" hugo with SASS support.
 - wget -q -O libstdc++6 http://security.ubuntu.com/ubuntu/pool/main/g/gcc-5/libstdc++6_5.4.0-6ubuntu1~16.04.10_amd64.deb
 - sudo dpkg --force-all -i libstdc++6

However in later versions of Hugo (including the one I am using today) this dependency is no longer required, hence why it is commented out in .travis.yml.

You may not need this stage in your pipeline, I know my project no longer requires it. However this may come in handy later knowing that you have the option of specifying more pipeline steps if the build distribution you’re using requires some additional dependencies.

Build Script

Now onto the juicy stuff - building the project. This is pretty much where the deploy.sh starts from, since on our local machines we already had the hugo binary installed.

script:
- hugo version
- hugo --gc --minify
- cp CNAME public/

Not a lot is going on here, but to detail what each step is doing:

Print out the version of hugo being used
- This is helpful for debugging the build. By printing out the version used we can try to replicate the bug locally for troubleshooting.
Build the project
- We do this with two flags, --gc and --minify. These weren’t defined in the deploy.sh script we used earlier so let’s cover them here.
  - --gc tells hugo to cleanup some unused cache files after the build
    - This helps keep a tidy environment for the build server.
  - --minify performs minification on your website to reduce the size of the generated content, enabling it to load faster on your users’ devices
    - Coupled with a CDN like in the previous part, your website will load almost instantly
Copy the CNAME file from the project root to the generated public/ directory.

This ensures that the custom domain name we set up in the previous part continues to be set in the generated website code that gets pushed to our GitHub Pages repo. See here for a refresher.

This section is effectively the ‘CI’ part of ‘CI/CD’. We could stop here and just use Travis as a build server to determine whether the website is able to be built successfully. Any errors in the pipeline would have resulted in a failed build.

The next section details the Continuous Deployment, ensuring that on the successful build of a project we deploy it to our production environment.

Deploy to GitHub Pages

Once the project has been built, we need to push it to the GitHub Pages repository. Travis has good documentation on this. My setup follows the below:

deploy:
  provider: pages
  skip-cleanup: true
  github-token: "$GITHUB_TOKEN"
  keep-history: true
  local-dir: public
  repo: jdheyburn/jdheyburn.github.io
  target-branch: master
  verbose: true

For the last time, let’s walk through what each of these is doing.

provider tells Travis this is a GitHub Pages deployment
skip-cleanup set to true, so that Travis does not delete the build before we’ve got the chance to upload it
github-token is set to the environment variable $GITHUB_TOKEN which was set for us earlier on in the build environment.
- This is passed to the provider so that it has valid credentials to push the code to the GitHub Pages repo.
keep-history performs an incremental commit against the project
- Setting it to true allows us to view back the changes in the commit history such as this one
local-dir specifies the directory that should be pushed to the target repo
- We set it to public because that is the name of the generated website directory from the previous script step
repo is the target repo where we should be deploying to
target-branch is the branch that we want local-dir to be pushed to
- For our setup we are using master
verbose specifies how much detail Travis will log about its deploy activities

That concludes the configuration section. Remember to change it or add in some other functionality needed for your project and save it to .travis.yml.

Bringing It All Together

Now you’ve completed your config file, let’s check it all in to GitHub.

git add --all
git commit -m 'Adding CI'
git push

Travis will automatically detect that a new change has been made against your repo. And with the inclusion of .travis.yml, it will now work against that file.

You can view the status of your repository’s build by navigating to it from the dashboard and clicking on it.

If any of the commands in the script section of the config return a nonzero status code, the build will fail. If this happens then have a look at a build log and investigate the issue. More likely than not someone else has encountered the problem before, so Google is your gatekeeper to solutions!

Once you’ve got a successful build, that means your website has been deployed to the GitHub Pages repo and will be available on the web to view. In a browser you can now view your changes on the website.

If they aren’t there then your browser is most likely caching an older version. By default, websites served by GitHub Pages have a browser cache TTL set to 10 minutes. So you can either wait 10 minutes, or clear your browser cache!

Conclusion

You’re on the way to DevOps masterclass. As mentioned the next post will focus on migrating to GitHub Actions so if you’re happy with TravisCI then there’s no more you need to do! Go ahead and add other repositories you may have to Travis and build up multiple pipelines 🙌

On Becoming An Open Source Software Contributor

Sun, 26 Jan 2020 00:00:00 +0000

This is going to be a relatively shorter post that’ll be in two parts since it’s more of a self-promotion… But as of this Friday just gone, I finally had a pull request approved to be merged to terraform–provider-aws!

Okay so it’s only a small piece of documentation, I’ve actually got another larger PR waiting to be merged in to the same project with new added functionality - and a bug fix in the pipeline too - which I hope to talk about in the future at some point. Although I can’t say I’m surprised this one got approved first - since it is by far the simplest change, and increased documentation is rarely a bad thing to add to a project.

Background

To talk a bit more about the context behind it, AWS SSM Patch Baselines is used to define the rules of what patches should be installed on your EC2 instances.

For patch baselines tied to Linux instances, you can only define what patches should be installed at the OS level. However for Windows instances - in addition to defining OS level patches - you can also define Microsoft application patches to be installed too!

While this can be done in the AWS Console easily enough, it is very common to define these resources as infrastructure-as-code (IaC), for which there are several to choose from. For my team we use Terraform - and we wanted a way to be able to replicate the patch baselines we could create in the console into Terraform code too.

After some digging I came across this issue on terraform-provider-aws that indicated the functionality was missing from the provider. When in fact the support had been there all along, since the AWS provider is merely a wrapper around aws-sdk-go which calls the necessary APIs to create the resources. Where Terraform comes in is generating the dependency graphs for said resources and creating them in order asynchronously.

As you can see from the ticket, I was suggested to improve on the documentation - which leads us to where we are now. It is still pending release, however when it is out I’ll link it below.

In the meantime, let me talk about the importance of documentation in the next section.

Documentation Case Study

I’m a bit of a sucker when it comes to documentation. Yes everyone talks about how important it is to include it not just open source projects but also in your day-to-day job. A whole lot of the time there is far too much knowledge-retention and not enough knowledge-sharing.

This reminds me of the character Brent from the novel The Phoenix Project - who doesn’t share a lot of the domain knowledge he experiences. FWIW I highly recommend the book!

In fact, documentation isn’t always necessarily for sharing knowledge with others, it is also about sharing knowledge with yourself.

What do I mean by that?

Documenting down architecture, processes, runbooks, etc., can help to give you clarity into the subject you are writing about. Once you’ve extracted all the details and are able to view them at a high-level, you’re able to iron out any creases and make improvements from there.

However, more importantly, and one that is most beneficial to me is that…

You will forget that shit and will need to reference it back again!

Not everyone can have the memory of an elephant…

We are only human right? There is no way we can remember what most of us did a few weeks ago let alone a project from 6+ months ago.

Example

As an example of this a few years ago at a previous employer, I worked on a project to build a number of microservices that acted as an onboarding layer to integrate between internal corporate services (e.g. user entitlements, processes that user had access to, etc.) and AppDynamics.

I laid down all the flows, the design decisions, the architecture, the models, etc., on my documentation tool of choice (currently it’s Confluence) as a place of reference should either my team, or myself need to refer back to it later on.

A few months later I took an internal move to another team within the same firm. In this new team I picked up a whole load of new information, of which there was no way I was able to retain everything all at once.

Later on when my former teammates wanted to add an additional feature to the previous project, or wanted to troubleshoot a particular bug and wanted some context behind why things were operating the way that they were, they would reach out to me.

At that point I directed myself and my former teammate to the documentation page for a refresher, and to let them know where they can go to solve the issue without me.

Without that documentation I guarantee that I would’ve spent a whole load more time on trying to recollect my thoughts, only to waste the time of my teammate. If there was an ongoing incident too, then the time to resolution would be hastened too.

Conclusion

Don’t be that person to retain knowledge, it’ll end up coming round and biting you in the arse.

After you’re done reading this, make a mental note to write down any process or piece of information you come across. Even if you’re doing a task or exercise for the first time, document your steps to how you got there - you’ll certainly need it in order to replicate the steps in future.

Make a small start now that will make a big difference later on.

Who Goes Blogging 2: Custom Domain

Thu, 12 Dec 2019 00:00:00 +0000

In the previous post, we got ourselves up and running with a website generated by Hugo, deployed to GitHub, and hosted by GitHub Pages.

Now, we’re going to add a custom domain to our website so that we hide the <username>.github.io domain that GitHub Pages is kindly hosting for us for free.

At the same time, we’re going to make our website blazingly fast for users by adding a caching layer with a content distribution network (CDN).

Lastly, I’m going to throw in a bonus guide on how to redirect from multiple top-level domains (TLDs) to one (e.g. <your-domain>.com redirects to <your-domain>.co.uk).

Pre-requisites

There’s not much more to add from the last post. I talk a lot about domains and domain name system (DNS), which is the address-finder of the Internet, and an entirely huge beast in its own right.

Again - I won’t try to replicate already great guides out there on the topic. So if you’d like to find out more, see below for some helpful guides.

Applying a Custom Domain to GitHub Pages

Let’s add a degree of professionalism to our site by having a custom domain apply to it. You’ll need to make sure you own a domain first before you go ahead, so have a look at a few providers and see which works best for you from a comparison list. I bought mine from namecheap just because of the price and WhoisGuard features. There may be other providers that have the same features, so make sure to make your own comparison!

In the case of .co.uk domains, because it is a UK domain that resides in the EU (for now), the WhoIS lookup is disabled by default - which is a huge win for privacy. WhoisGuard is available for non-EU domains and I highly recommend it.

GitHub has a series of documentation on applying a custom domain to GitHub Pages in much greater detail than what I am about to write out, should you wish to find out more information.

Acquire a Domain

The rest of the post will depict a lot of Namecheap semantics, since that is the registrar I have access to. You can choose to follow the guide alongside a different registrar if you wish, at a high-level they will be pretty similar. For now, let’s move on ahead with Namecheap and navigate through to the domain purchase page.

The hardest part is deciding on the domain name…

Once you’ve set up an account and purchased your domain, your accounts domain landing page will look something like the below.

Make sure you have auto-renew selected, otherwise you can kiss that domain goodbye when it expires!

When you purchase a domain from Namecheap, by default it will be pointing to their own domain name service (DNS) nameservers, as you can see from the picture above.

This means when we type in our new domain into a browser, it will contact Namecheap for the IP address for that record.

Currently Namecheap is none the wiser about these records, which isn’t very exciting. Let’s move on to adding the CDN for the website.

Adding Our CDN Layer

As discussed in a previous post, a CDN can provide us with many benefits. Go check out the page for a refresher of what those are and for why I selected Cloudflare. You can use whichever you like, however the remainder of this guide will focus on Cloudflare in particular - the concepts can still be applied at a high level to other CDNs.

Like with any introduction of an architectural component, a CDN has some drawbacks, such as making your service now dependent on a third party for which you have no control over. Namely Cloudflare in particular has had some high profile outages of recent date, but has been extremely reliable in my previous experiences with them.

You can go here for a list of pros and cons of CDNs. Given this information I believe you can make your own mind up on what is best for yourself. For me, I the benefits far outweigh the downsides.

Cloudflare our Domain

Create an account with Cloudflare if you haven’t done so already. Once done you’ll need to click Add Site at the top of the browser dashboard. Enter your newly purchased domain from the previous section.

Enter your newly purchased domain from the previous section

Go ahead now and select the free plan, if you want to go more advanced then you can do so.

Once that’s created you’ll see Cloudflare scan the DNS records for this domain you’ve added - for now let’s navigate back to the Cloudflare Dashboard and selecting the domain. You’ll be presented with a page similar to below, minus all the activity!

Name me a more iconic duo than numbers and graphs. I’ll wait…

Update Cloudflare DNS Records

In order for Cloudflare to provide its benefits, it acts as the DNS server for your domain. This means that it will forward requests of our website to the IP addresses where our website is being hosted. It’s place in the topology is like this:

1. Domain registrar ---> 2. DNS server and CDN provider ---> 3. Web server location

Translated to our architecture:

1. Namecheap ---> 2. Cloudflare ---> 3. GitHub Pages

What we need to do is instruct Cloudflare where to direct clients of the website. There are several ways of doing this, whether you want your website to be available at www.<your-domain>.co.uk (CNAME record), or <your-domain>.co.uk (APEX (A) record). Both of which are well documented by GitHub already. For the purpose of this post, I’ll guide you on my set up which is an ALIAS record.

Back to Cloudflare, you’ll need to load up the dashboard for your domain and navigate to the DNS icon in the taskbar at the top. Clicking on Add record will allow you to add the A records that point to GitHub Pages’s IP addresses. As of writing (and documented) they are:

185.199.108.153
185.199.109.153
185.199.110.153
185.199.111.153

Let’s add these to the Cloudflare DNS page below.

Add in the GitHub IP addresses one by one

Once done - your records will look something like this:

You’ll notice an additional CNAME record at the bottom for www. This will redirect any requests made to www.jdheyburn.co.uk to jdheyburn.co.uk. This could be something you’d want to replicate too if you wish.

One thing to note down before we move on is to capture the Cloudflare DNS Nameservers that have been assigned to our domain. You can find these on the same DNS management page we are on, but by just scrolling down we can see these nameservers.

Make a note of these nameservers for your domain

Direct Namecheap to Cloudflare

Earlier in this post, I mentioned that a newly created Namecheap domain will default to their own DNS nameservers. We want to change this to Cloudflares DNS nameservers from which we configured our DNS records.

Navigate to the Namecheap management page for your domain and enter the Cloudflare nameservers once you have selected Custom DNS.

Add in your Cloudflare domains from previously

Once this is done you may need to wait a while for the DNS updates to propagate throughout the world. While we’re waiting for that, there’s one final piece of the puzzle which can keep us busy.

GitHub Pages Configuration

The last place to configure is GitHub Pages. Currently it is hosting at our .github.io domain, but we need to instruct it to redirect to our custom domain. There are two methods for doing this:

Configure the repository settings
Use a CNAME file in your repository

Configure the repository settings

This is the quicker of the two solutions, so I advise to follow this step to understand if you have everything in place correctly. Once done then you can lock-in your changes with step 2 above.

For this step, you need to navigate to the settings pages for your .github.io repo containing your rendered code.

https://github.com/<USERNAME>/<USERNAME>.github.io/settings

Now scroll down, keep going until you hit the GitHub Pages heading. Here you will see a form for entering a custom domain; do the honours and enter it in like below.

Ignore my already published domain…

Now again wait for DNS to propagate across the world. When GitHub is happy with the changes then you will see the green banner similar to the one in the screenshot above. This means everything is being served up! Why not give it a try ourselves? Head to your domain now and see if everything is working!

You should end up with something like this!

Solidying our changes with a CNAME file

We can use the above method to quickly try using GitHub repository settings to see if everything is working, however I’m a big fan of setting changes in code (Infrastructure-as-code anyone?). GitHub supports another method which is to use a file named CNAME in our generated .github.io repo that contains the domain name we wish to use.

In my case, I would have the following…

# CNAME
jdheyburn.co.uk

This then tells a repo that is enabled for GitHub Pages to use the domain in this file as our custom domain, effectively producing the steps in the previous section. Neato.

The change to implement this is fairly easy. I have to admit I picked it up from somewhere but I don’t have the source to reference it to.

So back in your blog-source repo, you want to execute the below, replacing the template with your domain.

echo "<DOMAIN-NAME>" > CNAME

Building into our deploy script

Once that is done, you will need to modify your deploy.sh script to copy over the file into your generated Hugo site, because Hugo won’t do it for you! Don’t have the deploy script or need a refresher? Head back to the previous post.

You will want to copy it after Hugo has done its thing, take a look at the Gist below - line 9 is your friend.

What this is doing is taking the CNAME file that already exists in our blog-source repo and moving it to the generated public/ directory which Hugo created for us. It is then this public/ directory that gets committed to the GitHub Pages repo.

Then that’s it! Take a look at my finished repo and you’ll see where CNAME fits in.

Bonus: TLD Redirection

Let me tell you a story. Your website is up and operational. You’re super proud of it, and you give yourself a round of applause 👏

But you don’t want to be the only person looking at it, you want the whole world to! You tell your parents, your significant other, the dog off the street - they all remember the name of your website, but was it at .com or .<insert snazzy TLD here>?

Of course, you domain isn’t at .com, that’s boring as hell! You just forked out $50 on a .dev TLD, and no one will see it!

Luckily there is a way…

You can buy additional domains at different TLDs and have them redirect to your one-domain-to-rule-them-all with little to no hassle! There is the cost of purchasing the domain and renewing it year after year, but with that being ~£10 or so per year - I’d call that a good insurance policy to ensure people land at your website!

In my case, I purchased jdheyburn.com and had it redirect to jdheyburn.co.uk - why not give it a try: https://jdheyburn.com

The steps are already defined in this post. For a breakdown of what they are:

Steps 1 and 2 are pretty easy to perform yourselves - so for this exercise I’ll join in at step 3.

Cloudflare Domain Redirection

Assuming that you’ve completed steps 1 and 2, we’ll need to configure the redirection - but we don’t do this via inserting a DNS record like we did previously, Cloudflare has a feature that handles that for us called Page Rules.

On your Cloudflare website landing page, navigate to Page Rules in the toolbar at the top and then Create Page Rule.

In the next screen we’re going to define the rule. Cloudflare have documentation on Page Rules, and even more specifically on redirection.

From this screen you will want something that appears as below.

Let’s break down what’s happening here:

We define a pattern of *jdheyburn.com/*, indicating that the rule is active should a URL be queried to Cloudflare

This pattern will match any request at this domain

Next comes the rule action, which is:
Set this URL as a forwarding URL - to reply back to the client with 301 Permanent Redirect status code
With the forwarded URL being https://jdheyburn.co.uk/$2

The /$2 at the end of the rule is crucial here. This will carry across any URL path parameters or query parameters to the redirected URL. In fact $2 refers to anything that matches the second asterisk (*) in the rule pattern (*jdheyburn.com/*). So https://jdheyburn.com/contact will redirect to https://jdheyburn.co.uk/contact.

Having the forwarded URL set to the https:// scheme will ensure that your users will receive your website encrypted, safe from man-in-the-middle attacks.

Lastly, click on Save and Deploy to finalise your changes; you’ll have a view such as below.

That’s all you need - once again you will have to wait for DNS replication to trickle down. You can repeat this over again for other domains you may have.

Conclusion

That it for this part, we covered quite a lot of things:

Purchasing a custom domain
Applying a CDN cache layer
HTTPS redirection
Redirect multiple domains to one location

Next up I’ll document the various deployment methods I have used for the website.

Who Goes Blogging 1: Getting Started

Sat, 09 Nov 2019 00:00:00 +0000

I’ve done a lot of talking about how this website is currently, but let’s talk about how you can get set up with the same as I have done here. Let’s recap exactly what that is:

A GitHub repo with the site source code
Another GitHub project with the rendered site
Hosted on GitHub Pages
Fronted by a custom domain
Globally cached by a CDN
Redirecting multiple domains

We’ll cover the first three points in this post, with the remainder to come in a follow up post.

Pre-requisites & Assumptions

These series of posts is aimed at between the newcomer, to intermediate programmer. There may be some concepts which I won’t be covering to due brevity, and because there are several guides out there who have explained these concepts better than I could.

Having said that, I assume you are comfortable with the following:

Navigating your way through a terminal
Happy with the basic set of git commands
Understand how website requests are made (though this isn’t crucial)

Hugo Static Site Builder

There are many website building platforms out there for your static sites. There is no one platform to rule them all, you should decide on which one works best for you. For myself, I wanted to explore further into the realm of Golang as a learning opportunity. I’ve built websites from scratch before from basic HTML, CSS, and JavaScript all the way to using frameworks such as React and Angular.

However while using these frameworks provide you with the highest flexibility in terms of customisability, they can take a while to get something out there and hosted, when really you just want something to generate the HTML and CSS for you and allow you to focus on the content.

That’s where static site frameworks come in. They usually have some opinionated folder structure and some nuances on how things in a website should be, but comply by these rules and the website will be generated for you! Once such example is Hugo - which is what this website is built on; it’s opensource and written in Golang, you can even check out the source code to see how it works!

While Hugo itself is free and comes with a bunch of themes you can use for free, there are fancier and more feature-rich themes available to purchase from designers which may have a cost attached to them.

Hugo isn’t the only static site framework out there, there are plenty more too each with their own pros and cons. I won’t aim to reproduce a complete list of comparisons against other platforms, other people have already made the comparisons much better than I could! But nevertheless, some that I am aware of off the top of my head are Jekyll, Ghost, and the infamous WordPress - so investigate using your search engine of choice if you wish to know the differences.

UPDATE 2019-11-30

I came across this thread on HackerNews which ultimately lead me to these comparison websites for static site generators:

https://www.staticgen.com/

https://staticsitegenerators.net/

All set on Hugo? Awesome - let’s get started on some project foundations.

Git Set Up

I mentioned before I split my website into two repos:

jdheyburn.co.uk holds all the source code
jdheyburn.github.io holds the rendered website

For the first repo, it doesn’t really matter what you call it - you can call it your destined domain name, or blog-source, or dogs-are-great. It’s the second one which you will need to think about, where it must be <YOUR_USERNAME>.github.io - which will ultimately be made available at https://<YOUR_USERNAME>.github.io.

Create two empty Github repos now, then clone both to your environment. If you don’t have git available in your terminal, check out this guide to get set up.

mkdir ~/projects
cd projects
git clone https://github.com/<USERNAME>/blog-source.git
git clone https://github.com/<USERNAME>/<USERNAME>.github.io.git

Now that we have the repos set up, let’s get started on building the website templates.

Blog Bootstrapping

Before you get started, you’re going to need to install Hugo - head to this page for instructions on going so. Once done continue on below.

Firstly you would need to create your Hugo template. You can do this from either executing hugo new <site|theme> and building up from there - or do what I did which was to browse the Hugo themes and git clone the example site for the chosen theme and then make your changes around that. It may entirely depend on your learning approach which way works best for you.

In my case, I wanted to get up and running in the smallest time possible (isn’t that the point of static site generators?), so I followed the approach above.

Serves Up!

The theme this blog uses as of publication is hugo-coder, written by Luiz de Prá. You may wish to use it, or something else. It’s entirely up to you! Let’s get started by laying down the foundations and cloning the theme.

cd ~/projects
git clone https://github.com/luizdepra/hugo-coder.git
cp -r hugo-coder/exampleSite blog-source
mkdir blog-source/themes
ln -s ~/projects/hugo-coder blog-source/themes/hugo-coder
cd blog-source
hugo serve

That last command will locally serve the example site, so that you can view it at http://localhost:1313/.

Success!

Experimenting

Once you have the site hosted locally, feel free to make as many changes as you like to gain an understanding of how everything plugs together.

The blog-template/config.toml file will contain the majority of configurations that are used to generate the site. Go ahead and even comment out some config and see what affect that has. You can even be nosey at my config file if you’re looking for some inspiration! The hugo serve command will watch all files in the directory for any changes you make, rebuild them, and refresh your webpage too with the changes - making for hasty development!

The Hugo config file will be a source of many informations, and the documentation for it is very thorough. While we’re on the subject, the rest of the Hugo documentation is great, so check the rest of it out if you haven’t done so already - particularly how the directories are structured, and how content is managed.

I advise you now go off and explore all the customisation options for your site, when you’re ready to have it deployed, continue on reading.

Theme Git Submodules

When we were setting up the exampleSite locally, we created a symlink from ~/projects/blog-source/themes/hugo-coder -> ~/projects/hugo-coder. This is okay for local development, however not particularly the best practice in the real world. We want to take an existing git repo (hugo-coder) and apply that to a directory in our project, but what we don’t want to do is duplicate the code in our own repo.

This problem is solved exactly by git submodules, and we define it within the .gitmodules file of our source code repo.

# .gitmodules
[submodule "themes/hugo-coder"]
	path = themes/hugo-coder
	url = https://github.com/luizdepra/hugo-coder.git

Now when this file is checked into the repo, any future clones will also include the hugo theme as a submodule in the themes/hugo-coder directory.

Once you’ve done this, your source code is all set! Why not share it with the rest of the world?

Releasing into the wild

So far we have only been playing with the hugo serve command, which is great for local development but not for production. There is a more appropriate command for building hugo projects - aptly named hugo; pretty simple right?

hugo

This renders the HTML and CSS files from your config and markdown for the theme and places them in the public/ directory of your source code repo. In theory once you’ve executed this command you can host a webserver at that address and everything would operate as normal. Why not give it a try?

cd public/
python -m SimpleHTTPServer 8080

We can even go one step further by adding flags to the hugo command, such as --minify which will minimalise all the supported files into a much smaller size, resulting in faster load times for your users.

Since we are still in our source code repo, we want to make sure that public/ directory doesn’t get included in future commits. This is because the blog-source repo should be entirely for source code, our other repo is the one that holds the rendered code.

For this we can utilise a .gitignore file, instructing git to ignore any files that match the terms in the contents. Let’s get one created now and check our code into the repo.

echo "public/\nresources/" > .gitignore
git add .
git commit -m 'Initial commit'
git push

We’re currently committing to one repo, however we want GitHub Pages to host it for us. As mentioned earlier, GP can only host repos at the domain github.io, so we need to get our rendered website into that repo.

Bash Script Deploying

Now the original way I did this was through a simple bash script which largely followed this process as documented by the Hugo team. Note that this is largely the process we have followed thus far, with the exception of step 6 onwards - so ensure you follow those steps.

I’ve since moved over onto a CI/CD platform which I will discuss in a future post.

After following the process in the link prior, you should be able to invoke your deploy script at ./deploy.sh, which will push the built public/ directory to your <USERNAME>.github.io. GitHub should pick up that this is a GitHub Pages repo and have your site ready for you at https://<USERNAME>.github.io!

Conclusion

In this post, we have done the following:

Created two GitHub repos:
- one for storing the source code for the website
- another containing the rendered web pages
Built Hugo locally for development using hugo serve
Written a script for deployment
Deployed to GitHub Pages

Next up, we’ll be adding a custom domain to the site, front it with a CDN, and redirect multiple domains to it.

Who Goes Blogging 0: Applying Cartography

Wed, 11 Sep 2019 00:00:00 +0000

Applying Cartography

With my academic background focused in infrastructure, I love seeing diagrams of topologies - they’re a pretty damn useful way of understanding architecture of an application flow amongst other things. Let’s take a look at how my portfolio site is architected out.

As I mentioned in my first blog post - I’ve got several ideas on how I can improve on the architecture of this site. But what good is evaluating where you’ve come from if you don’t document what you currently have?

That’s when you can make true comparisons in any system. As such the aim of this series of posts will be to explain how this website is architected, so that we may reference it in future posts. Later on, I’ll also talk about how you’re able to get yourself set up as well.

The Setup

Because we all love an architecture diagram, let’s slap one in now.

Wow. That is… really not much at all. Apologies if you were expecting a lot more arrows and boxes!

In a way, I shouldn’t be sorry, because one of the core priniciples in software development is KISS (Keep It Simple Stupid!) - and this setup is just an advantage of how I decided the first implementation of my website to be:

Easy to implement
Easy to maintain (there is none required!)

I’m a firm believer that you’re best off getting a minimum viable product out there ASAP and then work to perfect it afterward. What I hope to document as time goes on are the various enhancements and changes that I intend on making to the website so that yourselves can follow along too with your own site. That goes not just for this site (how much can it really be improved?!) but for whatever projects I work on.

So what does it all mean?

Back to the diagram, let’s follow it from right-to-left and start talking about what’s going on.

GitHub Pages - somewhere to call home

You can see that the site is hosted on GitHub Pages. Essentially this is GitHub’s platform for hosting the static resources that are checked into the public repos. This then presents the assets at a domain prefix of your choosing, suffixed by .github.io - in my case it is at jdheyburn.github.io. GitHub Pages only hosts your site via HTTPS so you know your pages aren’t being subject to a man-in-the-middle attack, and it takes care of your TLS certificates so you don’t need to worry about renewing them!

You might be wondering why you would need to serve a static site over HTTPS when you aren’t handling anything confidential. I’ll turn you to Troy Hunt’s excellent article Here’s Why Your Static Website Needs HTTPS.

Supercharge your delivery!

Now that the website is hosted at GitHub - I stuck a content delivery network (CDN) in front of GitHub for a whole load of reasons, some of which are:

Fast end-user response times

CDNs cache your content at edge locations dotted all over the globe, where the client is then served content from the one closest geographically (let me direct you to latency numbers every programmer should know).

Security

The CDN acts as a proxy between the client and your servers - their requests don’t actually hit you directly.

Not only that, a lot of CDNs provide DDoS protection to ensure excessive requests don’t bring down your servers.

Analytics

Because everyone loves graphs and numbers… right??… Alright just me then…

You’ll be able to see how many requests are being served up, where requests originate from, amongst others. Take a look at an example below.

Custom domain

This means you can access your site on something other than <my_site>.github.io, which is great for when you move off GitHub Pages to another platform, you can just tell Cloudflare to source requests from another server
It also gives your site that extra polish and professionalism about it - wouldn’t you agree?

My CDN of choice is Cloudflare, for no reason more than is it completely free to use and will most likely stay free until I decide I would benefit from the next pricing step. Did I mention the website is pretty damn easy to use as well?

That means that the operating cost of a site like this is exactly ZILCH (nada, et al.), which again is another great reason to get set up on a platform like this.

Top-level domain fatigue?

I’m from the UK - and I wanted my site to reflect that, so I purchased a domain with the top-level domain (TLD) .co.uk. However when users come to visit me on my site they may not always remember whether it was .com, .dev, .tk (remember those?). Therefore I also have jdheyburn.com set up in the same way as its .co.uk sibling to maximise that user experience.

Okay well that means there is a slight cost to maintain the site through purchasing and renewing the custom domains (notice how jdheyburn.com redirects to jdheyburn.co.uk?) but we’re talking in the £10s per year for these two.

So if you’re interested in having a setup like this, then over the next few posts I’ll be detailing how you can do the same.

- jdheyburn

Three Ways To Spice Up Your Python Code

Fri, 05 Jul 2019 00:00:00 +0000

Spice Up Your Life Python 🐍

I’m currently working on a side project that I’ve written in Python. Now I’ve have a lot of experience with Python before, I used it primarily to write a load of scripts for automating processes for when Bash didn’t quite cut it. Don’t get me wrong Bash is great, but like for every other language out there, each has its purpose in the world.

My experience with Python didn’t go beyond setting some variables, hitting endpoints, executing some Bash commands to install some vendor components. All of which is what makes Python so great right? Implying on types is useful for some cases. In fact this reminds me of a tweet I saw, I can’t find it again but it went something like this:

Stages of learning programming:

Learn a typed language, such as Java - complain at its complexity

Learn an untyped language, such as JavaScript or Python - marvel at its simplicity

Get frustrated at implied types in step 2

Revert to step 1

During said experiences I came across some poorly written scripts whiched triggered some pet peeves: what object a function is expecting, or what it returns? Did I break some downstream function? This made it terribly difficult to add new functionality to scripts.

With this side project I wanted to make it right from the start. So I said to myself these are the key areas I want to target:

The application must be extensively tested (did I mention I love tests?)
Functions must clearly define what the type of the objects the parameters are, and what the types of the objects they return are.
~~Best~~ Practices are upheld throughout the way

From this list, we can include the following to solve the above:

Tests

Alright so I know this one is a given. But in all honesty I’ve never really tested Python code before. Why so? I found it difficult to mock API calls and the effort required for the initial learning curve didn’t seem to pay off for the tiny script I was automating. Given the scope of my side project is rather large, this is a great opportunity to learn. Let’s start with a basic function:

# scratch.py
def increment_int_by_one(int_to_inc):
    incremented_int = int_to_increment + 1
    return incremented_int

While this is something easy enough to test by doing so manually, at some point its functionality may increase, and at that point we would require automated tests to ensure its original feature-set was unchanged.

Implementation

There are several testing frameworks out there, but for my use case I am using the built-in unittest which is easy enough to use. A basic structure of a test file is seen below:

# test_scratch.py
import unittest

import scratch


class TestIncrement(unittest.TestCase):
    
    def test_increment(self):
        """
        Should successfully return int value 2
        Given a parameter of int value 1
        """
        expected = 2
        int_to_inc = 1
        actual = scratch.increment_int_by_one(int_to_inc)
        self.assertEqual(actual, expected)


if __name__ == '__main__':
    unittest.main()

There’s a lot going on here, let’s break it down by section.

Imports

import unittest

import scratch

Here we are specifying the modules that the test file is going to interact with. unittest is the testing framework and scratch is the Python file containing our business logic, as written earlier.

Class Declaration

class TestIncrement(unittest.TestCase):

Simply enough, this is a class that extends the unittest.TestCase class, for which then unittest can then execute all the test methods defined within it, and you can name them like class TestXXXX. There is no limit to how many classes you can have, but I like to group my TestClasses together in terms of what logic they are testing.

Test Methods

def test_increment(self):
    """
    Should successfully return int value 2
    Given a parameter of int value 1
    """
    expected = 2
    int_to_inc = 1
    actual = scratch.increment_int_by_one(int_to_inc)
    self.assertEqual(actual, expected)

These methods are the real juicy bits of your tests. Here you are defining the parameters that are going to be used by your functions, and hitting the logic you are testing. For each test that you write you will alter the test name, description, expected values, and input parameters accordingly to what you are testing.

Main Method Invocation

if __name__ == '__main__':
    unittest.main()

To close off nice and easily, this will tell Python to invoke all test classes that are described within unittest.TestCase.

Execution

Now just execute the test file and it will now run as expected:

> python -m unittest test_scratch                                     
.                                                                     
----------------------------------------------------------------------
Ran 1 test in 0.001s                                                  
                                                                      
OK

A nice little touch is that we don’t even need to provide -m unittest to the Python intepreter since we have defined the main method invocation:

> python test_scratch.py                                              
.                                                                     
----------------------------------------------------------------------
Ran 1 test in 0.002s                                                  
                                                                      
OK

Static Type Checking

As I mentioned before, Python is great for throwing together a script to automate some task. It’s quick and easy to do mainly because the type of a variable is implied from whatever you set to it.

>>> my_int_var = 1
>>> type(my_int_var)
<class 'int'>

This also applies to functions that take in a parameter, the type for it is implied from what it receives! Let’s revisit our incrementing function from earlier:

def increment_int_by_one(int_to_inc):
    incremented_int = int_to_increment + 1
    return incremented_int

>>> print(increment_int_by_one(1))
2                            
>>> print(increment_int_by_one(100))
101

All good so far. But what happens when we pass in something that is not an int, such as a string?

>>> print(increment_int_by_one("1"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in increment_int_by_one
TypeError: must be str, not int

Yikes, that’s a misleading error! Having a look at the error straight away, you’d be confused at ‘what’ must be a string, not an int - didn’t you accidentally provide a str? So why is it telling us it must be a str and not an int?

What this is actually error-ing on is how an int is being concatenated to a str on the left-hand side of the + operator.

Under the hood, Python believe you are trying to do a string concatenation because the parameter is of type string, and is on the left-hand side of the + operator. The below shows a valid way to perform a string concatenation:

"1" + "1" # -> "11"

If the int value was on the left-hand side of the operator and you passed a string, you’d get something like this:
>> 1 + "1"
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

But changing the right-side of the operator won’t fix this for us. What we need is to implement typing.

Implementation

Python 3.6 onwards introduced static type checking, so make sure you upgrade to it if you haven’t already - which you might want to do very soon as Python 2 is EOL in 2020!

Taking our above increment_int_by_one function, we can add : int to the parameter which will tell the function what type it expects the parameter to be.

def increment_int_by_one(int_to_inc: int):
    incremented_int = int_to_increment + 1
    return incremented_int

Now when we pass in a str as the parameter, we still get the same error we saw before. However, if the IDE you are using supports it, you will receive type hints!

I’m using VSCode with the Python extension installed too, you can see the hint that appears includes the type of the parameter.

VSCode also has the ability to check the type of the value you are passing into the function:

Here we have a visual indicator that the string is incompatible with the int type of the function. Alongside this, in the Problems tab we have a full explanation on what has gone wrong. This is both provided by the pyright extension which can be found in the VSCode Extensions section.

Return Types

So all of that just describes how typing works for parameters, but how can we specify the return type of a function? That is done easily enough too. Let’s extend on our increment function from earlier.

def increment_int_by_one(int_to_inc: int) -> int:
    incremented_int = int_to_inc + 1
    return incremented_int

Note the only change here is the -> int which specifies the type it is returning. Now our type checker will highlight to us when we violate this in two scenarios:

The IDE has shown us that the function is expected to return a type different to what it is actually returning…

And in this snapshot, we are trying to assign a variable of type str to the output of the function which returns an int!

Another pattern you might see is if a method does not return anything (in Java-speak, it is void), an example of this is if the method is a constructor for a class.

def __init__(self) -> None:
    self.attribute = 'DEFAULT_VALUE'

Class Objects

So you might class this (pardon the pun!) as a type of its own, but this is something I’ve learnt all the same recently, so I hope it benefits you too! Python has the ability to construct more complex objects in the form of classes. While I won’t go into the details of that (I’ll leave it up to better resources), I wanted to show how I’ve been able to utilise them.

Test Mocks

One of the main reasons for implementing classes has been to resolve one of my requirements I described earlier on in this post, the ability to test my code easily. For my side project I interface with some third-party APIs to retrieve data, in order for me to test these I need to mock them appropriately. See below for a snippet of the code:

# spotify.py
class SpApi():
    client: spotipy.Spotify

    def __init__(self, id: str, secret: str) -> None:
        ccm = SpotifyClientCredentials(
            client_id=id,
            client_secret=secret
        )
        self.client = spotipy.Spotify(client_credentials_manager=ccm)

    def get_album_by_id(self, id: str) -> SpAlbum:
        return SpAlbum(self.client.album(id))

    def query_album_by_title(self, title: str) -> SpQueryRespWrapper:
        q = SpQueryBuilder(album=title).build()
        return self.execute_query(q)

    def query_album_by_title_and_artist(self,
                                        title: str,
                                        artist: str) -> SpQueryRespWrapper:
        q = SpQueryBuilder(album=title, album_artist=artist).build()
        return self.execute_query(q)

    def execute_query(self, q: str) -> SpQueryRespWrapper:
        return SpQueryRespWrapper(self.client.search(q=q, type='album'))

You might have guessed it, the API is Spotify’s, and I am using the spotipy library to do the interfacing. The SpApi class is essentially a wrapper around the client to allow me to easily re-use methods across the app. It’s usage would be sp_api = SpApi(), which would return us a client available at sp_api.client. It is this attribute of the class which we want the ability to mock for when we test it. See how I implemented the mock:

# test_spotify.py
import unittest
from unittest.mock import MagicMock

import munch

import spotify

get_album_by_id_response_fname = 'sp_get_album_by_id.json'
with open(get_album_by_id_response_fname, 'r', encoding='utf-8') as f:
    get_album_by_id_response = json.load(f)


class TestSpApiClass(unittest.TestCase):

    def get_mock_sp_api(self) -> spotify.SpApi:
        mock_sp_api = spotify.SpApi('id', 'secret')
        mocked_album = MagicMock(return_value=get_album_by_id_response)
        mock_sp_api.client = MagicMock(album=mocked_album)
        return mock_sp_api

    def test_get_album_by_id(self):
        """
        Given a request is made to Spotify API for Album ID
        Should verify that the API is hit
        """
        mock_sp_api = self.get_mock_sp_api()
        expected = spotify.SpAlbum(munch.munchify(get_album_by_id_response))
        actual = mock_sp_api.get_album_by_id('album_id')
        mock_sp_api.client.album.assert_called_once_with('album_id')
        self.assertEqual(actual, expected)

Talking about get_mock_sp_api first, this constructs a new SpApi wrapper class, but then it overwrites the client attribute with MagicMocks from the unittest framework. MagicMock essentially creates a mocked object or function for you. When I create a MagicMock with a return_value parameter, it will return that passed parameter when the MagicMock object is invoked.

Did you notice how it specifies the return type too?! -> spotify.SpApi

The test test_get_album_by_id creates an instance of this mocked Spotify API, so when we call the wrapper method get_album_by_id it makes a downstream call to client.album, which is what we mocked already with the chained MagicMock object! So now we can tell it what to return back when hit with certain parameters - and validating back with assert_called_once_with.

Trying to learn how to implement the above has probably been the biggest blocker for me to test Python code in this way - now that I’ve discovered it I can’t stop using it everywhere!

Complex Object Comparison

Much like the previous section, this relates to the static typing we implemented earlier. I was writing methods that returned complex objects encapsulated in classes, and I wanted to be able to test for equality using unittest’s self.assertEqual(actual, expected).

Let’s take this simple class:

class SomeObject():
    some_str: str
    some_int: int

    def __init__(self, some_str: str, some_int: int) -> None:
        self.some_str = some_str
        self.some_int = some_int

What we want to achieve is for us to be able to use the below test like so:

class TestObject(unittest.TestCase):

    def test_equality(self):
        """
        Should successfully test for object equality
        Given both objects are equal
        When using self.assertEqual
        """
        object1 = scratch.SomeObject(some_str='str', some_int=1)
        object2 = scratch.SomeObject(some_str='str', some_int=1)
        self.assertEqual(object1, object2)

However executing the test gives us an error:

> python test_scratch.py
F
======================================================================
FAIL: test_equality (__main__.TestObject)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_scratch.py", line 16, in test_equality
    self.assertEqual(object1, object2)
AssertionError: <scratch.SomeObject object at 0x000002A0A0199EB8> != <scratch.SomeObject object at 0x000002A0A0199EF0>

----------------------------------------------------------------------
Ran 1 test in 0.003s

FAILED (failures=1)

This fails because we have not overwritten the default __eq__ method on the class. Python then defaults to comparing the memory references for both objects. Since these are two entirely different objects they will be pointing to different memory references. What we need to do is tell Python what defines equality, so let’s add the method onto SomeObject.

def __eq__(self, other):
    if type(other) != type(self):
        return NotImplemented
    return self.some_str == other.some_str \
        and self.some_int == other.some_int

And here we are! First we are quickly checking to see the two objects are of the same type before we proceed with the underlying values (fail first, and early!). From there we are checking the attributes within the objects. Hurrah!

One neat little trick I like to do for complex objects that have multiple attributes, is that instead of writing out a new and comparator for every attribute, is to replace it all with this:

def __eq__(self, other):
    if type(other) != type(self):
        return NotImplemented
    return self.__dict__ == other.__dict__

The __dict__ attribute returns the objects in a Python dictionary format, which will contain all the attributes and their values. If the objects are entirely identical then they will return true! Now let’s run the test again.

> python test_scratch.py                                              
.                                                                     
----------------------------------------------------------------------
Ran 1 test in 0.001s                                                  
                                                                      
OK

Normal service is resumed!!

Closing

I hope the hints detailed in this helped you as much as it did for me! I’m sure there are many more tips out there - as I discover them be sure that I’ll share them once I find them, along with documenting more on the side project I’ve been working on!

Thanks for reading 😃

Extending Gotests for Strict Error Tests

Mon, 29 Apr 2019 00:00:00 +0000

UPDATE 2020-07-17

I wrote a follow up post to this one using the assert package instead.

Awesome - take me there!

Strict Error Tests in Java

I love confirming the stability of my code through writing tests and practicing Test-driven development (TDD). For Java, JUnit was my preferred testing framework of choice. When writing tests to confirm an exception had been thrown, I used the optional parameter expected for the annotation @Test, however I quickly found that this solution would not work for methods where I raised the same exception class multiple times for different error messages, and testing on those messages.

This is commonly found in writing a validation method such as the one below, which will take in a name of a dog and return a boolean if it is valid.

public static boolean validateDogName(String dogName) throws DogValidationException {

    if (containsSymbols(dogName)) {
        throw new DogValidationException("Dogs cannot have symbols in their name!");
    }
    
    if (dogName.length > 100) {
        throw new DogValidationException("Who has a name for a dog that long?!");
    }

    return true;
}

For this method, just using @Test(expected = DogValidationException.class) on our test method is not sufficient; how can we determine that the exception was raised for a dogName.length breach and not for containing symbols?

In order for me to resolve this, I came across the ExpectedException class for JUnit on Baeldung which enables us to specify the error message expected. Here it is applied to the test case for this method:

@Rule
public ExpectedException exceptionRule = ExpectedException.none();

@Test
public void shouldHandleDogNameWithSymbols() {
    exceptionRule.expect(DogValidationException.class);
    exceptionRule.expectMessage("Dogs cannot have symbols in their name!");
    validateDogName("GoodestBoy#1");
}

Applying to Golang

Back to Golang, there is a built-in library aptly named testing which enables us to assert on test conditions. When combined with Gotests - a tool for generating Go tests from your code - writing tests could not be easier! I love how this is bundled in with the Go extension for VSCode, my text editor of choice (for now…).

Converting the above Java validateDogName method to Golang will produce something like:

func validateDogName(name string) (bool, error) {
    if containsSymbols(name) {
        return false, errors.New("dog cannot have symbols in their name")
    }

    if len(name) > 100 {
        return false, errors.New("who has a name for a dog that long")
    }

    return true, nil
}

If you have a Go method that returns the error interface, then gotests will generate a test that look like this:

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr bool
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: true,
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if (err != nil) != tt.wantErr {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

From the above we are limited to what error we can assert for, here any error returned will pass the test. This is equivalent to using @Test(expected=Exception.class) in JUnit! But there is another way…

Modifying the Generated Test

We only need to make a few simple changes to the generated test to give us the ability to assert on test error message…

func Test_validateDogName(t *testing.T) {
    type args struct {
        name string
    }
    tests := []struct {
        name    string
        args    args
        want    bool
        wantErr error
    }{
        name: "Test error was thrown for dog name with symbols",
        args: args{
            name: "GoodestBoy#1",
        },
        want: false,
        wantErr: errors.New("dog cannot have symbols in their name"),
    }
    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            got, err := validateDogName(tt.args.name)
            if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
                t.Errorf("validateDogName() error = %v, wantErr %v", err, tt.wantErr)
                return
            }
            if got != tt.want {
                t.Errorf("validateDogName() = %v, want %v", got, tt.want)
            }
        })
    }
}

From the above there are three highlighted changes, let’s go over them individually:

wantErr error
- we are changing this from bool so that we can make a comparison against the error returned from the function
wantErr: errors.New("dog cannot have symbols in their name"),
- this is the error struct that we are expecting
if tt.wantErr != nil && !reflect.DeepEqual(err, tt.wantErr) {
- check to make sure the test is expected an error, if so then compare it against the returned error

Point 3 provides additional support if there was a test case that did not expect an error. Note how wantErr is omitted entirely from the test case below.

{
    name: "Should return true for valid dog name",
    args: args{
        name: "Benedict Cumberland the Sausage Dog",
    },
    want: true,
}

Customising Gotests Generated Test

Gotests gives us the ability to provide our own templates for generating tests, and can easily be integrated into your text editor of choice. I’ll show you how this can be done in VSCode.

Check out gotests and copy the templates directory to a place of your choosing
- git clone https://github.com/cweill/gotests.git
- cp -R gotests/internal/render/templates ~/scratch/gotests
Overwrite the contents of function.tmpl with the contents of this Gist
Add the following setting to VSCode’s settings.json
- "go.generateTestsFlags": ["--template_dir=~/scratch/templates"]

Once you have done that, future tests will now generate with stricter error testing! 🎉

Closing

I understand that the recommendations above will make your code more fragile, as the code is subject to any changing of the error message of say a downstream library. However for myself, I prefer to write tests that are strict and minimalise the chance of other errors contaminating tests.

I also understand that GoodestBoy#1 is probably a valid name for a dog! 🐶

Blog Bootstrap

Mon, 11 Feb 2019 00:00:00 +0000

Howdy! 👋

This site will be an area for me to braindump everything and anything that comes across my mind - although I’ll try my best to keep it tech-focused!

The blog platform itself is pretty basic. It is built using Hugo, hosted using GitHub Pages, while served over a custom domain via CloudFlare. While this isn’t exactly anything to brag about, it is only meant to be a quick and easy platform to spin up and get hosted so that I can focus on what really matters… the content!

It also gives me an opportunity to migrate the platform from GitHub Pages onto another hosted platform. Maybe that is AWS, which is a perfect learning opportunity, or even customise the theme of this blog (which is very cool already - thanks hugo-coder!) that I have left as its default. All of which provides me with more content to blog about!

See you around!

:file_folder: Projects

Mon, 01 Jan 0001 00:00:00 +0000

Portfolio Website

jdheyburn.co.uk is my portfolio website - how you are reading this content!
It is based the JAMstack framework Hugo
Hosted on GitHub Pages and built & deployed via GitHub Actions
Globally distributed via Cloudflare CDN

OSS Contributions

terraform-aws-provider

Add data source for aws_ssm_patch_baseline

Adding a data source component to aws_ssm_patch_baseline enables retrieval of AWS SSM Patch Baselines that already exist in your AWS account
You can assign baselines that have been created by AWS, or managed outside of your Terraform configuration.
Check out this post for what capability it allows for

Status: Merged ✅

Add patch_source block to resource_aws_ssm_patch_baseline

The patch source API enables AWS SSM Patch Baselines to specify alternate patch repositories at runtime of the SSM Document AWS-RunPatchBaseline.

Status: Merged ✅

:mailbox: Contact

Mon, 01 Jan 0001 00:00:00 +0000

You can get in contact with me via any of the below:

🐘 Mastodon

🐤 Twitter

📧 jdheyburn [at] gmail [dot] com

:wave: Hey Joe!

Mon, 01 Jan 0001 00:00:00 +0000

Hi there! 👋

I’m a technology enthusiast based in London, UK 💂‍♂️ 🇬🇧 with experience across software development, DevOps, and Cloud Engineering.

I mainly blog about stuff I’ve been working on - documenting my methodology throughout the way, along with writing up about any cool stuff I’ve built or contributed to.

If you’re interested in contacting me then please feel free to reach out.

💓 Passions

Automation
Code Quality
Documentation
Open Source
Security

🛠️ Tools & Technologies

Ansible
ElasticSearch
HAProxy
MongoDB
Redis
SQL
Terraform
… and a plethora of AWS services!

🏆 📜 Certificates

AWS Solutions Architect Associate
- 2019 - 2022
Cisco Certified Network Associate (CCNA)
- 2013 - 2019

🗣️ 🌐 Languages

Bash scripting
Golang
Java
Python
React

JDHeyburn

Reorganising Obsidian Journal Notes

Group new files

Updated explicit daily note links

Adding aria2 download manager to my NixOS homelab

aria2 introduction

Deploying with NixOS

Configuring AriaNg

Creating an aria2 NixOS module

Fin

I'm going to KubeConEU 2024

Simplifying Caddy configs in NixOS

tl;dr

A gentle reminder

I have seen the light

How I use Obsidian to journal

Edits

tl;dr

Introduction to Obsidian

Why journal?

Setting up the workflow

Template files

Plugins and configuration

Calendar

Dataview

Periodic Notes

Templater

Connecting everything together

Deep dive into note templates

Daily

Properties

Navigation

Summary

A bit about Dataview and querying

Weekly

Properties

Navigation

Summary

Monthly

Properties and navigation

Summary

Quarterly

Properties

Summary

Conclusion

Automating service configurations with NixOS

Catalog definitions

Node definitions

Service definitions

Reading catalog definitions

DNS rewrites

Caddy config

Route functions

Constructing Caddy config

Monitoring

Blackbox configuration

Visualising and alerting

Dashy

Building Dashy config

Deployment configurations

Conclusion and improvements

Nix Cheat Sheet

Documentation

Resources

Functions (and Lambdas)

Check if attribute is in attrset shorthand

Merge two attrsets shorthand

Add an attribute to an attrset

Converting a list to an attribute set

Converting to the Church of Nix

Why NixOS

Layout

Legacy

dee

frank

Today

dee

dennis

Additional features

Configurations