basnotes

 

Site CI/CD

tip

Whenever I write a new article for my site, I would like to have it published automatically. As mentioned earlier, I’m using Hugo to generate it and I have seen several solutions of other Hugo users for automated deployment.

Update August 2021:

I recently discovered OneDev.io, which is a self-acclaimed all-in-one devops platform. See https://code.onedev.io to check the code and see what OneDev looks like. It is packed with features, such as:

While I enjoyed using Gitea, the integrated CI/CD functionality really appealed to me, so I decided to try it out.

Deployment process change

With Gitea, I generated the static site in the hosts file system and had my main webserver host it. With OneDev I decided to create Docker image and use that to host the site. Obviously, there is some overhead, but idle there is no CPU usage and memory usage is with ~9MB not too bad.

Docker builder

To keep my image small and clean, I’m using a builder to generate the site and then copy the result in container running Caddy for hosting:

FROM klakegg/hugo:latest-ext AS builder

COPY . /src
RUN hugo --gc --minify

FROM caddy:2

COPY --from=builder /src/public/ /srv/
COPY Caddyfile /etc/caddy/Caddyfile

Deploy the image

The OneDev buildspec UI looks nice and clean and gives a good overview: OneDev buildspec UI

But, you get more control using the underlying yaml file:

version: 9
jobs:
- name: Build, Push & Deploy
  steps:
  - !CheckoutStep
    name: Checkout
    cloneCredential: !DefaultCredential {}
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Build image
    image: docker
    commands:
    - docker build -t @property:image_tag@ . --label "vcs-ref=@commit_hash@" --label
      "version=@build_number@" --label "build-date=$(date +"%Y-%m-%d %H:%M:%S")"
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Push to registry
    image: docker
    commands:
    - docker push @property:image_tag@:latest
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  - !CommandStep
    name: Deploy
    image: docker
    commands:
    - docker pull @property:image_tag@:latest
    - docker stop @property:container_name@ || echo "container @property:container_name@
      does not exist"
    - docker rm @property:container_name@ || echo "container @property:container_name@
      does not exist"
    - docker run --name @property:container_name@ -p 8082:80 --restart always -d @property:image_tag@:latest
    useTTY: false
    condition: ALL_PREVIOUS_STEPS_WERE_SUCCESSFUL
  triggers:
  - !BranchUpdateTrigger {}
  retryCondition: never
  maxRetries: 3
  retryDelay: 30
  cpuRequirement: 250m
  memoryRequirement: 128m
  timeout: 3600
properties:
- name: image_tag
  value: <image_tag>
- name: container_name
  value: <container_name>

Conclusion

I’m sticking with OneDev!

Previously

The Hugo documentation refers to several online services like Wercker, Netlify and GitLab.

The documentation also mentions GitHub, but those instructions have not been updated to use GitHub Actions. GitHub has numerous Actions to support Hugo site generation and deployment.

Other users have set up other creative solutions to automate their site deployment. Ryan Himmelwright decided to use Jenkins, while Sean Gransee used CircleCI and Chris Ferdinandi set up a webhook on a Digital Ocean server. I’m self-hosting my site and my Git server. Hosting my own Jenkins server as well, feels overdone.

Instead, I considered using Git Hooks to update my site after pushing new content to the repository. That would have worked nicely, if I wasn’t running Gitea in a container. I didn’t want to modify the image to include the required Hugo executable, so I had to find another way. Gitea also supports using webhooks which allows accessing resources outside Gitea’s container. This post describes how I used webhooks to deploy my Hugo based sites.

Toolchain

The workflow is implemented using the following chain of tools:

Toolchain

After writing my post, I commit it and push it to my Gitea Git server. Gitea is then calling an end-point which is implemented using webhook. webhook makes it very easy to define the end-point and have an application executed. It is calling a bash-script that is checking out the latest changes on Gitea, is then building the site using Hugo and is finally copying the result to the webroot of the site.

Gitea webhook configuration

Gitea has many configuration options to set up a webhook: Gitea webhook configuration

In my case, I want to post some information about the push, when it was pushed. Using the secret, the end-point can be secured.

webhook configuration

webhook allows to define many end-points using a JSON formatted config file:

    /etc/webhook.conf
    [
      {
        "id": "gitea",
        "execute-command": "/home/bas/dockerfiles/caddy2/deploy_site.sh",
        "pass-arguments-to-command":
        [
          {
            "source": "payload",
            "name": "repository.ssh_url"
          },
          {
            "source": "payload",
            "name": "repository.name"
          }
        ],
        "trigger-rule":
        {
          "and":
          [
            {
              "match":
              {
                "type": "value",
                "value": "<secret text>",
                "parameter":
                {
                  "source": "payload",
                  "name": "secret"
                }
              }
            }
          ]
        }
      }
    ]

End-point fields:

See the webhook project for more detailed information.

Deploy script

#!/usr/bin/env bash

REPOSITORY_SSH_URL=$1
REPOSITORY_NAME=$2
WORKDIR=/var/tmp
SITEDIR=/home/bas/sites

pushd $WORKDIR
        rm -rf $REPOSITORY_NAME
        git clone $REPOSITORY_SSH_URL $REPOSITORY_NAME
        docker pull klakegg/hugo:latest-ext
        docker run --rm klakegg/hugo:latest-ext version
        docker run --rm -v $WORKDIR/$REPOSITORY_NAME:/src klakegg/hugo:latest-ext --gc --minify
        rm -rf $SITEDIR/$REPOSITORY_NAME
        mkdir -p $SITEDIR/$REPOSITORY_NAME
        cp -r $WORKDIR/$REPOSITORY_NAME/public/* $SITEDIR/$REPOSITORY_NAME
popd

First a temporary working directory is deleted. Then I clone the repository to build from. The latest klakegg/hugo image is pulled and then used to generate the site. Finally, the generated site directory is replaced by the newly generated site files and the work is done.

This script is generic and can deploy any Hugo site as long as the repository.ssh_url and repository.name are passed as arguments. This makes it easy to use a single webhook end-point and deploy script to refresh multiple Hugo sites.

Conclusion

This relatively simple set of tools and scripts really work well for me. Push and forget!

October 19, 2020