Michael Murphy

Syncing Plex Watch State to Jellyfin

2024-04-08T00:00:00+00:00

I have been using Plex to manage and serve the media on my homelab for the last few years. After recent worrying trends have started to emerge on Plex I have started exploring a migration to Jellyfin instead.

In this post I’ll detail how I am syncing my watch state between both Plex and Jellyfin as I decide which one to stick with.

Plex Services

I currently use Plex for management of my media, which includes:

Movies
TV
Music - specifically Plexamp
Audiobooks - utilising audNexus as my metadata provider
YouTube

In addition to Plex, I setup Tautulli for tracking server usage and Overseerr for managing media requests.

Configuration for all of this is managed by NixOS - as seen in my Plex module

Jellyfin Configuration

I run a much more simple setup when it comes to Jellyfin, with it managing:

Movies
TV
Music - I will look to switch to a different solution if this proves too basic
YouTube

I manage audiobooks via Audiobookshelf as a dedicated service, as it does a much better job than Plex at managing audiobook librarires.

My Jellyfin configuration is also managed by NixOS and is available on GitHub

Note

The Jellyfin setup is a bit more involved when it comes to hardware transcoding. I have included a few useful notes in my linked config with helpful resources.

Syncing Watch Status

After doing some initial research I came across a project for this purpose aptly named watchstate.

Watchstatus Configuration

This was pretty easy to setup via docker-compose. I followed the instructions on GitHub and first created a docker-compose.yml:

YAML · docker-compose.yml

services:
  watchstate:
    image: ghcr.io/arabcoders/watchstate:latest
    network_mode: "host" # enable connection to host vpn
    user: "1000:1000"
    container_name: watchstate
    restart: unless-stopped
    environment:
      - WS_TZ=Australia/Melbourne
    ports:
      - "8001:8080" # avoid clashing with other service on 8080
    volumes:
      - ./data:/config:rw # mount current directory to container /config directory.

Following this I created a data folder (passed as a volume to container) in the same directory with: mkdir -p data

Finally I started the container by running:

Bash

# install docker-compose for the current shell in NixOS
nix shell nixpkgs#docker-compose

# start the container in the background
sudo docker-compose up -d

Adding Backend

With the container running I was then able to add a new backend. This refers to setting up the connection to a media service such as Plex or Jellyfin.

This was done by running the following commands and completing the prompts:

Bash

docker exec -ti watchstate console config:add

You will need to extract a Plex token and retrieve you Jellyfin API as part of this setup: Go to Dashboard > Advanced > API keys > then create new api keys

The only issue I ran into was due to me being unfamiliar with the terminology. I wanted to sync my watch status from Plex to Jellyfin. According to Watchstate terminology this means I want to import my Plex watch state to the database and export the database state into Jellyfin.

Once I understood this I was able to setup backends for both Plex and Jellyfin.

Syncing Backends

With both backends configured I first exported my Plex state:

Bash

# I named my plex backend plex_media
sudo docker exec -ti watchstate console state:import -v -s plex_media

Then finally I export the database state to Jellyfin, following initial instructions to forcibly sync the initial export:

Bash

# I named my jellyfin backend jellyfin_media
sudo docker exec -ti watchstate console state:import -vvifs jellyfin_media

Following this I checked Jellyfin and sure enough, everything was in sync!

Scheduling Regular Sync

The project mentioned the option of setting up cron jobs for regular syncing ongoing. I chose not to implement this, as I am happy to manage this manually myself. I’ll evaluate Jellyfin as a replacement to Plex and stick with one or the other.

Should I decide to run this services long-term then I’d create a NixOS module for it.

If you decide to schedule syncing its just a matter of setting the following environment variable in docker-compose.yml: WS_CRON_IMPORT=1 and WS_CRON_EXPORT=1

Configuring a Minecraft Server Using NixOS

2024-04-02T00:00:00+00:00

Minecraft can be configured simply by using the provided NixOS modules. Tailscale have published a blog post showing how to utilise this and get a working server.

This setup is a bit limited when it comes to installing mods and mod toolchains declaratively. Nix-minecraft was developed for this purpose.

NixOS Minecraft Configuration With Mods

To use Nix-minecraft, add the following to an existing flake.nix, or create a new one like so:

Nix · flake.nix

{
  description = "NixOS Minecraft Server";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    # Add nix-minecraft input
    nix-minecraft.url = "github:Infinidoge/nix-minecraft";
  };

  outputs = {
    self,
    nixpkgs,
    # Add nix-minecraft input
    nix-minecraft,
    ...
  } @ inputs: {
    nixosConfigurations.media = nixpkgs.lib.nixosSystem {
      specialArgs = {inherit inputs;};
      modules = [
        ./minecraft.nix
        agenix.nixosModules.default
        {
          # Add overlay for nix-minecraft
          nixpkgs.overlays = [
            nix-minecraft.overlay
          ];
        }
      ];
    };
  };
}

Minecraft Configuration File

Then create a new nix file for server configuration, the following shows an example of how to do this using a nix module format.

Available mods for install can be found at Modrinth.

Nix-minecraft provides a helper scrip to identify the fetchurl and sha512 hash needed to install a mod. To use it, find a mod on Modrinth and click on the version you want. In the displayed information, there is a Version ID string. Click on it to copy the version ID. Then run the following on a system with nix flakes enabled, replacing with the copied version ID: nix run github:Infinidoge/nix-minecraft#nix-modrinth-prefetch -- versionid

Once you’ve got the details for mods to be install, you can finalise server configuration like so:

Nix · minecraft.nix

{
  lib,
  config,
  pkgs,
  # Used to import nix-minecraft
  inputs,
  ...
}:
with lib; let
  cfg = config.common.minecraft;
  # Target minecraft version
  mcVersion = "1.20.1";
  # Version of toolchain (fabric in this case)
  fabricVersion = "0.15.7";
  # Format minecraft version, replacing . with _
  serverVersion = lib.replaceStrings ["."] ["_"] "fabric-${mcVersion}";
in {
  # Import nix-minecraft to provide path to mod toolchain packages
  imports = [
    inputs.nix-minecraft.nixosModules.minecraft-servers
  ];

  options.common.minecraft = {
    enable = mkEnableOption "Enable Minecraft Server";
  };

  config = mkIf cfg.enable {
    services = {
      # Server configuration module options
      minecraft-servers = {
        enable = true;
        eula = true; # accepting EULA is required for server to run
        # Rename and clear out /srv/minecraft to setup new server
        servers.server-name = {
          enable = true;
          package = pkgs.fabricServers.${serverVersion}.override {loaderVersion = fabricVersion;};
          serverProperties = {
            # Adjust server properties to fit needs
            server-port = 25565;
            gamemode = "peaceful";
            motd = "NixOS Pokemon"; # Server name
            max-players = 2;
            level-seed = "10292758"; # Seed for level generation
          };
          symlinks = {
            # List all mods to be installed
            mods = pkgs.linkFarmFromDrvs "mods" (builtins.attrValues {
              # Example values returned by nix run command
              FabricApi = pkgs.fetchurl {
                url = "https://cdn.modrinth.com/data/P7dR8mSH/versions/YG53rBmj/fabric-api-0.92.0%2B1.20.1.jar";
                sha512 = "53ce4cb2bb5579cef37154c928837731f3ae0a3821dd2fb4c4401d22d411f8605855e8854a03e65ea4f949dfa0e500ac1661a2e69219883770c6099b0b28e4fa";
              };
              Pokemon = pkgs.fetchurl {
                url = "https://cdn.modrinth.com/data/MdwFAVRL/versions/uWAkNUxZ/Cobblemon-fabric-1.4.1%2B1.20.1.jar";
                sha512 = "6955c8ad187d727cbfc51761312258600c5480878983cfe710623070c90eb437e419c140ff3c77e5066164876ecfe1e31b87f58f5ef175f0758efcff246b85a8";
              };
            });
          };
        };
      };
    };
  };
}

Additional Resources

Another helpful reference was Misterio77’s NixOS Minecraft config

Remote Deployment of NixOS Using nixos-anywhere and Disko

2024-03-27T00:00:00+00:00

Now that the use cases for my homelab are growing, I’m finding a need to separate different services into their own virtual machines. This has been a bit tedious in the past, as each time I’ve had to format disk and create a new NixOS configuration.

Nixos-anywhere and Disko combine together to make this simple. This post explains how to use these tools.

NixOS Anywhere & Disko

Nixos-anywhere allows for remote deployment of NixOS via kexec on any Linux based operating system. This provides a quick method to convert any virtual machine to NixOS and build the new system according to a specified NixOS configuration.

Nixos-anywhere utilises Disko, which allows for declarative disk preparation and formatting. This means we can deploy without the usual fdisk/gparted preparation.

Disko Configuration

First start by specifying how you want Disko to configure your disks. You can find an example Disko config in the Disko repository

The following configuration is what I used to format an SSD drive, mounted at /dev/sda, creating partitions for EFI boot and operating system storage:

Nix · hosts/services/disk-config.nix

{...}: {
  disko.devices = {
    disk = {
      # To specify an additional drive, create another entry e.g. disk.data
      main = {
        device = "/dev/sda";
        type = "disk";
        content = {
          type = "gpt";
          partitions = {
            # Boot partition formatted for EFI
            boot = {
              size = "1M";
              type = "EF02"; # for grub MBR
            };
            ESP = {
              name = "ESP";
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
              };
            };
            # Optional swap partition
            swap = {
              size = "8G";
              content = {
                type = "swap";
                randomEncryption = true;
                resumeDevice = true; # resume from hiberation from this device
              };
            };
            # Root partition for operating system storage
            root = {
              size = "100%";
              content = {
                type = "filesystem";
                format = "ext4";
                mountpoint = "/";
              };
            };
          };
        };
      };
    };
  };
}

NixOS Anywhere Configuration

With the Disko configuration complete, we now need to update our flake and add inputs for Disko. We also need to add the Disko module within our nixosConfiguration.

Here is an example:

Nix · flake.nix

{
  description = "Nixos-anywhere deployed flake";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";

    disko.url = "github:nix-community/disko";
    disko.inputs.nixpkgs.follows = "nixpkgs";

    # Optional: used for secret sharing
    agenix.url = "github:ryantm/agenix";
    agenix.inputs.nixpkgs.follows = "nixpkgs";
    agenix.inputs.home-manager.follows = "home-manager";

    # Optional: used for deployment of multiple systems
    deploy-rs.url = "github:serokell/deploy-rs";
    deploy-rs.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = {
    self,
    nixpkgs,
    disko,
    # Optional: as above, these aren't required for nixos-anywhere
    agenix,
    deploy-rs,
    ...
  } @ inputs: {
    # Specify hostname e.g. services
    nixosConfigurations.services = nixpkgs.lib.nixosSystem {
      # Allow inputs to be available from within module scope
      specialArgs = {inherit inputs;};
      modules = [
        # Specify system configuration to load
        ./hosts/services/configuration.nix
        # Optional: import secret management module
        agenix.nixosModules.default
        # Importing the disko module for formatting disks
        disko.nixosModules.disko
      ];
    };
  };
}

The most important part in the above is adding an input for disko and loading the disko module within the nixosConfiguration: disko.nixosModules.disko.

Importantly we must also add additional details to the configuration file specified. In the case of the above we provided the path: ./hosts/services/configuration.

Here are the details we need to add:

Nix · hosts/services/configuration.nix

{ modulesPath, config, lib, pkgs, ... }: {
  imports = [
    (modulesPath + "/installer/scan/not-detected.nix")
    # Specify qemu guest, relevant for virtual machines
    (modulesPath + "/profiles/qemu-guest.nix")
    # Import path to disko configuration, covered above
    ./disk-config.nix
  ];
  boot.loader.grub = {
    # Specify boot devices, should match disko configuration
    devices = ["/dev/sda"];
    # Whether to support EFI boot
    efiSupport = true;
    efiInstallAsRemovable = true;
  };
  # Enable connection via SSH post nixos-anywhere deployment
  services.openssh.enable = true;

  environment.systemPackages = [
    # Specify any packages you want installed by default
    pkgs.vim
  ];

  users.users.root.openssh.authorizedKeys.keys = [
    # Specify SSH key for connection post nixos-anywhere deployment
    "ssh-ed25519 AAAAC3NzaC1lZDI4uJE5AAAAIEVyN0R5mTtfcbkmVXjicuvSRotJY4IuT7h3H"
  ];

  # Default recommended with NixOS install
  system.stateVersion = "23.11";
}

Here are references to a couple of example configurations I found useful: nixos-anywhere example flake, nixos-anywhere example config

Deployment

With all of the above configuration in place, it’s time to deploy. I normally create a bare bones virtual machine on my homelab using a minimal nixos iso - we don’t have to use a NixOS iso here, we could use Debian, Ubuntu etc. After starting the virtual machine we can complete the following steps from within the console to prepare for deployment:

We need to be able to run the nixos-anywhere command and connect via SSH to the new virtual machine. The NixOS iso sets up openssh by default, we just need to set the root password to authenticate a connection: sudo passwd root and follow the prompts
Confirm the IP address of the virtual machine so that we can connect: ip a and look for the valid IP

Now we can complete the deployment. From a machine with nix installed and flakes enable we can run the following command:

Bash

nix run github:nix-community/nixos-anywhere -- --flake <path-to-flake>#<flake-name> root@<ip-address>

Here’s an example with details added - note you can optionally specify --build-on-remote if the host architecture is different on your target:

Bash

nix run github:nix-community/nixos-anywhere -- --flake .#services root@192.168.1.254 --build-on-remote

Agenix Post Deployment Setup

I’ve explained in another post how I use Agenix to manage secrets. Assuming you are doing this on your new NixOS host there are a couple of steps you need to take to configure Agenix.

First is to find the ssh host key for your new NixOS host: ssh-keyscan <host-ip>. This will show each available ssh key on your host. You can use this information to add the new ssh key to your Agenix configuration:

Nix · secrets/secrets.nix

let
  # Existing configuration
  existing_system = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL0idNvgGiucWgup/mP78zyC23uFjYq0evcWdjGQUaBH";
  # Newly added SSH key discovered by ssh-keyscan
  new_system = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILI6jSq53F/3hEmSs+oq9L4TwOo1PrDMAgcA1uo1CCV/";
  all_systems = [ existing_system new_system ];
in
{
  # Configured secrets
  "secret1.age".publicKeys = all_systems;
  "secret2.age".publicKeys = [ existing_system ];
}

Now that we have added a new SSH key to our secrets.nix file we must rekey our existing secrets, and specify our existing ssh key: nix run github:ryantm/agenix --rekey -i ~/.ssh/existing_key.

Restoring Nextcloud from Backup on NixOS

2024-03-25T00:00:00+00:00

Occasionally when self hosting I’ve found a need to backup and restore the Nextcloud database. I have a regularly scheduled Postgres backup thanks to the following nix config:

Nix

{ lib, pkgs, ... }:

{
  services.postgresqlBackup = {
    enable = true;
    # path where backup should be saved
    location = "/data/backups/postgresql";
    # specifying nextcloud as the database to backup
    databases = ["nextcloud"];
    # cron schedule for backup to be performed
    startAt = "*-*-* 23:15:00";
  };
}

This post will detail the process I follow to do a restore from the backup created by the above config.

Restoring Nextcloud Database

The first thing to understand is how to execute database commands from within the NixOS environment. My Nextcloud instance in NixOS is managed by the nextcloud user, this same user is configured for the database as shown in my Nextcloud post.

I can execute commands against the database (PostgreSQL in my case) like so:

Bash

sudo runuser -u nextcloud -- psql -U nextcloud <options>

Using the information above we can drop the old database which we are replacing and then create a new one, in which we will later restore data to. These steps are documented on the Nextcloud website. The commands I used are below:

Bash

sudo runuser -u nextcloud -- psql -U nextcloud -c "DROP DATABASE \"nextcloud\";"
sudo runuser -u nextcloud -- psql -U nextcloud -c "CREATE DATABASE \"nextcloud\";"

You can then restore the backup into the newly created Nextcloud database with the following command:

Bash

sudo runuser -u nextcloud -- psql -U nextcloud -d nextcloud -f nextcloud-sqlbkp.bak

I discovered the above command didn’t work for me, as the scheduled backup utilizes pg_dump, which creates a backup.sql.gz file. After some searching I came across a Stack Exchange post which lead me to the final solution:

Bash

sudo runuser -u nextcloud -- pg_restore -U nextcloud -d nextcloud nextcloud.sql.gz

Post Restore Steps

After the database has been restored head to the Nextcloud admin console: https://nextcloud.example.com/settings/admin/overview and see if the self check reveals any issues with the new database. Chances are you will need to resolve some issues with the database indices. The following command will add missing indices to the database:

Bash

sudo -i nextcloud-occ db:add-missing-indices

Troubleshooting Proxmox Installation

2023-07-18T00:00:00+00:00

I encountered some issues when installing Proxmox on a new homelab. One method of solving these would have been to install Debian and then Proxmox from there. I decided to try my hand at troubleshooting the issues instead.

Why Proxmox

Before describing the issues I faced and how I resolved them, I though I should explain why I’m using Proxmox in the first place - why persevere through the pain?

The end state for my homelab is to have a number of different Virtual Machines (VMs) running segregated services. There are a few reasons I want to run multiple VMs:

To segment any data used for services to the specific VM I’m running it in
Most VMs will be running NixOS - I want to manage them remotely and become more familiar with tools like deploy-rs and nixos-anywhere
I want to be able to use VMs from any machine to easily experiment with new Linux distributions

I mostly want to use VMs over containers as currently NixOS doesn’t work so well with LXC, it’s possible, but I haven’t been thrilled with the result. I will use LXC for running some services like [Home Assistant].

Initial Boot

The first issue I had was getting the Proxmox install step to boot from USB. I kept getting an error saying: /dev fully populated

I managed to find a post on the Proxmox forum which suggested changing the boot flags to resolve the error:

Bash

# e at boot menu add nomodeset
GRUB_CMDLINE_LINUX_DEFAULT="quiet nomodeset"

This let me proceed to the install step for Proxmox.

Booting After Install

After installing Proxmox I rebooted my machine and found I hit a similar error again - adding the aforementioned nomodeset didn’t work as I found Proxmox was using systemd-boot rather than grub.

This was resolved by the following thanks to a post on Reddit:

Text

for EFI boot (systemd-boot) add `nomodeset` to /etc/kernel/cmdline after installation; run `pve-efiboot-tool refresh` to update

Setting Safe Mode to Boot from Intel Graphics

I thought I was finally done and ready to start setting up some VMs - but no, I hit a fatal server error, which required me to change my vga settings. I found a YouTube video which explained the changes:

Bash

lspci | grep -i vga
mkdir -p /usr/share/X11/xorg.conf.d
vi /usr/share/X11/xorg.conf.d/10-fbdev.conf

Enter the following text in 10-fbdev.conf

Text · /usr/share/X11/xorg.conf.d/10-fbdev.conf

Section "Device"
    Identifier "Card0"
    Driver "fbdev"
    BusID "pci0:2:0:0"
EndSection

Finally the system booted and I was able to start setting up Proxmox.

Bonus Tip - GPU Passthrough

Finally I wanted to leave some references I found helpful for configuring GPU passthrough for a VM in Proxmox. I needed to do this for my Jellyfin service, so I could take advantage of Intel QuickSync. For further reference on setting up and testing hardware transcoding checkout the Jellyfin documentation.

You can pass through an Intel GPU via PCI passthrough, as detailed in Proxmox documentation

Useful References for Understanding Nix/NixOS

2023-05-08T00:00:00+00:00

I recently used Nix to configure my M2 Macbook Air, and manage all of my dotfiles. In the process I found a lot of useful references, detailing the ins and outs of Nix and NixOS.

I’m currently building out a similar solution for my homelab, and thought to document a list of useful references here.

References

Useful Posts

Matthias Benaet’s Dotfiles: this is where I first began, Matthias has a really good video series introduction as well, you can find the details in the linked repo
Jordan Isaac on Nix Flakes: comprehensive introduction into creating a Flake in Nix
Alexander Bantyev on Nix Flakes: great beginner friendly guide
Bob Vanderlinden on Customising Nix Packages: as you get deeper into the rabbit hole of Nix, you will likely need to customise Nix packages, this post clearly explains the different ways you can accomplish this, and why you would use one over another
Xe Iaso on Securing NixOS: details how to create a more secure NixOS when running a production server
Xe Iaso on Creating a Secure AWS NixOS Image: detailed walk through of code to create a secure NixOS image in AWS
Graham Christensen on Impermanence in NixOS: explains how to setup a NixOS system that erases at boot, and why the approach is useful - also checkout the nix-community impermanence repo
Elis Hirwing on tmpfs as root: alternative method of impermanence on NixOS

Using Dev Containers to Standardise Development Environments

2023-04-11T00:00:00+00:00

After running my own homelab for a while, I’ve become pretty familiar with using containers - mostly via Docker, though I may have to make the switch to podman after a recent announcement. One thing I wasn’t aware of was the ability to run containers for the specific purpose of creating a development environment.

Configuring a Dev Container

I first came across this feature in Visual Studio Code (I try to avoid it, but sometimes its a necessary evil :P), there is a lot of great documentation which covers the topic in far more detail than I plan to.

The pre-requisites for setting up a Dev Container are Visual Studio Code (VSCode) and the Dev Containers plugin.

The basic idea is that you create a Dockerfile or docker-compose file, which sets up a container, you can then specify a devcontainer.json file which lets you configure additional settings unique to VSCode e.g. plugins that you want installed inside of the container, and/or VSCode settings you want active whilst in the container.

VSCode will even walk you through the process of creating the container itself if you don’t have a Dockerfile already on hand: Ctrl + Shift + P -> Dev Containers: New Dev Container... will get you started.

Follow this link for a list of all the options available when configuring devcontainer.json.

Here is an example of a quick setup I configured:

JSON

{
  "name": "Python 3",
  // Base image provided by Microsoft or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
  "image": "mcr.microsoft.com/devcontainers/python:0-3.9",
  // Install pre-configured features into the container - in this case the fish shell
  "features": {
    "ghcr.io/meaningful-ooo/devcontainer-features/fish:1": {
      "fisher": true
    }
  },
  // Specify extensions which should be made available inside of the container
  "customizations": {
    "vscode": {
      "extensions": [
        "vscodevim.vim",
        "ms-python.python",
        "ms-python.pylance",
        "ms-python.black",
        "ms-python.flake8",
        "ms-python.isort"
      ],
      "settings": {
        "python.linting.flake8Enabled": true,
        "python.linting.flake8Path": "/usr/local/bin/flake8",
        "python.formatting.blackPath": "/usr/local/bin/black"
      }
    }
  }
}

Adding Dotfiles

One feature which I thought was really handy is the ability to specify a repository containing dotfiles (linux personalised configuration settings, such as my dwm dotfiles), which can be cloned and configured once the Dev Container is up and running.

This means you can create a Dev Contianer and share it across a team, so that everyone has a standardised development environment. Then you can add your own configuration after the container has been started by using the dotfiles functionality - no one is subjected to your strange configuration (e.g. running the vim extension).

Use of Dev Containers in Neovim

Naturally the first thing I did after discovering this feature in VSCode was to explore whether it is also available in Neovim (my editor of choice) - sure enough I found a plugin. I need to play around with this more to get it configured, but I’m glad to know the option is there.

Useful References for Understanding ZFS

2023-03-06T00:00:00+00:00

I’ve been looking to upgrade my existing homelab (currently a 2012 Mac Mini which I had lying around). I plan to use ZFS when I do upgrade, as it has a lot of great features for data integrity, redundancy and backups.

I’ve found a number of useful references in the process, which I thought to include here:

References

Useful Posts

ZFS 101 - Understanding ZFS Storage & Performance: great write up explaining what ZFS is and how it works, this article goes into a good level of depth too
ZFS: You Should User Mirror VDEVs, Not RAIDZ: recommendations on how to structure ZFS pools - spoiler alert mirroring is a great option
ZFS Metadata Special Device: explains how to add a metadata device in order to increase random read performance on a ZFS pool

Encrypting Secrets in NixOS with Agenix

2023-02-22T00:00:00+00:00

After playing around with NixOS and creating declarative configurations I quickly came across a need to provide encrypted secrets, rather than clear text. This is a requirement to be able to save configurations on GitHub and is recommended anyway, given that clear text secrets are made available on the world readable Nix store.

There are a number of solutions available to encrypt secrets in NixOS, I landed on Agenix, as it’s relatively easy to configure and works with my deployment tool of choice - Deploy-rs.

How Agenix Works

Agenix uses pre-existing SSH keys to encrypt secrets using the Age encryption tool. These secrets are then decrypted using the private SSH host key on the target system.

There are a couple of points worth considering when using Agenix:

Agenix doesn’t officially have support for Home Manager - though an alternative is available (I haven’t tried this)
Agenix provides a secret file, rather than a string. This means that if there is no option in an existing module to pass a filepath for a secret, we need to have the service read the clear text file path at runtime

Using Agenix

I’ll run through a simple example here, should you want more detail I recommend reading through the Agenix readme.

Agenix Configuration

Create a directory for secrets and secret config - this should be somewhere that makes sense e.g. /etc/nixos/secrets

Next we need to create a secret configuration file secrets.nix. This tells Agenix which SSH keys to use when encrypting secrets, which secrets are available, and which systems/users should have access to each secret.

Bash

mkdir -p /etc/nixos/secrets
vim /etc/nixos/secrets/secrets.nix

The contents of the file should look something like this:

Nix · /etc/nixos/secrets/secrets.nix

let
  system = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL0idNvgGiucWgup/mP78zyC23uFjYq0evcWdjGQUaBH";
in
{
  "secretpassword.age".publicKeys = [ system ];
}

Creating a Secret

We are now ready to create out first secret secretpassword. This next part will differ depending on how you have installed Agenix.

Given I’m using Nix Flakes, I can simply run:

Bash

nix run github:ryantm/agenix -- -e secretpassword.age

The above assumes my SSH keys have been added to ~/.ssh/. If this is not the case we can specify which SSH keys to use:

Bash

nix run github:ryantm/agenix -- -e secretpassword.age -i /path/to/ssh/key

Adding Secret to NixOS

Once the above has been completed you can now add secrets to your NixOS configurations like so:

Nix · /etc/nixos/configuration.nix

{
  users.users.captainsecure = {
    isNormalUser = true;
    home = "/home/captainsecure";
    passwordFile = config.age.secrets.secretpassword.path;
  };

  age.secrets.secretpassword.file = ../../secrets/secretpassword.age;
}

Secrets Folder Structure

I discovered an interesting technique for organising Agenix secrets into directories, thanks to Charlotte Van Petegem’s dotfiles. I didn’t see this explained in the Agenix manual, but it looks like we can nest age secrets within the secrets directory e.g.

Text · /etc/nixos/secrets

.
├── secrets.nix
├── passwords
│   └── secretpassword.age

When we need to refer to these nested secrets in our Nix configuration, we specify the path to each age file e.g.

Nix · /etc/nixos/configuration.nix

{
  users.users.captainsecure = {
    isNormalUser = true;
    home = "/home/captainsecure";
    passwordFile = config.age.secrets."passwords/secretpassword".path;
  };

  age.secrets."passwords/secretpassword".file = ../../secrets/passwords/secretpassword.age;
}

How to Configure Nextcloud on NixOS

2023-02-21T00:00:00+00:00

Nextcloud provides a very polished experience when looking for a solution to self host your personal files, and replace tooling such as Google Drive or Dropbox. In this post I run through how I configured Nextcloud for use on NixOS.

Jacob Neplokh wrote an incredibly helpful blog post which forms the basis of this guide, I recommend having a read through for a more in depth explanation of everything. The NixOS Wiki also had some very helpful information.

There are several services that need to be configured to setup Nextcloud:

Nextcloud
Postgresql (optional but recommended)
Redis (optional)
Nginx

Nextcloud Setup

Nextcloud itself is relatively straightforward to setup, I found knowing all the additional services to be the tricky part. The module has a multitude of options to configure, for more information about each one take a look at the module’s options.

Here is a look at my config - you’ll note that secrets are managed with Agenix, which I’ve written about before:

Nix · /etc/nixos/configuration.nix

{
  services.nextcloud = {
    enable = true;
    # specify either a domain you own or localhost
    hostName = "nextcloud.yourdomain.com";
    autoUpdateApps.enable = true;
    https = true;
    # only specify caching if using redis or alternative service
    caching.redis = true;
    config = {
      # only specify dbtype if using postgresql db
      dbtype = "pgsql";
      dbname = "nextcloud";
      dbuser = "nextcloud";
      # default directory for postgresql, ensures automatic setup of db
      dbhost = "/run/postgresql";
      adminuser = "admin";
      # specified using agenix, provide path to file as alternative
      adminpassFile = config.age.secrets.nextcloudPass.path;
      # error thrown unless specified
      defaultPhoneRegion = "AU";
    };
    # specify only if you want redis caching
    extraOptions = {
      redis = {
        host = "127.0.0.1";
        port = 31638;
        dbindex = 0;
        timeout = 1.5;
      };
    };
  };

  # only needed if using agenix for secret encryption
  age.secrets = {
    nextcloudPass = {
      file = ../../secrets/nextcloudPass.age;
      # allow default nextcloud user access to secret
      owner = "nextcloud";
    };
  };
}

Postgresql Setup

It’s highly recommended that you replace the default Sqlite database with Postgresql, here are the steps involved in making the change:

Nix · /etc/nixos/configuration.nix

{
  services = {
    postgresql = {
      enable = true;
      ensureDatabases = [ "nextcloud" ];
      ensureUsers = [{
        name = "nextcloud";
        ensurePermissions."DATABASE nextcloud" = "ALL PRIVILEGES";
      }];
    };
    # optional backup for postgresql db
    postgresqlBackup = {
      enable = true;
      location = "/data/backup/nextclouddb";
      databases = [ "nextcloud" ];
      # time to start backup in systemd.time format
      startAt = "*-*-* 23:15:00";
    };
  };

  # ensure postgresql db is started with nextcloud
  systemd = {
    services."nextcloud-setup" = {
      requires = [ "postgresql.service" ];
      after = [ "postgresql.service" ];
    };
  };
}

As mentioned in the comments above, I’ve included an optional backup service, which will export the database to the specified location. Following this I backup the specified folder to using BorgBackup.

Redis Caching

This allows for caching of frequently used files in Nextcloud, which likely provides a more snappy user experience. Here’s my configuration:

Nix · /etc/nixos/configuration.nix

{
  services = {
    redis.servers.nextcloud = {
      enable = true;
      port = 31638;
      bind = "127.0.0.1";
    };
  };
}

Nginx Configuration

In order to login once Nextcloud is configured, HTTPS must be setup. My Nextcloud service is run behind a Tailscale VPN, so I had a couple of different options for configuring HTTPS:

Having Tailscale do it with tailscale cert
Creating an A record to point an existing domain to my Tailscale IP, using my DNS provider, and configuring HTTPS via ACME

I opted for the second option, as such I also had to configure ACME to automatically generate my SSL certificate. NixOS has builtin tooling using a service called lego behind the scenes to create certificates. Here’s my config:

Nix · /etc/nixos/configuration.nix

{
  services.nginx = {
    enable = true;
    recommendedGzipSettings = true;
    recommendedOptimisation = true;
    recommendedProxySettings = true;
    recommendedTlsSettings = true;
    # pulls in the hostname we set for nextcloud above (nextcloud.yourdomain.com)
    virtualHosts.${config.services.nextcloud.hostName} = {
      enableACME = true;
      acmeRoot = null;
      addSSL = true;
      # directs traffic to the appropriate port for nextcloud
      locations."/" = {
        proxyPass = "http://localhost:8080";
        proxyWebsockets = true;
      };
    };
  };

  security.acme = {
    acceptTerms = true;
    preliminarySelfsigned = false;
    defaults = {
      email = "acme@yourdomain.com";
      dnsProvider = "cloudflare";
      # API for authentication to DNA provider e.g.
      # CF_API_KEY=<insert-global-key>
      # CF_API_EMAIL=<insert-dns-account-email>
      # specify file if not using agenix
      credentialsFile = config.age.secrets.acmeCredentials.path;
    };
  };

  # allow nginx to configure acme
  users.users.nginx.extraGroups = [ "acme" ];

  # if providing credentialsFile via agenix
  age.secrets.acmeCredentials.file = ../../secrets/acmeCredentials.age;
}

I pieced together this setup thanks to the documentation in the NixOS Manual, I recommend giving it a read to understand what’s going on here.

Conclusion

As you can see above, there are a few steps involved in properly configuring Nextcloud. Regarding Nextcloud apps, I found it easier to be able to install and manage these via Nextcloud itself.

I ended up putting all of the above config into its own module, for reference you can view the module on GitHub.

Configuring Backup Solutions in NixOS

2023-02-13T00:00:00+00:00

One of my first consideration in setting up a homelab has been about a backup strategy - it’s difficult to have trust in a self-hosted instance of Nextcloud when I don’t know whether data will exist after a hardware fault. One common backup strategy is the 3-2-1 approach:

3 copies of the data being backup up
2 different media types for storage
1 off-site backup

This post details 2 methods of creating encrypted off-site backups in NixOS using Duplicati or BorgBackup.

Duplicati

NixOS provides a fairly bare bones module for setting up Duplicati. Here’s how I configured it:

Nix · /etc/nixos/configuration.nix

{
  services = {
    duplicati = {
      enable = true;
      # Allow web access for machines other than localhost
      interface = "0.0.0.0";
    };
  };
}

You can optionally specify a different user, group, port and data directory should you wish.

After running nixos-rebuild switch you can manually configure backups via the web interface at <serverip>:8200. Duplicati has many options for configuring backups, I decided to backup to an S3 object store. Backups can be scheduled, and rules can be set to delete old backups at the target location. Duplicati will also capture logs and display warnings if it encounters any problems during the backup. The only issue I encountered was initially with access rights to the directories I wanted to backup. You can also set encryption keys for backups, with options for different encryption methods.

I found Duplicati worked quite well, and I did have to test a restore in the brief time I used it. There was one major factor that made me continue looking for a different solution: I wanted to avoid the need for manual configuration, rather having Nix manage as much as possible for me declaratively.

BorgBackup

Thanks to an episode of the Linux Unplugged Podcast I came across BorgBackup. I found that the NixOS module was pretty extensive, and I learnt that a number of cloud hosting options existed for storing these backups, and they seemed to be pretty competitively priced (BorgBase and Rsync.net).

After doing some further digging I came across a very helpful blog post which detailed all the steps in configuring BorgBackup on NixOS. After looking through the modules and reading the blog post I ended up with the following configuration:

Nix · /etc/nixos/configuration.nix

{
  services.borgbackup.jobs = {
    "media" = {
      paths = [
        "/data/media"
        "/data/backup"
        "/var/lib/nextcloud"
      ];
      # Note: you will need to edit the SSH url provided by BorgBase to match the below format
      repo = "o6h6zl22@o6h6zl22.repo.borgbase.com:repo";
      encryption = {
        mode = "repokey-blake2";
        passCommand = "cat ${config.age.secrets.mediaBorgPass.path}";
      };
      environment.BORG_RSH = "ssh -i ${config.age.secrets.borgSSHKey.path}";
      compression = "auto,lzma";
      startAt = "daily";
    };
  };
   
  age.secrets.mediaBorgPass.file = ../../secrets/mediaBorgPass.age;
  age.secrets.borgSSHKey.file = ../../secrets/borgSSHKey.age;
}

The above configures a backup to my BorgBase repo which runs daily at midnight, backing up the specified directories. The backup is encrypted and compressed, and the repo requires an authorized SSH key for access. This continually monitors and updates a single backup, however you can optionally configure BorgBackup for append-only mode which allows for some delay in deletion of older files.

Note

In the above I’ve used Agenix for managing my secrets (SSH key and encryption key). I’ve written a post on configuring it

Note

You’ll likely have to run an initial SSH connection to the Borg repo to add its keys to known_hosts:

Bash

ssh -i /var/run/agenix/borgSSHKey o6h6zl22@o6h6zl22.repo.borgbase.com

Monitoring Initial BorgBackup

I thought to add some additional notes on how to monitor the initial backup, as this process is not as straight forward as when using a tool such as Duplicati. Credit for the commands in this and the next section go to the blog post listed earlier.

After adding the above configuration and running nixos-rebuild switch a Systemd service is configured to connect to the BorgBase repo and begin a backup. You can manually start the backup by running:

Bash

systemctl start borgbackup-job-<name-of-job>.service

You can then keep an eye on the backup by monitoring it via journalctl (keep in mind the backup will take a long time initially, subsequent backups are incremental and much faster):

Bash

journalctl -fu borgbackup-job-<name-of-job>.service

All being well you should see a successful connection being established and a successful disconnection once the backup has completed.

Restoring From BorgBackup

We first need to create a directory and mount the Borg repo inside:

Bash

mkdir mount
borg-job-<name-of-job> mount o6h6zl22@o6h6zl22.repo.borgbase.com:repo ./mount

By default borg looks for any ssh keys in the location and format of ~/.ssh/id_*, if you need to pass in an alternate ssh key you can use the following format:

Bash

borg-job-<name-of-job> mount --rsh 'ssh -i /var/run/agenix/borgSSHKey' o6h6zl22@o6h6zl22.repo.borgbase.com:repo ./mount

We are then presented with a list of snapshot dates to choose from, and can select the time period at which we wish to restore. Copy across files as normal e.g. using the cp command

When everything has been restored we can unmount with the below command:

Bash

borg-job-<name-of-job> umount /mount

Final Thoughts

I ended up using BorgBackup as it meant I could configure almost everything once in my NixOS configuration and not have to think about it again. I setup email alerts in BorgBase to notify me if no backups are completed after a certain period of time, so that I don’t have to constantly monitor them. Another benefit of BorgBackup was that with my configuration the backups are performed as root, so I didn’t have to worry about file permissions when backing up e.g. /var/lib/nextcloud (obviously you need to decide whether this is the appropriate approach for you).

If you want an intuitive UI, without the fuss of the initial configuration then Duplicati is also a great solution. It also has the advantage of providing a much larger choice for the backup targets, with BorgBackup you need to store everything in a Borg repo (which you have the option of self hosting).

Installing NixOS on a Temporary File System

2023-02-06T00:00:00+00:00

In this post I explain how to install NixOS using a temporary file system (tmpfs). This is a precursor to the post I wrote about configuring NixOS for impermanence, and walks through the installation.

Useful References & Notes

The following is summarised from a combination of 2 different blog posts: one detailing how to configure a tmpfs on NixOS and another showing how to harden the NixOS install.

The instructions setup a very simple file system, without a swap partition, just a 512MB boot partition and a root partition on the remaining space.

Note

This method is used to create a legacy boot partition, as I use it to create a virtual machine inside of Proxmox. Refer to the linked post above on configuring a tmpfs for UEFI instructions.

Before proceeding you will need a copy of the NixOS Minimal ISO.

Partitioning & Labelling Drives

If unsure first check which drives are available, and confirm which to install on with lsblk. I will be installing on /dev/sda, replace this with the relevant drive if it’s different.

Bash

# Create legacy boot partition of 512MB
parted /dev/sda -- mklabel msdos
parted /dev/sda -- mkpart primary ext4 1M 512M
parted /dev/sda -- set 1 boot on

# Create root partition on remaining storage
parted /dev/sda -- mkpart primary ext4 512MiB 100%

# Label both partitions
mkfs.ext4 -L boot /dev/sda1
mkfs.ext4 -L nix /dev/sda2

Mount Drive & Create Folders

We need to setup the following folder structure on the drive we prepared before starting a NixOS install.

Bash

# Create root mount with tmpfs
mount -t tmpfs none /mnt

# Create folder structure to persist in /nix/persist - srv is optional, used as home directory for services
mkdir -p /mnt/{boot,nix,etc/{nixos,ssh},var/{lib,log},srv}

# Mount relevant partitions to each folder
mount /dev/sda1 /mnt/boot
mount /dev/sda2 /mnt/nix

# Create matching folders in /mnt/nix/persist
mkdir -p /mnt/nix/persist/{etc/{nixos,ssh},var/{lib,log},srv}

# Create temporary bind mounts (later replaced with impermanence - refer to post linked above)
mount -o bind /mnt/nix/persist/etc/nixos /mnt/etc/nixos
mount -o bind /mnt/nix/persist/var/log /mnt/var/log

# Generate a base config
nixos-generate-config --root /mnt

Editing Configuration Files

Before completing the final install we need to make some changes to the generated configuration:

Nix · /mnt/etc/nixos/hardware-configuration.nix

{
  fileSystems."/" = {
    device = "none";
    fsType = "tmpfs";
    # add the line below - limits root storage to 2G maximum
    options = [ "defaults" "size=2G" "mode=755" ];
  };

  # Update both /boot and /nix to use labels rather than UUID
  fileSystems."/boot" = {
    device = "/dev/disk/by-label/boot";
    fsType = "ext4";
  };

  fileSystems."/nix" = {
    device = "/dev/disk/by-label/nix";
    fsType = "ext4";
  };
}

You will also need to edit /mnt/etc/nixos/configuration.nix, how you go about this depends on your needs. Personally I only need to prepare the configuration for deployment via deploy-rs:

Nix · /mnt/etc/nixos/configuration.nix

{ config, pkgs, ... }:

{
  imports = [ ./hardware-configuration.nix ];

  boot.loader.grub.enable = true;
  boot.loader.grub.version = 2;
  boot.loader.grub.device = "/dev/sda";

  # Enable SSH
  services.openssh.enable = true;

  # Configure user with sudo access and SSH key authentication
  users.mutableUsers = false; # prevents any changes to users outside of config file
  users.users.mm = {
    isNormalUser = true;
    extraGroups = [ "wheel" ];
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMne13aa88i97xAUqU33dk2FNz+w8OIMGi8LH4BCRFaN"
    ];
  };

  # Persist host ssh keys to enable decrycption of agenix secrets (see below)
  environment.etc."ssh/ssh_host_rsa_key".source
    = "/nix/persist/etc/ssh/ssh_host_rsa_key";
  environment.etc."ssh/ssh_host_rsa_key.pub".source
    = "/nix/persist/etc/ssh/ssh_host_rsa_key.pub";
  environment.etc."ssh/ssh_host_ed25519_key".source
    = "/nix/persist/etc/ssh/ssh_host_ed25519_key";
  environment.etc."ssh/ssh_host_ed25519_key.pub".source
    = "/nix/persist/etc/ssh/ssh_host_ed25519_key.pub";

  # enable nix flakes
  nix.extraOptions = ''
    experimental-features = nix-command flakes
  '';

  # allow for deployment without entering password
  security.sudo.wheelNeedsPassword = false;

  system.stateVersion = "22.11";

}

If you plan on using /etc/nixos/ocnfiguration.nix for configuration, then you will want to ensure your created user also has a password. You may also want to edit the timezone, network settings, hostname and bootloader.

Complete NixOS Install

Finally we run the following to install NixOS:

Bash

nixos-install --no-root-passwd

Deploy-rs & Agenix Preparation

This final section is likely irrelevant to most users. I plan on writing about these tools in a future post. Deploy-rs (linked above) allows for deployment of nix flakes to remote machines, and Agenix encrypts any secrets used in the flakes using SSH keys (I’ve written about it).

To ensure the host has the correct SSH keys to allow for decryption of secrets be sure to update secrets.nix with the new host key /etc/ssh/ssh_host_ed25516_key.pub. Agenix will then need to be rekeyed to allow any new SSH keys to decrypt secrets:

To find out the new host key

Bash

ssh-keyscan -t ssh-ed25519 hostname

To rekey Agenix secrets

Bash

nix run github:ryantm/agenix -- --rekey -i /path/to/original/ssh/key

Configuring NixOS for Impermanence

2023-01-18T00:00:00+00:00

After reading posts from both Graham Christensen and Elis Hirwing about the idea of impermanence in NixOS I decided to try setting it up on my homelab. The basic idea is that we configure the system to use a temporary file system, where everything is wiped after a reboot, unless specified otherwise.

One of the major benefits of this kind of setup is that it forces you to properly declare the configuration, as everything will break after a reboot otherwise.

Initial Setup

For the initial installation of NixOS I followed Elis’ walk through (linked above) and configured root using a temporary file system. Once this was complete I added the nix community tool impermanence to /etc/nixos/configuration.nix to allow for a declarative configuration going forward.

Nix · /etc/nixos/configuration.nix

{ ... }:

let
  impermanence = builtins.fetchTarball {
    url = "https://github.com/nix-community/impermanence/archive/master.tar.gz";
  };
in
{
  imports = [
    ./hardware-configuration.nix # should be present by default
    "${impermanence}/nixos.nix"
  ];
}

Configuring Persistence

With the above complete I was able to configure directories I needed to persist from within /etc/nixos/configuration.nix.

Nix · /etc/nixos/configuration.nix

{ ... }:

{
  environment = {
    persistence."/nix/persist" = {
      directories = [
        "/etc/nixos" # configuration files
        "/srv" # service data
        "/var/log" # where journald dumps logs
        "/var/lib" # system service persistant data
        "/data" # where I store local media
      ];
      files = [
        "/etc/machine-id" # ensures logs are retained after reboot
        "/users/admin" # directory where I store passwordFile for user
      ];
    };
    etc = {
      # ensures SSH keys can be correctly generated
      "ssh/ssh_host_rsa_key".source = "/nix/persist/etc/ssh/ssh_host_rsa_key";
      "ssh/ssh_host_rsa_key.pub".source = "/nix/persist/etc/ssh/ssh_host_rsa_key.pub";
      "ssh/ssh_host_ed25519_key".source = "/nix/persist/etc/ssh/ssh_host_ed25519_key";
      "ssh/ssh_host_ed25519_key.pub".source = "/nix/persist/etc/ssh/ssh_host_ed25519_key.pub";
    };
  };
}

Initially I ran into issues configuring Tailscale as I would be logged out after each reboot, I found persisting /var/lib fixed the issue as it ensures the Tailscale state, located at /var/lib/tailscale/tailscale.state, is always available.

I setup /srv as the directory I keep data relating to other services I run on the machine, for services which allow me to specify it.

Managing Users

You may have noticed in the configuration above I also persist /users/admin, this allows me to store a hashed password for the user inside of the directory, which is read from within /etc/nixos/configuration.nix. The alternative is to either include a password directly, which would then be available for everyone to see in the nix store, or to add a temporary password and change it on each reboot:

Nix · /etc/nixos/configuration.nix

{ ... }:

{
  users = {
    mutableUsers = false;
    users = {
      admin = {
        isNormalUser = true;
        home = "/home/admin";
        # generate with: nix-shell --run 'mkpasswd -m SHA-512 -s' -p mkpasswd
        passwordFile = "/nix/persist/users/admin";
        extraGroups = [ "wheel" ];
      };
    };
  };
}

How to Configure systemd Services and Timers in NixOS

2023-01-12T00:00:00+00:00

Whilst migrating my homelab from Ubuntu to Nixos I came across the need to schedule a cron job, which I had setup to sync my media collection from a remote server. After doing a bit of research I discovered that rather than using a cron job, the preferred method is to create a systemd service and schedule it using a systemd timer. The major benefits of doing so are:

Jobs can be isolated to specific users
Environment variables and paths can be explicitly declared
Logging is easily viewed by systemctl start example followed by journalctl --unit=example
Timers let you know when the service will next run systemctl status example.timer

Initial Setup

The first step is to create a dedicated user and group which we can use to isolate the service to, I edited my /etc/nixos/configuration.nix to look like this:

Nix · /etc/nixos/configuration.nix

{

  users = {
    groups.server-sync = {};
    users = {
      server-sync = {
        group = "server-sync";
        isSystemUser = true;
        createHome = true;
        home = "/srv/server-sync";
      };
    };
  };

}

Note

I store all relevant files for my services in /srv, this is how I like to organise my system.

I also created a directory to store all of my media mkdir -p /data/media/music, and one to store my backups (for roon-server at this point) mkdir -p /data/backups/roon-server).

Finally correct ownership of the new directories to be as follows:

chown root:root /data
chown -R server-sync:server-sync /data/media
chown -R roon-server:roon-server /data/backups/roon-server

Creating Service & Timer

I then add the following to /etc/nixos/configuration.nix, which defines the service and the timer:

Nix · /etc/nixos/configuration.nix

{

  systemd = {
    services.server-sync = {
      # specify all packages required by the script being scheduled
      path = [
        pkgs.rsync
        pkgs.openssh
      ];
      serviceConfig = {
        Type = "oneshot";
        # specify the user and group we setup earlier
        User = "server-sync";
        Group = "server-sync";
        # security settings to prevent service from having too many priviliges
        ProtectSystem = "full";
        ProtectHome = true;
        NoNewPriviliges = true;
        ReadWritePaths = "/data/media";
      };
      # the action taken when the service runs
      script = builtins.readFile ./server-sync.bash;
    };
    timers.server-sync = {
      wantedBy = [ "timers.target" ];
      timerConfig = {
        # frequency of the service
        OnCalendar = "hourly";
        # the service to associate the timer with
        Unit = "server-sync.service";
      };
    };
  };

}

If it makes sense for your use case, you may simply want to add the script to be run inside of /etc/nixos/configuration.nix.

If you want to read more about the systemd security settings above you can find a high level explanation and details of each individual setting at systemd.exec(5). A helpful command you can run to being reviewing security of services is systemd-analyze security example.service.

Separating Service Script

In case it’s helpful I thought to detail what I have inside of the script being read into the service.

I’m using rsync to pull my music from a remote server to my homelab. I specify the path to an SSH key for authentication to the server by -e "ssh -i /srv/server-sync/.ssh/server" and then the folders which should be kept in sync.

Bash

rsync -nat -e "ssh -i /srv/server-sync/.ssh/server" user@hostname:/home/mm/music/ /data/media/music/

If you run into any errors with SSH, its likely due to incorrect permissions being set for /srv/server-sync. Working permissions are as follows:

chmod 700 /srv/server-sync/.ssh
chmod 644 /srv/server-sync/.ssh/server.pub
chmod 600 /srv/server-sync/.ssh/server
chmod 755 /srv/server-sync

Configuring Proxmox Package Repositories

2022-12-22T00:00:00+00:00

I use Proxmox to create and manage VMs on my existing homelab (2012 Mac Mini). I use a combination of both VMs and containers for running applications and services, as I don’t always like to force everything to run in a container. For a deeper discussion on this topic I recommend listening to this podcast.

Failed Update Error

I recently opened up Proxmox to begin configuring a Nixos VM, and noticed I had a bunch of errors when updating the package database Error: command apt-get update failed: exit code 100. I SSH’d into the machine and tried to update manually, this provided a more detailed output, it appeared I was trying to pull updates from a repo requiring a Proxmox subscription (which I don’t have, as I use it for personal use).

Updating Source List

I managed to find a Proxmox wiki page which provided some clarity. In summary I fixed the issue by commenting out the following repository inside of /etc/apt/sources.list.d/pve-enterprise.list:

Bash

# deb https://enterprise.proxmox.com/debian/pve bullseye pve-enterprise

I then edited /etc/apt/sources.list to look like the following, adding the pve-no-subscription repo:

Bash

deb http://ftp.debian.org/debian bullseye main contrib
deb http://ftp.debian.org/debian bullseye-updates main contrib

# PVE pve-no-subscription repository provided by proxmox.com,
# NOT recommended for production use
deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription

# security updates
deb http://security.debian.org/debian-security bullseye-security main contrib

After the above changes I ran apt update and apt dist-upgrade, everything updated without issue.

Note

Proxmox highlight in the linked wiki that the pve-no-subscription repo isn’t tested as thoroughly as the repo for the paid subscription. For me this was an acceptable compromise, this may be different for you.

Automated Script

Tteck has provided a number of useful Proxmox scripts, one of them completes the above automatically, as well as removing subscription notices and optionally adding beta repositories. Have a read through the script before running:

Bash

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/misc/post-pve-install.sh)"

How to Mount S3 Object Storage in NixOS

2022-12-14T00:00:00+00:00

I previously wrote about how I mounted s3 object storage to my homelab running Ubuntu. I’ve since been configuring my homelab to run using NixOS. NixOS has a major advantage over Ubuntu for me, in that the entire system is configured declaratively, meaning once you have a working configuration you can use it to rebuild everything exactly how it was setup before.

Updating NixOS Configuration

Whilst you could in theory install the required tools and manually mount an s3 object store in NixOS, a better approach is to specify mount instruction in the NixOS configuration file.

After a default install of NixOS the configuration file is available to edit at /etc/nixos/configuration.nix, to mount an s3 bucket we will expand on the configuration.nix by adding a new module, called s3fs. We will do this by importing a separate s3fs.nix file, which we will create shortly, for now edit /etc/nixos/configuration.nix and add the following:

Nix · /etc/nixos/configuration.nix

{ config, pkgs, ... }:

{
  imports = [
    ./hardware-configuration.nix # should be present by default
    ./modules/s3fs.nix
  ];

  services.s3fs.enable = true;
}

Add Modules Directory

Next we need to create the modules folder and create the s3fs.nix file:

Bash

sudo mkdir /etc/nixos/modules && sudo touch s3fs.nix

Add s3fs Module

Finally we can edit /etc/nixos/modules/s3fs.nix and add the following:

Nix · /etc/nixos/modules/s3fs.nix

{ config, lib, pkgs, ... }:

with lib;

let
  cfg = config.services.s3fs;
in {
  options.services.s3fs = {
    enable = mkEnableOption "Mounts s3 object storage using s3fs";
    keyPath = mkOption {
      type = types.str;
      default = "/etc/passwd-s3fs";
    };
    mountPath = mkOption {
      type = types.str;
      default = "/mnt/data";
    };
    bucket = mkOption {
      type = types.str;
      default = "data";
    };
    url = mkOption {
      type = types.str;
      default = "https://ap-south-1.linodeobjects.com/";
    };
  };

  config = mkIf cfg.enable {
    systemd.services.s3fs = {
      description = "Linode object storage s3fs";
      wantedBy = [ "multi-user.target" ];
      serviceConfig = {
        ExecStartPre = [
          "${pkgs.coreutils}/bin/mkdir -m 777 -pv ${cfg.mountPath}"
          "${pkgs.e2fsprogs}/bin/chattr +i ${cfg.mountPath}" # stop files being written to unmounted dir
        ];
        ExecStart = let
          options = [
            "passwd_file=${cfg.keyPath}"
            "use_path_request_style"
            "allow_other"
            "url=${cfg.url}"
            "umask=0000"
          ];
        in
          "${pkgs.s3fs}/bin/s3fs ${cfg.bucket} ${cfg.mountPath} -f "
            + lib.concatMapStringsSep " " (opt: "-o ${opt}") options;
        ExecStopPost = "-${pkgs.fuse}/bin/fusermount -u ${cfg.mountPath}";
        KillMode = "process";
        Restart = "on-failure";
      };
    };
  };
}

The above sets up a module for services.s3fs, allowing us to enable/disable the service as well as edit the default settings. When enabled a systemd service is created which mounts the s3 bucket, making it available at the specified path (/mnt/data by default).

The service will look for a file containing an access and API key by default in /etc/passwd-s3fs, before running the service you will need to run the following to configure authentication details:

Bash

echo "ACCESS_KEY:SECRET_KEY" | sudo tee /etc/passwd-s3fs
sudo chmod 600 /etc/passwd-s3fs

Note

I recently ran into an issue when setting this up and had to comment out "${pkgs.e2fsprogs}/bin/chattr +i ${cfg.mountPath}" to resolve the issue. I’ll update if I figure out a solution. For now everything is working without it.

Much of the credit for this post goes to a forum post on the NixOS Discourse.

How to Mount S3-Compatible Object Storage on Ubuntu

2022-11-29T00:00:00+00:00

I’m currently in the process of setting up a homelab, one of the major tasks I want to manage is storage and streaming of music.

For streaming music and managing metadata I’m using Roon Server, this may change in the future as I explore other options. I store my music on a 2012 Mac Mini (the only computer I had lying around) which is running Ubuntu 20.04 as a VM in Proxmox. My Mac Mini has 2 x 250GB SSD storage, which has been quickly filling up as my music collection grows.

My end goal is to build a dedicated homelab with a large amount of storage, either as ZFS or a combination of MergerFS and SnapRAID. Unfortunately building a dedicated homelab requires a significant financial contribution. I needed to find an interim solution - that’s where S3 compatible object storage comes in.

I hadn’t previously used object storage, but had heard it worked well for storing files that don’t change frequently e.g. media files and music. After doing some quick research I discovered that I could mount an object store to a computer running Linux, macOS or FreeBSD using s3fs-fuse. I decided to test how well this worked by mounting the object store, adding some music to it and then pointing Roon Server to the object store. The steps I followed to mount the object store are detailed in this guide, my own summarised version is below.

Mounting Storage

The steps may vary depending on the provider you are using for object storage, for reference I am using Linode.

Install s3fs-fuse

On Ubuntu this is nice and easy, the package name will likely vary depending on your Linux distro:

Bash

sudo apt install s3fs

Authentication

After creating an object store you will also need to create access keys. These should be a combination of:

An ACCESS_KEY
A SECRET_KEY

We need to store the access key on the machine we plan to mount the object store to, as it will be needed to authenticate the connection:

Bash

echo "ACCESS_KEY:SECRET_KEY" | sudo tee /etc/passwd-s3fs
sudo chmod 600 /etc/passwd-s3fs

Mounting

First we need to create the directory the object store will be mounted to:

Bash

mkdir /mnt/my-object-storage

Then we can mount the storage at the directory created above, filling in the placeholders with the appropriate values:

Bash

sudo s3fs {bucketname} {/mnt/my-object-storage} -o passwd_file=/etc/passwd-s3fs -o allow_other -o url=https://{private-network-endpoint}

{bucketname} = the name of the bucket you wish to mount
{/mnt/my-object-storage} = directory to mount to the object store to (created earlier)
{private-network-endpoint} = the url given by your provider to the object store e.g. https://ap-south-1.linodeobjects.com
the allow_other flag allows users other than root to have access to the bucket

Testing

Assuming you didn’t run into any errors after following the above instructions, you can now test the storage. I did this by opening a web browser and viewing the object store within my account dashboard at Linode, and copying some new files in to the object store on my homelab. If the file match between the web browser and your local machine then its a good sign that things are working. If this isn’t the case unmount the storage and try again, double checking the values provided when making the connection.

Persisting After Reboot

Now that everything has been mounted properly we can setup an entry within /etc/fstab, so that the object store is mounted whenever the local machine reboots.

First we need to unmount:

Bash

sudo umount /etc/my-object-storage

Then we can edit /etc/fstab in whichever editor you are most comfortable with (vim is the correct answer), and add the following entry on a new line:

Bash

s3fs#bucketname /mnt/my-object-storage fuse _netdev,allow_other,passwd_file=/etc/passwd-s3fs,url=https://ap-south-1.linodeobjects.com/ 0 0

If you want more details on what each of the options do then I recommend checking out the FAQs at the s3fs GitHub repo

Final Thoughts

How well did this work for music streaming? I was able to point Roon to music on the object store, scan tracks and stream music, though there were some limitations in performance:

When selecting music to play there was ~3-5 second delay before the track started, when streaming from local storage or Tidal this is pretty much instant
Scanning tracks and collecting metadata with Roon took significantly longer - 1 album on the object store took longer than my entire library of ~1,500 tracks on local storage
Copying data across to the object store took significantly longer than an rsync between machines, which I do between my local homelab and another server outside of my network

The above thoughts seem to align with my initial research into object storage - it works well for reading data that doesn’t change frequently, once you start copying data across regularly then things slow down. If you are happy to accept a slight delay when first choosing music to play (this doesn’t seem to be an issue when moving on to play the next track in an album), then this solution could work well. Backups seem an obvious use case for object storage, I’m currently backing up my Roon settings and library to the object store, I schedule these backups at night and it is working well.