Deploying stateless web app on a NixOS server

I deploy this website with a single command. No Docker, no CI pipeline, just 1 VPS and 1 public repo. The whole CD is a small bash script and Nix configurations.

graph TD
    DEPLOY["deploy.sh (personal laptop)"] -->|"1. git push"| REPO
    DEPLOY -->|"2. ssh: nix flake update && nixos-rebuild"| SERVER

    subgraph REPO["App Repository (Codeberg)"]
        FLAKE["flake.nix"]
        PKG["package.nix<br/>(production build)"]
        MOD["nixos-module.nix<br/>(systemd service definition)"]

        PKG -->|"included in"| MOD

        FLAKE --> PKG
        FLAKE --> MOD
    end

    subgraph SERVER["NixOS Server"]
        SCONFIG["server config"]
        NGINX["Nginx<br/>SSL via ACME"]
        SERVICE["ffloyd-space.service<br/>(hardened systemd unit)"]

        SCONFIG --> NGINX
        SCONFIG --> SERVICE

        SERVICE -->|"exposed via"| NGINX
    end

    MOD -->|"imported by"| SCONFIG

The article walks through each piece of this diagram. We'll package the app as a Nix derivation, wrap it in a hardened systemd service, proxy it through Nginx with automatic SSL, and write ./deploy.sh for single command deployment.

Going beyond Docker

I feel that Docker did a bad thing to the engineering community. It became "the default way to run web apps". It has advantages, totally agree. But it looks like engineers stopped considering and actively discussing alternatives. For example:

• A mere systemd service
• systemd-nspawn
• FreeBSD jails

When I can, I prefer to try and learn something more foundational. NixOS simplifies work with the first two mentioned alternatives; this article focuses on the first one. In the end, the knowledge I gain this way helps me resolve issues with Docker when I encounter them.

Packaging the App

In order to deploy something as a systemd service I need a way to run the app. Practically, it means to create a Nix package that starts the web server. In my case of using combination of Bun and Vite (because of SvelteKit) I found that half of packaging work is already generalized by bun2nix.

bun2nix cannot fully solve packaging for me because building is done by Vite (using Bun runtime), not directly via Bun.

Packaging the app is the app's responsibility. App's repository already had Nix flake for dev shell, so I merely extended it. Here's how it looks at the time of writing:

{
  description = "A Nix-flake-based Node.js/Bun development environment";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs?ref=master";

    bun2nix = {
      url = "github:nix-community/bun2nix";
      inputs.nixpkgs.follows = "nixpkgs";
    };
  };

  # Use the cached version of bun2nix from the nix-community cli
  nixConfig = {
    extra-substituters = [
      "https://nix-community.cachix.org"
    ];
    extra-trusted-public-keys = [
      "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs="
    ];
  };

  outputs = {
    self,
    nixpkgs,
    bun2nix,
    ...
  }: let
    supportedSystems = ["x86_64-linux" "aarch64-linux" "x86_64-darwin" "aarch64-darwin"];

    forEachSupportedSystem = f:
      nixpkgs.lib.genAttrs supportedSystems (system:
        f {
          pkgs = import nixpkgs {
            inherit system;
            #
            # !!! I've added overlay, so I have `bun2nix` in my packages "implicitly"
            #
            overlays = [bun2nix.overlays.default];
          };
        });
  in {
    formatter = forEachSupportedSystem ({pkgs, ...}: pkgs.alejandra);

    devShells = forEachSupportedSystem ({pkgs, ...}: {
      default = pkgs.mkShell {
        packages = with pkgs; [
          # runtime
          bun
          nodejs # without it even `bun dev` takes +10 second to start (I dunno why)

          pkgs.bun2nix

          # LSP
          typescript-language-server
          svelte-language-server
          tailwindcss-language-server
          vscode-langservers-extracted

          # For LLMs (they like to write scripts sometimes)
          python3
        ];
      };
    });

    packages = forEachSupportedSystem ({pkgs, ...}: rec {
      #
      # !!! Here I wire package definition to the flake output
      #
      default = pkgs.callPackage ./nix/package.nix {};
      dockerImage = pkgs.callPackage ./nix/docker-image.nix {app = default;};
    });

    nixosModules.default = import ./nix/nixos-module.nix self;
  };
}

I refer to 3 external files here. Each will be shown and discussed in the article. This section's goal is achieved by package.nix (which is callPackage-style package definition):

{
  bun2nix,
  bun,
  lib,
  makeWrapper,
  ...
}:
bun2nix.mkDerivation rec {
  pname = "ffloyd.space";

  packageJson = ../package.json;

  nativeBuildInputs = [makeWrapper];

  # changes in Git state and flake.nix should not trigger app rebuild
  src = lib.cleanSourceWith {
    filter = path: _type: !(lib.elem (baseNameOf path) ["flake.nix" "nixos-module.nix" "docker-image.nix" "deploy.sh"]);
    src = lib.cleanSource ./..;
  };

  NODE_ENV = "production";

  bunDeps = bun2nix.fetchBunDeps {
    bunNix = ../bun.nix;
  };

  buildPhase = ''
    runHook preBuild

    bun --bun run build

    # Remove development dependencies
    rm -rf node_modules
    bun install --ci

    runHook postBuild
  '';

  installPhase = ''
    runHook preInstall

    mkdir -p $out/bin

    cp -R ./build $out
    cp -R ./node_modules $out
    cp -R ./package.json $out

    makeWrapper ${lib.getExe bun} $out/bin/${pname} \
      --chdir $out \
      --add-flag "./build/index.js" \
      --set NODE_ENV production

    runHook postInstall
  '';
}

I had to override bun2nix's build and install phases because of Vite.

As a result, in the app's repo I can build & start production build of my app merely by running nix run. The web server becomes a mere Nix package and can be executed in any environment that supports Nix (which covers basically most Linux systems, macOS, and even Windows).

If you have Nix installed (and since the repo is public), you can build and run my app locally by running:

nix run git+https://codeberg.org/ffloyd/ffloyd.space.git

It's not the only way how to package such app. For example, you can rely on a different tool than bun2nix or write more explicit package using default stdenv.mkDerivation.

bun2nix focuses on reproduciblity. It translates bun.lock to Nix dependencies. It makes dependency resolution slow when building for the first time (because Nix is slower than Bun). But after first build everythings is cached. From the server perspective it's like having a separate cache for each dependency. Unlike Docker, if only one dependency changes - it will not force rebuild of all the dependency layer.

If you want you can speed it up reporoducing "single dependency layer made by bun install" approach. But this trick is out of the scope of this article.

Systemd service

NixOS is a systemd-based OS. A systemd service is the natural way to run a web server natively.

Another file referenced in flake.nix is nixos-module.nix. It defines a NixOS module that I can use in my NixOS server configuration. The NixOS module defines options that enable a systemd service wrapping our app.

flake: {
  config,
  lib,
  pkgs,
  ...
}: let
  cfg = config.services.ffloyd-space;
in {
  options.services.ffloyd-space = {
    enable = lib.mkEnableOption "ffloyd.space personal website (SvelteKit app)";

    package = lib.mkOption {
      type = lib.types.package;
      description = "The package to use";
      default = flake.packages.${pkgs.system}.default;
    };

    port = lib.mkOption {
      type = lib.types.port;
      default = 3000;
      description = "Port for the web server to listen on";
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.ffloyd-space = {
      description = "Personal Web Site";

      wantedBy = ["multi-user.target"];

      after = ["network-online.target"];
      wants = ["network-online.target"];

      serviceConfig = {
        Type = "simple";
        ExecStart = "${lib.getExe cfg.package}";
        Restart = "always";

        # Hardening
        # Docs: https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
        # Guides:
        # - https://lincolnloop.com/blog/sandboxing-services-systemd/
        # - https://github.com/HorlogeSkynet/systemd-hardene.d/blob/master/docs/systemd_service_hardening.md

        # Very convenient setting. In addition to creating a temporary user/group for a unit
        # it implies the following:
        # - RemoveIPC (and cannot be turned off)
        # - NoNewPrivileges & RestrictSUIDSGID implicitly enabled
        # - ProtectSystem=strict, ProtectHome=read-only
        DynamicUser = "yes";
        ProtectHome = "tmpfs"; # completely remove access to real home dirs

        PrivateTmp = "yes";
        PrivateBPF = "yes";
        PrivateDevices = "yes";
        PrivateMounts = "yes";
        PrivateUsers = "yes";

        ProtectKernelLogs = "yes";
        ProtectKernelModules = "yes";
        ProtectKernelTunables = "yes";
        ProtectClock = "yes";
        ProtectProc = "noaccess";
        ProtectHostname = "yes";
        ProtectControlGroups = "yes";
        ProcSubset = "pid";

        SystemCallArchitectures = "native";
        SystemCallFilter = [
          "@system-service"
          "~@privileged"
        ];

        RestrictNamespaces = true;
        RestrictAddressFamilies = ["AF_INET" "AF_INET6" "AF_UNIX"];
        RestrictRealtime = "yes";

        LockPersonality = "yes";

        CapabilityBoundingSet = "";

        IPAddressAllow = "localhost";

        DeviceAllow = ["/dev/stdin r"];
        DevicePolicy = "closed";

        UMask = "0027";
      };

      environment = {
        PORT = toString cfg.port;
      };
    };
  };
}

And on the side of NixOS server configuration, after I add my app's flake to the inputs:

{
  # ...
  inputs = {
    # ...
    ffloyd-space = {
      url = "git+https://codeberg.org/ffloyd/ffloyd.space.git";
      inputs.nixpkgs.follows = "nixpkgs";
    };
    # ...
  };
  # ...
}

I can create the following NixOS module on the server that uses NixOS module from app's flake:

{
  inputs,
  lib,
  config,
  pkgs,
  ...
}: let
  domain = "ffloyd.space";
  localPort = 3001;
in {
  imports = [inputs.ffloyd-space.nixosModules.default];

  # this part is explained in the previous article
  services.nginx.virtualHosts.${domain} = {
    forceSSL = true;
    enableACME = true;

    locations."/".proxyPass = "http://localhost:${toString localPort}";
  };

  # that's how we use options from the app's NixOS module
  services.ffloyd-space = {
    enable = true;
    port = localPort;
  };
}

The trickiest part here is the systemd service unit configuration that is secure. I followed the mentioned guides, systemd-analyze security command output, systemd documentation, and inspected existing systemd service definitions (ones from the previous article: Forgejo and FoundryVTT). You can use systemctl cat [SERVICE_NAME] to inspect definitions.

During this process I learnt about potential attack vectors I wouldn't normally consider. I think Docker hides such problems from you instead of making you face them and understand them.

What if I need more security & isolation?

You may be not happy with the isolation level of the app when it's mere systemd service. I may not be happy with the current approach in the future.

As a solution we can elevate isolation level by using systemd-nspawn, or to be more specific: NixOS containers. With a NixOS container we can run our systemd service in a separate OS instance. It's an addition to your existing Nix config — not a rewrite. It provides isolation and security guarantees on the level of Docker best practices.

What if I really need Docker?

You can be forced to use Docker. For example, when you have to deploy to Kubernetes.

Remember that 3rd file we were referring from flake.nix: docker-image.nix?

{
  app,
  dockerTools,
  buildEnv,
  ...
}:
dockerTools.buildImage {
  name = app.pname;

  # I can use hash here, but I prefer to overwrite existing latest image
  # to avoid polluting disk with tagged stale images
  tag = "latest";

  copyToRoot = buildEnv {
    name = "image-root";
    paths = [app];
    pathsToLink = ["/bin"];
  };

  # https://github.com/moby/docker-image-spec/blob/v1.3.1/spec.md#image-json-field-descriptions
  config = {
    Cmd = ["/bin/${app.pname}"];
    ExposedPorts = {
      "3000" = {};
    };
  };
}

It allows to wrap your app and all runtime dependencies into a Docker image. No Dockerfile needed. You can build and load Docker image by running:

nix build .#dockerImage && docker image load -i result

In my experience, the resulting image is usually similar in size to a multi-stage Dockerfile for the same app.

Single command deploy

Previously, I hosted the project on fly.io and liked the experience of deploying with a single command: fly deploy.

I achieved similar experience by writing... 1 bash script deploy.sh. It expects that all the changes I want to deploy are locally committed to the main branch.

#!/usr/bin/env bash
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
set -euo pipefail
IFS=$'\n\t'

echo_green() {
  echo -e "\033[0;32m$1\033[0m"
}

echo_green "Make sure we're on the main branch..."
git branch --show-current | grep -q "main"

echo_green "Push the changes (ignore uncommitted)..."
git push origin main

echo_green "Pull and apply changes on the server..."

ssh -T root@server << END
cd /etc/nixos && \
nix flake update ffloyd-space && \
nixos-rebuild --flake .#vps switch
END

It pushes changes to the public repo. Then on the server side it pulls those changes to the flake. This makes the latest source code, systemd unit definitions, and everything app-related available on the server. Then it applies configuration to the server that basically triggers:

• build of a new app version if needed
• update to systemd service definition if needed
• restart of the systemd service if any of above happened

That's all I needed to reproduce fly deploy experience.

And I haven't even touched Nix-based orchestration tools for more complex deployments! Just basic stuff. Very satisfying.

Nix supremacy

I believe that Docker is more popular than systemd-nspawn (and other alternatives) because of the Dockerfile — a portable build definition that works on both dev machines and servers.

Nix fills this gap and goes further. The same flake gives you:

• A dev shell with the exact tooling you need
• A production build that runs on any system with Nix installed (and aligned with software versions from the dev shell)
• A Docker image if you need one — built from the same production artifact
• Reusable NixOS modules that define services, containers, and even entire server configs

And having NixOS server pushes synergy to the next level.

Is it perfect?

Of course not!

For the serious stuff with huge amount of clients you need something more advanced. Being it Nix-based or not.

This setup is good for personal or small projects. It's fun and satisfying to work with. I learnt a lot and felt aestetic pleasure during the process.

And I feel proud each time I write ./deploy.sh.