Skip to main content

Step-CA PKI Deployment — Internal Certificate Authority on fablehaven

Summary​

TheBurrow required an internal Certificate Authority to issue TLS certificates for fleet services — Pi-hole admin UIs, Caddy reverse proxy, NetBox, and devpi — without relying on public ACME challenges or self-signed certs per service.

Step-CA was selected as the CA (see ADR #430) and deployed on fablehaven as a systemd service via Ansible. The deployment covers the CA itself (step_ca role), client certificate issuance and renewal (step_ca_client role), and the integration patterns that connect them to fleet services.

This lab note documents the architecture, the Ansible role design, the key operational decisions, and the notable failures encountered during deployment.

Architecture​

CA topology​

fablehaven (192.168.95.6)
└── step-ca (systemd service, port 9000)
    ├── Root CA certificate
    ├── Intermediate CA certificate
    └── JWK provisioner (fablehaven-provisioner)

Fleet hosts (mistborn, hogwarts, fablehaven, locker9, conair, cosmere)
└── step_ca_client role
    ├── step CLI installed
    ├── Root CA cert trusted (system trust store)
    ├── Host certificate issued via JWK provisioner
    └── Renewal timer (systemd, runs daily)

Certificate flow​

  1. step_ca role deploys CA on fablehaven, generates root and intermediate certs, writes ca.json with JWK provisioner config
  2. step_ca_client role runs on each fleet host:
    • Installs step CLI
    • Fetches and trusts root CA cert
    • Issues host cert via step ca certificate with JWK provisioner
    • Installs renewal wrapper script and systemd timer
  3. Caddy on fablehaven terminates TLS for Pi-hole admin UI using the host cert issued by the internal CA
  4. NetBox and devpi use certs issued by the same CA

Trust model​

All fleet hosts trust the TheBurrow root CA via the system trust store (/usr/local/share/ca-certificates/). Internal services use certs signed by the intermediate CA. No public CA involvement.

Notes​

Ansible role design​

step_ca role (server)​

Numbered task files following the repo convention:

roles/step_ca/tasks/
├── 00_guard.yml        # scope guard
├── src/
│   ├── 05_preflight.yml  # assert vault vars, OS check
│   ├── 10_install.yml    # download step-ca binary, verify SHA256
│   ├── 20_user.yml       # create step-ca system user
│   ├── 30_dirs.yml       # create /etc/step-ca, /var/lib/step-ca
│   ├── 40_config.yml     # initialize CA if not already done
│   ├── 50_service.yml    # deploy systemd unit, enable, start
│   └── 55_service.yml    # reload after config changes

Key design decisions:

  • CA initialization is idempotent: ConditionFileNotEmpty on ca.json prevents re-init on subsequent runs
  • Binary installed to /usr/local/bin/step-ca with SHA256 checksum verification against the fleet versions manifest
  • systemd service runs as step-ca user (not root) with ProtectSystem=strict and PrivateTmp=true
  • JWK encryptedKey stored in vault as a valid JSON string (see case study: JWK encryptedKey corruption)

step_ca_client role (fleet)​

roles/step_ca_client/tasks/
├── 00_guard.yml
├── src/
│   ├── 05_preflight.yml   # assert fingerprint, CA URL, vault password
│   ├── 10_install.yml     # download step CLI, verify SHA256
│   ├── 20_root_ca.yml     # fetch root CA cert, add to trust store
│   ├── 30_cert.yml        # issue host cert via JWK provisioner
│   └── 40_renewal.yml     # install renewal wrapper + systemd timer

Key design decisions:

  • step ca certificate uses argv list format (not shell string) to avoid quoting issues with SANs and paths
  • Provisioner password written to temp file, used for cert issuance, deleted in always block regardless of success or failure
  • Renewal wrapper script checks exit code: 0 = renewed, 1 = not yet due (normal), other = error
  • ansible_fqdn returns short hostnames on fleet — use inventory_hostname for cert subject (e.g., fablehaven.home.arpa)
  • --fingerprint flag removed in step CLI v0.30.2 — use --root with the root CA cert path instead

Checksum verification pattern​

All step CLI and step-ca binary downloads verified against SHA256 checksums stored in the fleet versions manifest (group_vars/all/fleet_versions.yml):

fleet_versions:
  step_ca:
    version: "0.30.2"
    arm64_sha256: "abc123..."
    amd64_sha256: "def456..."
  step_cli:
    version: "0.30.2"
    arm64_sha256: "..."
    amd64_sha256: "..."

Architecture map in role vars handles arm64 (Pi fleet) vs amd64 (x86 hosts) automatically.

JWK provisioner bootstrap​

The JWK provisioner private key cannot be generated by Ansible — it requires an interactive step crypto jwk create command. The bootstrap process:

  1. Generate JWK keypair on fablehaven manually
  2. Store encryptedKey (JSON string) in vault via vault_edit.py
  3. Store provisioner password in vault under vault_secrets_step_ca.provisioner_password
  4. Run step_ca role — ca.json is templated from vault values

See ADR #448 for the full decision rationale. See the JWK encryptedKey corruption case study for what happens when vault_edit.py serializes the value incorrectly.

Certificate subject and SANs​

Fleet host certs use:

  • Subject: inventory_hostname (e.g., fablehaven.home.arpa)
  • SANs: inventory_hostname + ansible_host (IP address)

This ensures certs are valid for both DNS and IP-based access. Caddy uses the cert's SAN list when terminating TLS.

Renewal​

Each fleet host runs a systemd timer daily that calls the renewal wrapper script:

#!/bin/bash
/usr/local/bin/step ca renew \
  /etc/step/certs/host.crt \
  /etc/step/certs/host.key \
  --expires-in 720h \
  --daemon
EXIT=$?
if [ $EXIT -eq 0 ]; then
  systemctl reload caddy 2>/dev/null || true
fi
exit $EXIT

--expires-in 720h means renewal only occurs if the cert expires within 30 days. Exit code 1 from step means "not yet due" — the wrapper treats this as success.

Insights​

The fragility of binary flag APIs​

Step CLI v0.30.2 removed several flags that appeared in documentation and were present in earlier versions: --fingerprint, --no-password, --key-type, --curve. Each removal caused a deployment failure that required a fix PR. Version-pinning the CLI and testing against the pinned version before fleet rollout is essential.

argv format over shell strings for certificate commands​

The step ca certificate command takes a subject, cert path, key path, and multiple flags. Building this as a shell string creates quoting problems with spaces in paths and SANs. Using Ansible's argv list format eliminates the quoting surface entirely:

- name: Issue host certificate
  ansible.builtin.command:
    argv:
      - /usr/local/bin/step
      - ca
      - certificate
      - "{{ inventory_hostname }}"
      - "{{ step_ca_client_cert_path }}"
      - "{{ step_ca_client_key_path }}"
      - "--provisioner"
      - "{{ step_ca_client_provisioner_name }}"
      - "--provisioner-password-file"
      - "{{ step_ca_client_provisioner_password_file }}"
      - "--root"
      - "{{ step_ca_client_root_ca_path }}"
      - "--san"
      - "{{ ansible_host }}"
      - "--not-after"
      - "{{ step_ca_client_cert_lifetime }}"
      - "--force"

Provisioner password cleanup is non-negotiable​

The provisioner password grants the ability to issue certificates for any hostname in the CA's namespace. It must be written to disk only for the duration of cert issuance and deleted immediately after — success or failure. Ansible block/always pattern:

- block:
    - name: Write provisioner password
      ansible.builtin.copy:
        content: "{{ vault_step_ca_client_provisioner_password }}"
        dest: "{{ step_ca_client_provisioner_password_file }}"
        mode: "0600"

    - name: Issue certificate
      ansible.builtin.command:
        argv: [...]

  always:
    - name: Remove provisioner password file
      ansible.builtin.file:
        path: "{{ step_ca_client_provisioner_password_file }}"
        state: absent

Root CA path must use inventory_dir not playbook_dir​

The root CA cert is stored relative to the inventory directory. Using playbook_dir caused failures when playbooks were called from different locations. inventory_dir is stable regardless of where the playbook is invoked from.

Results​

  • Step-CA running on fablehaven, issuing certs for the fleet
  • All Pi-hole nodes (mistborn, fablehaven, hogwarts) hold valid certs
  • Caddy on fablehaven terminates TLS for Pi-hole admin UIs at pihole-mistborn.home.arpa, pihole-fablehaven.home.arpa, pihole-hogwarts.home.arpa
  • NetBox accessible at netbox.home.arpa over HTTPS
  • devpi accessible at devpi.home.arpa over HTTPS
  • Daily renewal timers active on all cert-holding hosts
  • Root CA trusted fleet-wide via system trust store
  • newcomb-labs/engineering-journal-kb #430 — ADR: internal CA tooling selection
  • newcomb-labs/engineering-journal-kb #448 — ADR: JWK provisioner bootstrap strategy
  • newcomb-labs/engineering-journal-kb #458 — lab note: Caddy reverse proxy
  • newcomb-labs/engineering-journal-kb #467 — case study: JWK encryptedKey corruption
  • mistborn-ops/ansible #287–#404 (step_ca, step_ca_client role PRs)
  • newcomb-labs/engineering-journal-kb #457 (this issue)

Related Content​