Skip to main content

Pi-hole VRRP Topology Misconfiguration — Three-Node Cluster Post-Mortem

Summary​

The Pi-hole cluster at TheBurrow runs three nodes — mistborn (.3), fablehaven (.6), and hogwarts (.158) — with keepalived providing VRRP failover across two virtual IPs: VIP .250 (primary DNS) and VIP .251 (secondary DNS). Kea DHCP on nevermore is configured to serve only these VIPs to clients, never the raw Pi-hole IPs.

In June 2026, a routine audit revealed that VRRP failover had been silently broken since the cluster was stood up. Three independent bugs combined to create the failure:

  1. keepalived was not installed on mistborn — the intended preferred master for VIP .250 — so the VIP had never been owned by the correct node.
  2. The VRRP topology was inverted — fablehaven was misconfigured as preferred master for both VIPs instead of just .251.
  3. Kea was serving raw Pi-hole IPs (.3, .6, .158) as DNS servers instead of the VIPs (.250, .251), bypassing VRRP entirely.

Any one of these bugs alone would have been recoverable. Together, they meant clients never used the VIPs, VRRP had never actually protected anyone, and the failure was invisible because DNS appeared to work.

Date: 2026-06-18 Severity: High — silent infrastructure failure, no active outage Resolution: newcomb-labs/opnsense-lab #95, #96, #97

Situation​

TheBurrow's Pi-hole cluster was designed with the following topology:

NodeIPVIP Role
mistborn192.168.95.3Preferred master — VIP .250
fablehaven192.168.95.6Preferred master — VIP .251
hogwarts192.168.95.158Warm standby — both VIPs

Kea DHCP on nevermore (192.168.95.1) should serve only:

  • DNS1: 192.168.95.250
  • DNS2: 192.168.95.251

This design means clients always talk to a VIP. If mistborn goes down, keepalived promotes hogwarts to own .250 within seconds. Clients never need to know which physical node answered.

The intent was sound. The implementation was not.

Task​

Identify why Pi-hole VRRP failover was not working as designed and restore the cluster to the intended topology without causing a DNS outage.

Symptoms​

There were none. That was the problem.

DNS appeared to work. Pi-hole was blocking ads. The cluster looked healthy from every external angle. The failure was discovered only when auditing the cluster configuration directly:

  • ip addr on mistborn showed no VIP address
  • systemctl status keepalived on mistborn returned Unit not found
  • Kea lease inspection showed clients receiving .3, .6, .158 as DNS servers — the raw Pi-hole IPs, not the VIPs

Investigation​

Bug 1 — keepalived not installed on mistborn​

The Ansible role that deployed keepalived had been run against fablehaven and hogwarts but never against mistborn. mistborn was running Pi-hole but had no keepalived process and no VIP configured.

## On mistborn
$ systemctl status keepalived
Unit keepalived.service could not be found.

$ ip addr show | grep 192.168.95.250
## (no output)

Since mistborn never owned VIP .250, the VIP fell to the next eligible node. fablehaven picked it up — but fablehaven was already preferred master for .251. This made fablehaven responsible for both VIPs simultaneously, with hogwarts as the only standby.

Bug 2 — VRRP topology inverted on fablehaven​

Inspecting the keepalived configuration on fablehaven revealed it was configured with a higher priority for both VI_250 and VI_251. The intended design had fablehaven as preferred master only for VI_251, with mistborn owning VI_250.

## fablehaven keepalived.conf (incorrect)
vrrp_instance VI_250 {
    priority 150    # should be 100 — backup priority
    ...
}

vrrp_instance VI_251 {
    priority 150    # correct
    ...
}

This meant even after mistborn got keepalived installed, it would have lost the election for .250 to fablehaven immediately.

Bug 3 — Kea serving raw Pi-hole IPs​

The Kea DHCP configuration on nevermore was serving DNS options with the raw Pi-hole host IPs:

{
  "option-data": [{ "name": "domain-name-servers", "data": "192.168.95.3, 192.168.95.6" }]
}

This meant every DHCP client received mistborn and fablehaven's IPs directly as their DNS servers. VRRP VIPs were configured but never actually used by any client. If mistborn went offline, clients with .3 cached as DNS1 would have experienced DNS failures until their DHCP lease renewed — and even then only if Kea was fixed first.

Why it was invisible​

DNS worked because mistborn (.3) and fablehaven (.6) were both up and answering. The VIPs existed on the network (fablehaven held both), but no client was talking to them. The failure mode — a node going down — had never been triggered, so the silent misconfiguration had never surfaced.

Root Cause​

Three independent configuration defects compounding:

  1. Incomplete Ansible role deployment — keepalived role was not run against mistborn during initial cluster setup.
  2. Incorrect priority values in keepalived config — fablehaven was given master priority for both VIPs instead of just .251.
  3. Kea serving raw IPs instead of VIPs — the DHCP option-data was populated with host IPs, defeating the entire purpose of VRRP.

No single bug caused an outage. Together they meant the cluster had never operated as designed.

Resolution​

Step 1 — Install and configure keepalived on mistborn​

Ran the Ansible keepalived role against mistborn with the correct priority for VI_250:

## mistborn host_vars
keepalived_instances:
  - name: VI_250
    priority: 150 # preferred master
    virtual_ip: 192.168.95.250
  - name: VI_251
    priority: 100 # backup
    virtual_ip: 192.168.95.251

Step 2 — Correct fablehaven keepalived priority for VI_250​

Updated fablehaven's keepalived configuration to reduce its priority for VI_250 to backup level:

## fablehaven host_vars
keepalived_instances:
  - name: VI_250
    priority: 100 # backup
    virtual_ip: 192.168.95.250
  - name: VI_251
    priority: 150 # preferred master
    virtual_ip: 192.168.95.251

After both nodes reloaded keepalived, mistborn won the election for .250 and fablehaven retained .251. Verified with ip addr:

## On mistborn
$ ip addr show | grep 192.168.95.250
    inet 192.168.95.250/32 scope global eth1

## On fablehaven
$ ip addr show | grep 192.168.95.251
    inet 192.168.95.251/32 scope global eth0

Step 3 — Fix Kea to serve VIPs only​

Updated the Kea DHCP subnet configuration on nevermore to serve the VIPs as DNS servers, never the raw Pi-hole IPs:

{
  "option-data": [{ "name": "domain-name-servers", "data": "192.168.95.250, 192.168.95.251" }]
}

Applied to all subnets: LAN (95.0/24) and IoT (94.0/24).

Existing DHCP leases continued to use the old IPs until renewal. No active DNS outage occurred during the transition.

Result​

After remediation:

  • mistborn holds VIP .250 as preferred master
  • fablehaven holds VIP .251 as preferred master
  • hogwarts is warm standby for both VIPs
  • All DHCP clients receive .250 and .251 as DNS servers
  • Failover tested: stopping keepalived on mistborn causes hogwarts to claim .250 within 3 seconds

VRRP is now operating as designed for the first time since the cluster was built.

Security Implications​

Silent infrastructure failure is a security risk​

The cluster appeared healthy while being fundamentally broken. A monitoring system that checks only "is DNS resolving" would have reported green throughout. This is a reminder that infrastructure correctness and infrastructure availability are not the same thing.

DHCP serving raw IPs bypasses security controls​

If the Pi-hole cluster is the enforcement point for DNS filtering, serving raw IPs means a client that hard-codes one of those IPs continues to get filtering — but failover behavior is undefined. If a node went down during an active threat, filtering could have been bypassed for clients that cached the offline node's IP.

keepalived health checks were insufficient​

The check_pihole.sh script used by keepalived checked for the presence of the pihole-FTL process, not actual DNS resolution. A Pi-hole that was running but not answering DNS would not have triggered a failover. This is tracked as future hardening work.

Failure Modes​

What would have happened if mistborn went offline before remediation​

  • Clients with .3 as DNS1 would fail DNS resolution immediately
  • Clients with .6 as DNS2 would fall back to fablehaven — but only if their resolver tried the second server
  • No automatic failover would have occurred
  • Recovery required either mistborn coming back online or manual DHCP lease renewal across the fleet

What happens now if mistborn goes offline​

  • keepalived promotes hogwarts to own .250 within 3 seconds
  • Clients talking to .250 experience no interruption
  • mistborn coming back online triggers a preemptive takeover of .250

Impact to TheBurrow​

DNS is the most critical service on the network. Every device, every service, every Ansible run depends on it. A DNS outage would have:

  • Blocked all internet access for family devices
  • Broken Home Assistant automations
  • Disrupted ADS-B and 3D printer services that phone home
  • Potentially blocked Ansible from reaching fleet nodes

The fact that no outage occurred during the period when VRRP was broken was luck, not design. Failover was untested and would have failed.

Problem​

The Pi-hole VRRP cluster was silently misconfigured from initial deployment. keepalived was missing on mistborn, fablehaven held both VIPs incorrectly, and Kea was serving raw Pi-hole IPs to clients instead of the VIPs. DNS appeared healthy because the underlying nodes were up — but failover had never worked and clients were never using the VIPs.

Impact​

No active outage occurred. The impact was latent: if any Pi-hole node had gone offline, clients pointing at that node's raw IP would have lost DNS resolution with no automatic recovery. For a household where DNS failure means no internet, no Home Assistant, and no Ansible connectivity, this was a high-severity silent risk.

Lessons Learned​

Verify the thing you designed, not the thing you hope you built. DNS working is not the same as VRRP working. The correct test is to simulate node failure and confirm the VIP moves.

DHCP is the trust anchor for client DNS. If Kea serves raw IPs, VRRP is theater. The VIPs must be the only DNS addresses clients ever receive.

Incomplete role deployment is a silent risk. The keepalived role existed and worked — it just was not applied to all nodes. A canary check-mode run across the full inventory would have caught this immediately.

Monitor what matters, not what is convenient. Checking for a running process is not the same as checking that the service is functioning correctly. DNS resolution health checks should verify actual query responses, not process presence.

  • newcomb-labs/opnsense-lab #95 — keepalived missing on mistborn
  • newcomb-labs/opnsense-lab #96 — VRRP topology inversion
  • newcomb-labs/opnsense-lab #97 — Kea serving raw Pi-hole IPs
  • newcomb-labs/engineering-journal-kb #445 (this issue)

Related Content​