Skip to main content

Vault Split Committed Directly to Main — Governance Incident

Summary​

On June 14, 2026, during the restructure of the Ansible vault from a single monolithic vault.yml into per-service files, the changes were committed directly to the main branch of mistborn-ops/ansible.

This violated the core governance rule — all changes to main must go through a branch and PR — and bypassed every CI gate: lint, check-mode validation, idempotence checks, and secret scanning. The vault itself was not compromised and no secrets were leaked. This was a process failure, not a security breach.

Date: 2026-06-14 Severity: Medium — governance violation, no secrets compromise Resolution: mistborn-ops/ansible #817 (incident PR)

Situation​

TheBurrow's Ansible repo enforces a strict governance workflow:

  1. All changes go through a feature branch
  2. A PR is opened and CI must pass
  3. Squash merge only — no direct commits to main
  4. Signed commits required throughout

Branch protection on main enforces this at the GitHub level, requiring PRs and blocking direct pushes from non-admin users.

The vault restructure (#814) was a high-stakes change: splitting the monolithic group_vars/all/vault.yml into per-service files (vault_secrets_pihole.yml, vault_secrets_netbox.yml, etc.) with unique top-level keys per file to prevent dict clobber on auto-load.

Task​

Determine how a direct commit to main was possible given branch protection rules, assess the impact, and document the incident per the break-glass governance process.

Symptoms​

Post-commit, the git log showed the vault restructure commits landing directly on main with no associated PR. The commit history had no merge commit, no PR reference, and no CI run associated with the changes.

$ git log --oneline main | head -5
a1b2c3d chore(vault): split vault.yml into per-service files
e4f5g6h chore(vault): add unique top-level keys per vault file
...
## No merge commit, no PR number in commit message

Problem​

Branch protection on mistborn-ops/ansible requires PRs for all changes to main — but the repo owner (admin) can bypass branch protection rules. As the sole operator and repo admin, a direct push to main is technically possible even with protection rules enabled.

The commit was made during an extended session where the vault restructure was being worked through iteratively. The working pattern drifted from the governed workflow — changes were being committed locally and tested, and at some point a git push origin main was run instead of pushing to a feature branch.

The CI gates that would have caught any issues — vault syntax validation, check-mode playbook runs, idempotence checks — never ran.

Investigation​

How branch protection was bypassed​

GitHub branch protection rules have an exception: repository admins can bypass protection rules unless "Include administrators" is explicitly enabled. This option was not enabled on mistborn-ops/ansible.

The sole operator is the repo admin. A direct push succeeded silently.

What was in the commit​

The vault restructure commit contained:

  • Deletion of group_vars/all/vault.yml
  • Creation of per-service vault files under group_vars/all/
  • Updates to secrets_contract.yml, vault_policy.yml, and all role-level secrets adapters referencing the new paths

All of these were encrypted vault content — no plaintext secrets were exposed in the commit diff. The risk was not secrets exposure but rather untested structural changes landing on main without CI validation.

Impact on CI​

Because the commits landed directly on main without a PR, no CI workflows ran against the changes. The vault restructure was not validated by:

  • YAML lint (vault files are YAML)
  • Secret scanning (TruffleHog, detect-secrets)
  • Check-mode playbook runs
  • Vault path reference validation

The changes were functional — subsequent Ansible runs succeeded — but they were unvalidated at time of merge.

Root Cause​

Two factors:

  1. Admin bypass not restricted — "Include administrators" was not enabled in branch protection rules, allowing the repo owner to push directly to main.
  2. Session discipline failure — during an extended iterative session, the governed workflow was abandoned in favor of speed. The commit was made directly to main rather than to a feature branch.

Resolution​

Immediate — incident documentation​

Opened incident PR chore/incident-vault-split per the break-glass governance process. Created incident document at:

docs/incidents/2026-06-14_vault-split-direct-to-main.md

Incident document captured:

  • Summary of what happened
  • Impact assessment
  • Why the governed workflow was bypassed
  • Manual steps taken
  • Verification that the vault restructure was correct
  • Follow-up actions

Follow-up — enable admin restriction on branch protection​

Enabled "Include administrators" in branch protection rules on mistborn-ops/ansible to prevent future admin bypasses:

Settings → Branches → main → Require a pull request before merging
→ ☑ Include administrators

This means even the repo owner cannot push directly to main. Break-glass overrides require temporarily disabling the rule — which creates an audit trail in the GitHub audit log.

Result​

  • Incident documented per governance process
  • Branch protection hardened to include administrators
  • Vault restructure verified correct via subsequent Ansible check-mode runs across the full fleet
  • No secrets compromised, no operational impact

Impact​

No operational impact — the vault restructure was functionally correct and subsequent Ansible runs succeeded. The impact was governance:

  • CI gates were bypassed for a high-stakes change
  • No audit trail connecting the change to a PR, review, or CI run
  • Precedent risk: if direct-to-main is possible for admins, it may happen again under pressure

For a home lab with a sole operator, the real risk is future self: the next time a change feels urgent, the bypass path exists and has been used once.

Security Implications​

Admin bypass is a privileged escalation path​

The ability to push directly to main — even as the repo owner — is a privileged action that bypasses security controls. It should require the same friction as any other break-glass action: deliberate, audited, and documented.

Unvalidated vault changes carry structural risk​

The vault is the most sensitive file in the repo. Changes to vault structure — key names, file paths, per-service splits — affect every role and playbook that reads secrets. Bypassing CI for these changes means structural errors could silently break automation across the fleet, potentially in ways that only surface during incident response.

The audit trail gap​

A direct push to main leaves no PR, no CI run, no reviewer trail. If the vault restructure had introduced a subtle error — a clobbered key, a missing secret, an incorrect path — the audit trail would not have pointed to a PR or CI run. Debugging would have been harder.

Failure Modes​

What could have gone wrong​

  • A vault file with a duplicate top-level key would silently clobber another service's secrets — no error, wrong values injected at runtime
  • A missing vault file would cause variable undefined errors only when that service's playbook ran — potentially during an incident
  • A syntax error in a vault file would break all Ansible runs fleet-wide

None of these occurred. All were possible.

Lessons Learned​

"Include administrators" must always be enabled on protected branches. Branch protection without admin inclusion is a policy with a hole. For a sole-operator repo, this means protecting against future self under pressure. Enable it, document the break-glass override process, and enforce it.

High-stakes changes deserve extra process discipline, not less. Vault restructures, secrets migrations, and anything touching group_vars/all/ should go through the full governed workflow regardless of how well the operator understands the change. The value of the process is consistency, not just catching mistakes.

Speed is not an excuse for bypassing governance. The session was long, the change was iterative, and the push happened without thinking. This is exactly the failure mode governance workflows are designed to prevent. When fatigue or momentum pulls toward a shortcut, that is the moment to slow down, not speed up.

Incident documentation is a forcing function for learning. The act of writing this document — tracing exactly how it happened, what could have gone wrong, and what was changed — produces more durable learning than simply fixing the immediate problem.

  • mistborn-ops/ansible #817 — incident PR
  • mistborn-ops/ansible #814 — vault split implementation
  • mistborn-ops/ansible #443 — journal issue (this document)
  • newcomb-labs/engineering-journal-kb #444 — CI vault exposure (related)

Related Content​