Skip to main content

Step-CA JWK encryptedKey Corruption — Python Dict String in ca.json

Summary​

After deploying the Step-CA role to fablehaven, the step-ca systemd service failed to start. Investigation revealed that ca.json contained a malformed encryptedKey field — the JWK provisioner entry had been written as a Python dict string representation (single quotes, True instead of true) rather than valid JSON.

The corruption originated in vault_edit.py, which serialized the JWK dict using Python's str() instead of json.dumps(). The Ansible vault contained a Python repr string, which was templated directly into ca.json.j2 and written to disk.

Date: 2026-05-06 Severity: High — Step-CA non-functional, fleet cert issuance blocked Resolution: mistborn-ops/ansible #380, #467

Situation​

TheBurrow's internal PKI runs Step-CA on fablehaven as a systemd service. The CA uses a JWK (JSON Web Key) provisioner to issue certificates to fleet hosts. The JWK provisioner configuration — including the encryptedKey field containing the encrypted private key — is stored in Ansible Vault and templated into ca.json during deployment.

The vault_edit.py script was used to add the JWK provisioner entry to the vault. The script opened the vault, accepted the JWK value from the operator, and wrote it back.

Task​

Identify why Step-CA failed to start after deployment, restore the CA to a functional state, and prevent the corruption from recurring.

Symptoms​

## On fablehaven
$ systemctl status step-ca
● step-ca.service - Step-CA Certificate Authority
   Loaded: loaded (/etc/systemd/system/step-ca.service; enabled)
   Active: failed (Result: exit-code)

$ journalctl -u step-ca -n 20
error parsing ca.json: invalid character '\'' looking for beginning of value

The CA refused to start with a JSON parse error pointing to a single quote character — the tell-tale sign of a Python repr string where JSON was expected.

Problem​

The encryptedKey field in a JWK provisioner must be a valid JSON string. A correctly formed entry looks like:

{
  "type": "JWK",
  "name": "fablehaven-provisioner",
  "key": {
    "use": "sig",
    "kty": "EC",
    "kid": "...",
    "crv": "P-256",
    "alg": "ES256",
    "x": "...",
    "y": "..."
  },
  "encryptedKey": "{\"kty\":\"EC\",\"use\":\"sig\",...}"
}

What was written to ca.json instead:

"encryptedKey": "{'kty': 'EC', 'use': 'sig', 'kid': '...', 'x': '...', 'y': '...', 'crv': 'P-256', 'alg': 'ES256', 'd': '...', 'key_ops': ['sign'], 'ext': True}"

Single quotes, True capitalized, colon-separated key-value pairs — this is Python's repr() output for a dict, not JSON.

Investigation​

Tracing the corruption path​

The JWK value flowed through three stages:

  1. vault_edit.py — operator pastes JWK value, script writes to vault
  2. Ansible Vault — encrypted storage
  3. ca.json.j2 template — vault variable rendered into ca.json

Inspecting the vault directly (after decryption) revealed the encryptedKey value was stored as a Python dict string:

vault_step_ca_jwk_encrypted_key: "{'kty': 'EC', 'use': 'sig', ...}"

The template ca.json.j2 rendered this value directly:

"encryptedKey": "{{ vault_step_ca_jwk_encrypted_key }}"

No JSON serialization was applied. The Python repr string landed verbatim in ca.json.

Root cause in vault_edit.py​

The script used Python's str() to serialize the value before writing to the vault:

## Incorrect
vault_data[key] = str(user_input)

## Correct
vault_data[key] = json.dumps(user_input) if isinstance(user_input, dict) else user_input

When the operator pasted a JSON object, vault_edit.py parsed it with json.loads() (producing a Python dict) and then serialized it back with str() — producing a Python repr string instead of a JSON string.

Why it wasn't caught earlier​

  • The vault contains encrypted content — syntax errors are invisible until decryption
  • The template rendered the value without validation
  • No preflight assertion checked that the JWK value was valid JSON before rendering ca.json
  • The step-ca service only validates ca.json at startup — not during Ansible deployment

Root Cause​

vault_edit.py used str() instead of json.dumps() when serializing dict values back to the vault, corrupting JSON content to Python repr format. The corruption was invisible until Step-CA attempted to parse ca.json at service start.

Resolution​

Step 1 — Repair the vault entry​

Manually decrypted the vault, corrected the encryptedKey value to valid JSON (re-serialized with json.dumps()), and re-encrypted:

ansible-vault edit inventories/home/group_vars/all/vault_secrets_step_ca.yml
## Corrected encryptedKey to valid JSON string

Step 2 — Re-run the Step-CA role​

Ran the step-ca Ansible role against fablehaven to regenerate ca.json from the corrected vault value. Verified service started cleanly:

$ systemctl status step-ca
● step-ca.service - Step-CA Certificate Authority
   Active: active (running)

Step 3 — Fix vault_edit.py serialization​

Updated vault_edit.py to use json.dumps() for dict values:

## After fix
if isinstance(value, dict):
    vault_data[key] = json.dumps(value)
else:
    vault_data[key] = value

Step 4 — Add preflight assertion in Step-CA role​

Added a preflight task to the step-ca role to validate the JWK value is valid JSON before attempting to render ca.json:

- name: Assert JWK encryptedKey is valid JSON
  ansible.builtin.assert:
    that:
      - vault_step_ca_jwk_encrypted_key | from_json is mapping
    fail_msg: "vault_step_ca_jwk_encrypted_key is not valid JSON — check vault_edit.py output"

Step 5 — Automate corruption repair​

Created an automation task in the step-ca role to detect and repair encryptedKey corruption without manual vault editing (ansible #467).

Result​

  • Step-CA service running on fablehaven
  • ca.json contains valid JSON throughout
  • vault_edit.py correctly serializes dict values as JSON strings
  • Preflight assertion catches corruption before service deployment
  • Fleet cert issuance restored for Pi-hole nodes, Caddy, and devpi

Impact​

During the period Step-CA was non-functional:

  • No new TLS certificates could be issued to fleet hosts
  • Certificate renewal via the step-ca-client role was blocked
  • Existing certificates continued to function (not yet expired)
  • Caddy reverse proxy on fablehaven continued serving existing certs

The impact was bounded because the failure occurred immediately after initial CA deployment — no existing certs had been issued that needed renewal. If this had occurred mid-fleet with active certificates approaching expiry, the impact would have been more severe.

Security Implications​

Corrupt CA configuration is a silent failure mode​

A CA that fails to start is visible. A CA that starts but silently uses wrong provisioner configuration is not. The preflight assertion added to the role ensures structural correctness is validated before deployment, not discovered at runtime.

Vault content is opaque until use​

Encrypted vault content cannot be validated without decryption. Errors introduced at write time — wrong format, wrong type, wrong encoding — are invisible until the value is consumed. This argues for validation at both write time (vault_edit.py) and consume time (preflight assertions in roles).

JWK material is high-value​

The JWK provisioner private key is the credential that authorizes certificate issuance for the entire fleet. Corruption, exposure, or substitution of this value has PKI-wide impact. Extra validation and careful handling are warranted.

Failure Modes​

What would have happened if existing certs had been near expiry​

If fleet certificates had been within the renewal window when the CA went non-functional, automatic renewal via the step-ca-client systemd timer would have failed silently. Certificates would have expired, causing TLS errors across internal services — Caddy reverse proxy, Pi-hole admin UI, devpi, NetBox — until the CA was restored.

What would have happened if corruption went undetected​

If the CA had somehow started with a malformed ca.json (e.g., in a future version of step-ca that handled the parse error differently), it might have operated in a degraded state — accepting requests but using incorrect provisioner configuration. Certificates issued with wrong provisioner metadata would have been structurally valid but governance-incorrect.

Lessons Learned​

Validate at write time, not just at read time. The corruption was introduced at vault write time and only discovered at service start. A validation step in vault_edit.py that checks JSON validity before writing would have caught this immediately.

Preflight assertions are the last line of defense before deployment. The step-ca role now asserts that the JWK value is valid JSON before rendering ca.json. This costs one task and prevents a class of deployment failures.

Python repr is not JSON. str(dict) and json.dumps(dict) produce different output. Single quotes vs. double quotes, True vs. true, None vs. null. These are not interchangeable. Any code that serializes Python objects for use in JSON contexts must use json.dumps().

High-value cryptographic material deserves explicit format contracts. The vault schema for JWK entries should document that the value must be a JSON string, not a Python object. Making the contract explicit reduces the chance of future tooling errors introducing the same corruption.

  • mistborn-ops/ansible #380 — initial vault repair
  • mistborn-ops/ansible #467 — automated corruption repair
  • mistborn-ops/ansible #318 — safe JWK vault variable handling
  • mistborn-ops/ansible #367 — JWK provisioner bootstrap ADR
  • newcomb-labs/engineering-journal-kb #448 — ADR: JWK provisioner bootstrap strategy
  • newcomb-labs/engineering-journal-kb #467 (this issue)

Related Content​