Skip to main content

Incident Response Runbook

Purpose​

This runbook provides specific response procedures for security incidents affecting this repository and its infrastructure. Each playbook is tailored to the actual architecture (GitHub, Cloudflare Pages, static Docusaurus site).

Playbook 1: GitHub Token Compromised​

Detection​

Canary token alert, unexpected repository activity in audit log monitor, or notification from GitHub.

Response​

  1. Go to GitHub Settings, Developer settings, Personal access tokens
  2. Revoke the compromised token immediately
  3. Check recent repository activity: gh api repos/newcomb-labs/engineering-journal-kb/events --jq '.[].type'
  4. Review commits for unauthorized changes: git log --oneline --since="24 hours ago"
  5. Check for new collaborators, deploy keys, or webhooks
  6. Generate a new token with minimum required scopes
  7. Update GitHub Actions secrets if the token was used in CI
  8. Update the Secret Rotation Schedule

Recovery time: minutes​

Playbook 2: Dependency Backdoored​

Detection​

Dependabot alert, dependency-audit workflow finding, community advisory, or malware pattern scanner flag.

Response​

  1. Identify the affected package and version
  2. Check if the package is a direct or transitive dependency: cd website && npm ls <package>
  3. If direct: pin to the last known good version in package.json
  4. If transitive: add an npm override to force a safe version
  5. Run npm ci and npm run build to verify the site builds cleanly
  6. Check the deployed site for any signs of compromise
  7. Open a PR with the fix, referencing the advisory
  8. Run the malware pattern scanner: python3 scripts/scan_malware_patterns.py --path website/

Recovery time: hours​

Playbook 3: Unauthorized Repository Access​

Detection​

Audit log monitor workflow creates an issue, or manual observation of unexpected collaborators/deploy keys.

Response​

  1. Remove unauthorized collaborators: gh api repos/newcomb-labs/engineering-journal-kb/collaborators/<username> -X DELETE
  2. Remove unauthorized deploy keys via GitHub UI (Settings, Deploy keys)
  3. Remove unauthorized webhooks via GitHub UI (Settings, Webhooks)
  4. Review branch protection rules: run gh workflow run branch-protection-audit.yml
  5. Check recent commits and PRs for unauthorized changes
  6. Rotate all secrets (follow Secret Rotation Schedule)
  7. Enable or verify 2FA on the GitHub account

Recovery time: minutes to hours​

Playbook 4: Cloudflare Account Compromised​

Detection​

Unexpected DNS changes, site downtime, certificate warnings, or Cloudflare notification.

Response​

  1. Log into Cloudflare and change the account password immediately
  2. Verify 2FA is enabled
  3. Check DNS records for unauthorized changes
  4. Check WAF rules for tampering
  5. Check Pages deployments for unauthorized builds
  6. Revoke and regenerate all Cloudflare API tokens
  7. Verify SSL/TLS settings (Full Strict, minimum TLS 1.2)
  8. Contact Cloudflare support if access is lost
  9. Update the Secret Rotation Schedule

Recovery time: hours (DNS propagation may take up to 48 hours)​

Playbook 5: Secret Leaked in Commit​

Detection​

Gitleaks CI alert, commit message scanner, or manual discovery.

Response​

  1. Do NOT try to remove the commit — it is already in git history
  2. Immediately revoke the leaked credential in the relevant service
  3. Generate a new credential
  4. Update the GitHub Actions secret or environment variable
  5. Verify all workflows pass with the new credential
  6. Consider using git filter-repo or BFG Repo Cleaner to remove from history (optional, only if the repo is not widely cloned)
  7. Update the Secret Rotation Schedule
  8. Add the pattern to Gitleaks allowlist if it was a canary token

Recovery time: minutes (credential rotation), hours (history cleanup if needed)​

Playbook 6: Build Artifact Tampered​

Detection​

Content integrity checksums mismatch, unexpected content on the deployed site, or user report.

Response​

  1. Trigger a fresh deployment: gh workflow run deploy-pages.yml
  2. Verify the build output matches expectations
  3. Check the Cloudflare Pages deployment log for the last deployment
  4. If the tampering occurred at the CDN level, purge Cloudflare cache
  5. Review CI workflow logs for the build that produced the tampered artifact
  6. Check for unauthorized changes to build scripts or docusaurus.config.js

Recovery time: minutes (redeploy)​

General Principles​

  • Revoke first, investigate second
  • Document every action taken during the incident
  • Notify affected parties if applicable
  • Update this runbook with lessons learned after every incident
  • Follow the Vulnerability Disclosure Process for external reporting

Related Content​