Operational Runbooks & Day-2 Support Centre
Internal workspace for BLACKSWAN's day-2 operating procedures — incident runbooks, recurring checks, client/regulator/board prep routines, evidence refresh, release support, and post-launch handover. Every runbook references its primary policy/control IDs from the Policy & Control Library and ties incident hooks into Production Monitoring, Regulatory Notification, and Stakeholder Rooms. Conservative posture: internal readiness only — not legal advice, not regulatory approval, not certification, not audit opinion, not regulator submission, not client acceptance, not authorization for external launch.
Twelve runbooks · owner · linked policy/control · cadence · state
| ID | Runbook | Owner | Linked policy / control | Cadence | Last exercised | State |
|---|---|---|---|---|---|---|
| RB-T1-MASTER | Tier-1 incident master | CISO · SRE Lead | POL-012 · CTL-INCIDENT · CTL-WAF · CTL-SIEM | Quarterly tabletop | 2026-05-09 | Live |
| RB-RC-ROLLBACK | Release rollback | SRE Lead · Programme | POL-006 · CTL-RELEASE | Quarterly drill | 2026-05-08 | Live |
| RB-DR-FAILOVER | DR / regional failover | SRE Lead · CISO | POL-012 · CTL-008 RTO/RPO | Quarterly | 2026-03-12 · next 2026-06-12 | Drill Scheduled |
| RB-DG-INCIDENT | Data incident / privacy | DG Lead · CISO · Counsel | POL-003 · CTL-DSR · CTL-RETENTION-HOLD | Quarterly tabletop | 2026-05-04 | Live |
| RB-REG-NOTIFY | Regulator notification | Head of Regulatory · External Counsel | POL-014 · POL-012 · CTL-COUNSEL-COUNTERSIGN | Quarterly tabletop | 2026-05-09 | Counsel Review |
| RB-ROOM-SHARE | Stakeholder room per-share | Head of Stakeholder Rooms · CISO | POL-003 · CTL-ROOM-ACCESS · CTL-LINK-SIGNING | Per-share | Continuous | Live |
| RB-EVIDENCE-PUBLISH | Evidence pack publish | Head of Evidence | POL-013 · CTL-DUAL-SIGNOFF · CTL-FOOTER | Weekly | 2026-05-15 | Live |
| RB-VENDOR-ONBOARD | Vendor onboarding & DD | Head of Procurement · CISO | POL-007 · POL-016 · CTL-VENDOR-CLASS · CTL-DPA-SCC | Per-vendor + monthly | 2026-04-22 | Live |
| RB-FINANCIAL-CLOSE | Monthly financial close | CFO | POL-008 · POL-009 · POL-017 · CTL-RECON | Monthly | 2026-04-30 | Live |
| RB-MODEL-OVERRIDE | Model output override | CISO · Head of Regulatory | POL-010 · POL-011 · CTL-MODEL-OVERRIDE · CTL-HUMAN-IN-LOOP | On override + monthly | 2026-05-12 | Live |
| RB-KYC-FLOW | Client KYC / KYB onboarding | Operations · Compliance · External Counsel | POL-002 · CTL-005 | Per-client + monthly | 2026-04-12 | Stale |
| RB-PILOT-TEARDOWN | Pilot tenant tear-down | Operations · Tech/Security | POL-018 · CTL-003 · CTL-006 | Per pilot | n/a · pending first pilot | Draft |
Ten recurring checks · cadence · owner · evidence captured
| Check | Cadence | Owner | Inputs | Output / evidence | State |
|---|---|---|---|---|---|
| Daily readiness scan | Daily | Programme · SRE Lead | Production posture · open incidents · gate movement | Daily note + attention list | Live |
| Weekly launch review | Weekly | Programme · CISO · CFO | Readiness scan · gate status · evidence packs | Readiness sign-off · weekly status | Live |
| Monthly evidence spine briefing | Monthly | Head of Evidence | 21 evidence packs · cross-Centre updates | Refreshed evidence pack scorecard | Live |
| Monthly board pack issue | Monthly | CoS · CEO · Board Chair | Board pack draft · decision requests · open exceptions | Board pack · attestations · decision record | Live |
| Incident review windows | Per Sev-1/2 + monthly aggregate | CISO · SRE Lead | Incident timeline · root cause · remediation | Post-incident review · timeline log | Live |
| Quarterly control testing | Quarterly | CISO · Risk Committee | Control library · test cadence · evidence source | Control testing evidence pack | Live |
| Quarterly rollback & DR drill | Quarterly | SRE Lead · CISO | RTO/RPO targets · runbook · failover route | Drill report · RTO/RPO actuals · evidence pack | Drill Scheduled 2026-06-12 |
| Quarterly break-glass drill | Quarterly | Tech/Security · COO | Sealed credentials · dual approver · auto-rotate flow | Drill evidence · revocation log | Drill Overdue |
| Annual policy attestation | Annual | CISO · DG Lead · Head of Regulatory | Policy library state · role attestation register | Attestation log · policy refresh | Live |
| Annual board effectiveness review | Annual | Board Chair | NED rotation · committee charter · attendance | Board effectiveness note | Live |
Four severities · paging SLA · response SLA · resolution SLA · escalation
| Severity | Definition | Paging SLA | Response SLA | Target resolution | Escalation path | Communications |
|---|---|---|---|---|---|---|
| Sev-1 | Tier-1 service down · MNPI leak · regulator-notify trigger · client-impact widespread | ≤ 5 m | ≤ 15 m | RTO 30 m · MTTR 4 h | SRE → CISO → CEO → Board · Counsel for regulator-notify | Internal war-room · counsel-locked external comms only |
| Sev-2 | Tier-1 degraded · Tier-2 down · stakeholder room anomaly · vendor outage | ≤ 15 m | ≤ 30 m | MTTR 8 h | SRE → CISO → COO | Internal · client status page (counsel-approved) |
| Sev-3 | Single-tenant degraded · isolated defect · minor data quality | ≤ 30 m | ≤ 4 h | MTTR 5 business days | SRE → CISO | Internal |
| Sev-4 | Cosmetic · low-severity defect · documentation gap | n/a | ≤ 1 business day | Next release | SRE Lead | Internal |
Six stages · paging → containment → counsel-notify → recovery → review
Signal / triage
Monitoring/SIEM signal · client report · counsel-flagged event; SRE on-call triages severity.
Page on severity
Sev-1 ≤ 5m · Sev-2 ≤ 15m. War-room opened; incident commander named.
Containment
Isolate · revoke · rate-limit · freeze rooms. Evidence preserved before any rollback.
Notify (counsel-bound)
Regulator-notify trigger evaluated by counsel; board paged per threshold; counterparty rooms locked.
Recovery
Rollback or failover via RB-RC-ROLLBACK / RB-DR-FAILOVER; service restored within RTO.
Post-incident
Post-mortem · controls tightened · audit-event family preserved · evidence pack diff committed.
Ten per-incident / per-cycle artefacts · owner · audience · counsel countersign
| Artefact | Owner | Audience | Counsel countersign | Retention | Linked centre(s) |
|---|---|---|---|---|---|
| Incident timeline | SRE Lead · CISO | Internal · Regulator-review (counsel-bounded) | For regulator-notify | 10 y | Incident · Production Monitoring · Reg. Escalation |
| Post-mortem | SRE Lead | Internal | Where regulator-facing | 10 y | Incident · Release Control · Strategic Reporting |
| Rollback evidence | SRE Lead | Internal · Regulator-review (OpRes) | For OpRes filing | 10 y | Release Control · Testing/QA |
| DR drill report | SRE Lead · CISO | Internal · Regulator-review (OpRes) | For OpRes filing | 10 y | Production Monitoring · Release Control |
| Evidence-publish log | Head of Evidence | Internal · Auditor | For external read | 10 y | All 21 evidence packs · Strategic Reporting |
| Stakeholder room access ledger | Stakeholder Rooms · CISO | Internal · Auditor · Regulator-review | Mandatory for external | 10 y | Stakeholder Rooms · Data-Room MNPI |
| Regulator-notify decision | Head of Regulatory · External Counsel | Internal · Counsel · Regulator (per engagement) | Mandatory | 10 y | Regulatory Escalation · Jurisdiction Playbooks |
| Production handover checklist | Programme · SRE Lead | Internal | n/a | 10 y | Release Control · Testing/QA · Programme Governance |
| Break-glass drill evidence | Tech/Security · COO | Internal · Auditor | For audit binder | 10 y | Security Operations · User Role/Permission |
| External communications record | CoS · External Counsel | Internal · Audience | Mandatory for external | 10 y | Stakeholder Rooms · Strategic Reporting |
Ten role experiences of day-2 operations
Full operations board
Sees the full runbook ledger + recurring calendar; dual-signs Tier-1 incident go/no-go.
Operator board
Owns daily/weekly/monthly cycles; runs KYC, vendor, financial-close, evidence-publish runbooks.
Incident commander view
On-call rota; owns RB-T1-MASTER, RB-RC-ROLLBACK, RB-DR-FAILOVER, RB-MODEL-OVERRIDE. JIT for break-glass.
Counsel-bound view
Owns RB-REG-NOTIFY; counsel-countersigns regulator-facing comms (CTL-010).
Evidence refresh view
Drives weekly RB-EVIDENCE-PUBLISH; per-file footer enforced; hash + freshness captured per pack.
Per-pilot support view
Sees only assigned pilot tenant's queue · KYC flow status · pilot tear-down checklist; never cross-tenant.
Board-only view
Reads monthly board pack section after 2 stable cycles; sees post-incident review summaries; never raw logs.
Counsel-locked view
Read-only on counsel-locked incident timelines, DR drill reports, OpRes evidence; per-engagement TTL.
Engagement-scope view
Reads engagement-scope evidence (post-mortem · attestation · drill reports); engagement letter required.
Partner integration view
Vendor risk monitoring · DPA/SCC posture · exit-plan readiness; counsel-bounded for material outsourcing.
Workflow · accountable · approver · SOD pair · linked policy/control
| Workflow | Accountable | Approver / escalation | SOD pair | Linked policy / control |
|---|---|---|---|---|
| Incident paging (Sev-1) | SRE on-call | SRE → CISO → CEO | Triage ≠ Approver | POL-012 · CTL-INCIDENT |
| Regulator-notify decision | Head of Regulatory | External Counsel + CEO | Operator ≠ Submitter | POL-014 · CTL-010 |
| Board paging | CEO · CoS | Board Chair | n/a | Strategic Reporting workflow |
| Tier-1 rollback authority | SRE Lead | SRE + CISO (dual) | Single-party ≠ Rollback on Tier-1 | POL-006 · CTL-004 |
| Stakeholder room freeze | Head of Stakeholder Rooms | CISO + Compliance | Owner ≠ Approver on MNPI rooms | POL-003 · CTL-003 |
| Break-glass elevation | CISO · COO (dual) | Board (post-fact attestation) | Sealed · auto-rotate after use | POL-018 · CTL-001 |
| Counterparty engagement pause | Head of Commercial | Compliance + External Counsel | Operator ≠ Approver | POL-015 · POL-019 |
| External communications | CoS | External Counsel + CEO | Drafter ≠ Counsel | POL-014 · CTL-010 |
From release sign-off → operational stewardship
| Item | Owner | Evidence required | State |
|---|---|---|---|
| Runbook(s) updated for release scope | SRE Lead | Runbook diff hash + reviewer note | Met |
| Monitoring alerts & SLO baselines set | SRE Lead · CISO | SIEM index · alert config diff | Met |
| On-call rota confirmed for window | SRE Lead | Rota record · paging test | Met |
| Rollback rehearsal proven | SRE Lead | RB-RC-ROLLBACK evidence | Met |
| Comms templates counsel-approved | CoS · External Counsel | Template hash · counsel countersign | Counsel Pending |
| Vendor / outsourcing review current | Procurement · Risk Committee | Vendor inventory · concentration · DPA/SCC | CHG-006 Pending |
| Evidence packs fresh (≤ 30 d) | Head of Evidence | Per-pack hash + freshness date | Met |
| Day-2 baseline runbook live | Programme · SRE Lead | RB-T1-MASTER · RB-DG-INCIDENT · RB-REG-NOTIFY | Met |
| Policy attestation updated | CISO · DG Lead | POL-018 attestation register | Met |
| Handover sign-off (dual) | SRE Lead · Programme Manager + CISO | Handover attestation hash | In Review |
Open exceptions · owner · remediation
| Alert ID | Item | Issue | Severity | Owner | Remediation | State |
|---|---|---|---|---|---|---|
| AL-ORBS-001 | RB-KYC-FLOW | Stale (last 2026-04-12) · KYC partner contract pending | Sev-2 | Operations · Compliance · External Counsel | Refresh post KYC contract close (DF-005) | Stale |
| AL-ORBS-002 | RB-PILOT-TEARDOWN | Draft only · awaiting first pilot | Sev-3 | Operations · Tech/Security | Lock content before first pilot launch | Draft |
| AL-ORBS-003 | DR drill 2026-06-12 | Pre-drill posture; evidence pending | Sev-2 | SRE Lead · CISO | Run drill · capture RTO/RPO evidence | Scheduled |
| AL-ORBS-004 | Quarterly break-glass drill | Overdue (target 2026-05-12) | Sev-2 | Tech/Security · COO | Schedule drill · capture evidence · revoke | Overdue |
| AL-ORBS-005 | RB-REG-NOTIFY phrasing | Counsel countersign pending (2026-05-19) | Sev-1 | Head of Regulatory · External Counsel | Lock phrasing after counsel session | Counsel Pending |
| AL-ORBS-006 | External comms templates | Counsel sign-off pending | Sev-2 | CoS · External Counsel | Lock templates after counsel session | Counsel Pending |
| AL-ORBS-007 | Handover sign-off | Programme + CISO dual sign-off pending | Sev-2 | Programme · CISO | Complete handover attestation | In Review |
Four gates · all green before any external operational comms leaves internal scope
Counsel-locked phrasing
External counsel countersigns every external comm (regulator-notify, client status, board paging language) per CTL-010.
Evidence attached
Per-pack evidence hashes + freshness · timeline · counsel countersign attached to comm.
Audience-bound
Recipient role + classification explicitly permitted per User Role & Client Lifecycle matrices; per-recipient signing.
Revocation readiness
Single-action revocation pulls comm distribution + per-recipient keys; revocation runbook proven.
Day-2 custody · rotation cadence · recovery drill posture
/api/secret-rotation-key-custody, in the fixture, or in any commit. Day-2 operations track only ownership, custodian, custody model, declared (non-secret) reference NAMES, presence flags, approval state, rotation cadence, last-rotation evidence date, next rotation due, recovery drill status / date, evidence references, and unlock criteria. Staging or demo credentials do not count as production secret custody evidence. Internal key-custody readiness posture only — not security certification, not regulatory approval, not legal advice, not audit opinion, not compliance certification, not regulator submission, not permission grant, not launch authorisation, not external endpoint authorisation, and not external-use authorisation.
- Per-rotation evidence (last-rotation date + next rotation due) captured on every required item: session signing, JWT/OIDC, Entra app credential, evidence export storage, SIEM forwarder token, monitoring webhook, TLS private key, mTLS partner key, backup vault key, data-store key, break-glass credential, CI/CD deploy token, regulatory data API key.
- Quarterly restore drills (backup vault, evidence export storage), annual revocation drills (TLS / mTLS), per-rotation recovery drills (session / JWT / OIDC / CI/CD / regulatory data API), and quarterly retrieval drills (break-glass) — all logged with evidence references and reviewed at day-2 cadence.
- Staging founder MFA factor is quarantined — never used as evidence of production secret custody.
- Any missing, in-review, blocked, rotation-overdue, or recovery-untested required item keeps production launch at HOLD · NO-GO and blocks external-use bundle release.
- Authoritative row table is rendered in the Security Operations · IAM · Zero-Trust Centre and the Final Production Launch Control Tower. Read-only fixture exposed via
/api/secret-rotation-key-custody; cross-references/api/production-config-readiness,/api/production-ingress-route-readiness,/api/entra-oidc-readiness, and/api/auth/posture.
Backup scope · restore drills · RPO/RTO · retention · DR · recovery authority
/api/backup-restore-recovery-evidence, in the fixture, or in any commit. Day-2 operations track only ownership, custodian, custody model, declared (non-secret) reference NAMES, presence flags, approval state, backup cadence, last-backup evidence date, last restore drill date, RPO/RTO targets, measured RPO/RTO where captured, retention / legal-hold posture, evidence references, and unlock criteria. Staging or demo backups do not count as production recovery evidence. Internal backup/recovery readiness posture only — not security certification, not regulatory approval, not legal advice, not audit opinion, not compliance certification, not regulator submission, not permission grant, not launch authorisation, not external endpoint authorisation, and not external-use authorisation.
- Production launch requires: backup scope inventory, encryption / custody, successful restore drill, RPO/RTO evidence, retention / legal-hold alignment, monitoring & alerting, and recovery authority counter-sign. Staging / demo backups do not count.
- Per-data-class restore drills (database, PITR, evidence export, audit log) plus annual DR / region-failover exercise — all logged with evidence references and reviewed at day-2 cadence.
- Backup encryption key custody links to Secret Rotation, Key Custody & Recovery Drill Evidence Loop; staging founder MFA factor and staging keys are quarantined.
- Any missing, in-review, blocked, recovery-untested, RPO/RTO-unverified, retention-unverified, or approval-pending required item keeps production launch at HOLD · NO-GO and blocks external-use bundle release.
- Authoritative row table is rendered in the Data Governance & Retention Centre, the Production Monitoring Centre, the Security Operations · IAM · Zero-Trust Centre, the Final Production Launch Control Tower, and the Completeness Command Centre. Read-only fixture exposed via
/api/backup-restore-recovery-evidence; cross-references/api/production-config-readiness,/api/production-ingress-route-readiness,/api/secret-rotation-key-custody, and/api/jurisdictional-permissions-matrix.
Health checks · SLOs · alert routing · SIEM · incident command · PIR · escalation
/api/observability-slo-incident-evidence, in the fixture, or in any commit. Day-2 operations track only ownership, declared (non-secret) reference NAMES, presence flags, approval state, SLO targets, measured values where captured, alert route names, last alert-test dates, last incident-drill dates, log retention evidence references, linked runbook / escalation paths, notification trigger statuses, evidence references, and unlock criteria. Staging or demo monitoring does not count as production observability evidence. Internal observability/incident readiness posture only — not security certification, not regulatory approval, not legal advice, not audit opinion, not compliance certification, not regulator submission, not permission grant, not launch authorisation, not external endpoint authorisation, not incident notification submission, and not external-use authorisation.
- Production launch requires: health checks & synthetic probes, uptime/latency/error-rate SLOs with measured evidence, tested alert routing & on-call coverage, SIEM/audit log forwarding with retention evidence, incident command room & per-severity runbook, escalation SLA, regulator/board/stakeholder notification trigger matrix, post-incident review evidence, and a captured incident authority counter-sign. Staging / demo monitoring does not count.
- Per-dependency monitors (Entra OIDC, database, partner routes) cross-link to Secret Rotation, Key Custody & Recovery Drill Evidence Loop, Production Backup, Restore & Data Recovery Evidence Centre, and the Entra OIDC readiness layer. Staging founder MFA factor and any staging probe outputs are quarantined.
- Notification trigger matrix and customer/stakeholder comms templates remain review-only on this platform: nothing here auto-files a supervisor notification or auto-sends a message.
- Any missing, in-review, blocked, slo-unverified, alert-route-untested, logging-unverified, pir-untested, or approval-pending required item keeps production launch at HOLD · NO-GO and blocks external-use bundle release.
- Authoritative row table is rendered in the Security Operations · IAM · Zero-Trust Centre, the Production Monitoring & Incident Command Centre, the Final Production Launch Control Tower, the Regulatory Notification & Board Escalation Centre, and the Completeness Command Centre. Read-only fixture exposed via
/api/observability-slo-incident-evidence; cross-references/api/production-config-readiness,/api/production-ingress-route-readiness,/api/secret-rotation-key-custody,/api/backup-restore-recovery-evidence,/api/entra-oidc-readiness, and/api/jurisdictional-permissions-matrix.
Audit log of day-2 operational events
| Timestamp (UTC) | Actor | Event family | Action | Evidence hash | Limitation recorded | Next step |
|---|---|---|---|---|---|---|
| 2026-05-08 09:00 | SRE Lead · Programme | Runbook inventory | ORBS-RB — 12 runbooks catalogued | sha256:orbs…aa01 | "Internal readiness only." | Quarterly review |
| 2026-05-09 11:14 | Programme · CoS | Operational calendar | ORBS-OC — 10 recurring checks | sha256:orbs…aa02 | "Internal." | Refresh on cadence change |
| 2026-05-10 09:22 | CISO · SRE Lead | Severity matrix | ORBS-SV — 4 severities & SLAs published | sha256:orbs…aa03 | "Internal." | Re-test on SLA change |
| 2026-05-11 14:08 | SRE Lead · CISO · Compliance | Incident workflow | ORBS-IW — 6-stage flow | sha256:orbs…aa04 | "Internal." | Quarterly tabletop |
| 2026-05-12 09:50 | Programme · Head of Evidence | Support evidence register | ORBS-SE — 10 artefacts catalogued | sha256:orbs…aa05 | "Internal." | Refresh per incident |
| 2026-05-13 11:42 | CoS · CISO · Compliance | Stakeholder views | ORBS-SV — 10 views scoped | sha256:orbs…aa06 | "Counsel-bounded for external." | Lock per-share counsel sign-off |
| 2026-05-14 08:18 | Programme · CISO · External Counsel | Escalation authorities | ORBS-EA — 8 workflows scoped | sha256:orbs…aa07 | "Internal." | Re-test on workflow change |
| 2026-05-15 09:00 | Programme · SRE Lead | Handover checklist | ORBS-HC — 10 items tracked | sha256:orbs…aa08 | "Internal." | Close dual sign-off |
| 2026-05-15 11:30 | External Counsel · CEO · CoS | External comms gates | ORBS-VG — 3 of 4 gates met | sha256:orbs…aa09 | "No external comm without all 4 gates." | Close Gate 1 post 2026-05-19 |
| 2026-05-16 07:55 | CEO · CISO · Compliance | Centre attestation | ORBS-ATTESTATION — monthly attestation | sha256:orbs…aa10 | "Internal; not regulatory approval." | Re-attest monthly |
What this Centre is — and is not
- Staging / simulated day-2 data. All runbook IDs, drill records, evidence hashes, dates, and audit events shown here are seed values for an internal readiness workflow. They are not a live ITSM, not a live paging system, and not a live audit log.
- Internal readiness workflow only. This Centre captures BLACKSWAN's internal day-2 operating posture. It is not legal advice, not regulatory approval, not certification, not an audit opinion, not regulator submission, not client acceptance, and not authorization for external launch.
- Counsel-bound for external surfaces. Every regulator- or client-facing communications artefact requires external counsel countersign (CTL-010) before any external comms gate is opened.
- P0 blocks comms; P1 requires dual sign-off. No external operational comm ships under P0; P1 comms require Programme + CISO dual sign-off and counsel countersign where regulator-facing.
- Founder-root is a standing accepted risk. Per Programme Decision Log DEC-005, founder-root permanence carries MFA + re-auth + monthly attestation as compensating controls.