Skip to content
UniDPP Docs

Operations: incident response

Incident response

Section titled “Incident response”

Every scenario below ends in a state you can verify, which is the design constraint: journals are append-only, backups are checksummed, and drills prove the restore path before you need it.

A public hostname answers 502

Section titled “A public hostname answers 502”

The tunnel is alive; the origin service is down.

Terminal window
./ops/target/release/unidpp-stack status # confirms: which origin, which tunnel
./ops/target/release/unidpp-stack up      # the idempotent repair: only the dead restart

If the service refuses to start, read its log under run/<name>.log. A journal that fails to replay is a restore situation, not a delete-the-journal situation, because the journal is the record.

A port is held by the wrong listener

Section titled “A port is held by the wrong listener”

status names the holder (the service identity check). Kill by the exact PID (run/*.pid for the stack’s own, lsof -nP -iTCP:<port> for a foreign one) and never by process name.

A verification unexpectedly degrades

Section titled “A verification unexpectedly degrades”

Run the verdict with the same --as-of: the three readings (cryptographic, evidentiary, current-state) name the check that degraded. Trust-side questions (anchor rotation, revocations) answer from GET /revocations on the trust service, including the retroactivity semantics and the evidentiary cutoff that protects pre-declaration verifiers.

The deployment itself is lost

Section titled “The deployment itself is lost”

The disk, the host, the room. Recovery is recover: stop everything, verify + unpack the newest backup at its original shape, restart, status. The proof you can do this under pressure is drill --recover; run it in your routine, not your incident.

Suspicion of state corruption

Section titled “Suspicion of state corruption”
  1. ./unidpp-ops verify <latest-archive>: is the last good backup intact?
  2. ./unidpp-ops drill: does restore still work end to end?
  3. Only then: ./unidpp-ops restore <archive> <tenant> --force (the current directory is renamed aside, never deleted, and the incident’s evidence survives by construction).

The transparency log is unreachable

Section titled “The transparency log is unreachable”

Everything degrades explicitly, never silently: packs still verify (their log-head commitment is inside the pack), the external TSA anchor reports unreachable rather than pretending, and the archive service says so. Reconnection is a restart; the log’s journal replays.

Communications

Section titled “Communications”

The reference deployment’s status surface IS the incident channel: unidpp-stack status output and the console’s services matrix are pasteable evidence. For a tenant-facing incident, the tenant’s manifest name and the affected service binds identify the blast radius without exposing anything else.