Gateway Operations Runbook

This runbook is for Duplo engineers and integration operators supporting the NRS Invoice Gateway. It explains which component owns a failure, which evidence to collect, and when a retry would be unsafe. Customer applications should still make decisions from the public GW_* Problem Details contract.

Failure ownership at a glance

Fresh Spend authorization is a safety requirement. Gateway must not serve an authenticated tenant from stale authorization data during a Spend outage.

Trace one request end to end

1. Obtain the Gateway-generated X-Request-ID or Problem Details requestId. A caller-supplied X-Request-ID is optional metadata and is not the authoritative identifier.
2. For an ambiguous or recoverable mutation, also retain operationId, the stable idempotency key, IRN, represented-business ID, target state, and UTC time.
3. Find the gateway_request_completed JSON event by requestId. Confirm route, state, mode, status, public error code, duration, and response size.
4. Follow gateway_dependency_call events with the same ID. Inspect dependency, operation, latency, upstream status, normalized failure class, contract version, provider correlation ID, and circuit state where present.
5. For a sign, inspect debit, reversal, recovery-marker, and reconciliation state. For a transmission, inspect the durable claim by operationId. Never infer “not committed” from a missing response.
6. Use the code-specific action in Gateway error responses. Escalate with the safe support packet, not raw headers or invoice data.

Gateway logs and alert signals must never contain API keys, authorization headers, NRS passwords, private keys, raw idempotency keys, invoice bodies, unredacted query strings, taxpayer data, or upstream response bodies. Hash an idempotency key when correlation is necessary.

Health endpoints

The protected dependency snapshot can include Spend registry, Spend authorization, Spend management, administrator, NRS test/live, Redis, Slack, processor-heartbeat, circuit state, contract version, in-flight capacity, and recovery counts. unknown means no trustworthy observation has been reported; it is not equivalent to healthy. The Spend processor may remain unknown until Spend publishes its heartbeat to Gateway.

Capacity, rate, and circuit protection

Gateway applies controls in layers so one domain cannot occupy every worker or outbound socket:

1. Canonical host, route, content type, declared body size, and routing-loop checks run before dependency work.
2. Distributed early admission control keys requests by canonical domain and a client identity supplied by the trusted edge. The edge must overwrite that header; caller X-Forwarded-For values are not trusted.
3. A process-local, no-wait capacity guard caps total and per-domain in-flight work. A full guard returns 503 GW_GATEWAY_CAPACITY_EXCEEDED with Retry-After instead of building an unbounded queue.
4. After authentication, route-specific quotas apply. Sign remains 120 requests per 60 seconds and management 30 per 60 seconds by default. Other route budgets are enabled only after observation and load testing.
5. Dependency bulkheads cap concurrent calls separately from Gateway admission. Current safe defaults are configurable and must be validated against the deployed 1 CPU/1 GiB pod profile, outbound socket pool, and downstream capacity.

Redis backs fleet-wide admission and mutation coordination. Protected work fails closed with structured 503 when Redis cannot enforce a safety decision; local hard caps remain in place to protect the process.

The same edge-owned client address is carried over Gateway's authenticated internal authorization hop for credential IP allowlisting. An empty ipWhitelisted list is unrestricted. With a non-empty list, a missing, malformed, or non-matching trusted address fails authorization with 403; never bypass that failure by reading a caller-supplied forwarding header.

Circuits are independent for Spend registry, authorization, management, administrator, and each NRS state. The default policy opens after at least 20 eligible calls have a failure ratio of 50% or more in 30 seconds, remains open for 30 seconds, and admits one half-open probe. Timeouts, transport errors, 5xx responses, rejected deployment-owned credentials, and invalid dependency contracts are failures. Customer 4xx or business rejections do not count.

When a circuit or bulkhead rejects work, honor Retry-After and retain the logical operation's idempotency key. Do not bypass the protection by switching hosts, replicas, credentials, or mutation keys.

Routing-loop response

Gateway rejects startup configuration when a Spend or NRS origin resolves to a configured Gateway public origin or the invoice-domain suffix. Network egress allowlists remain the primary control.

Every approved Gateway outbound dependency request carries the reserved X-Duplo-Gateway-Hop: 1 marker. Any request returning to Gateway with a non-zero marker is rejected immediately with 508 GW_ROUTING_LOOP_DETECTED, logged, and alerted. Customers must not send or copy this header.

For a loop incident:

1. Stop retrying the affected route.
2. Identify the configured upstream from the dependency event; never follow arbitrary request-supplied URLs.
3. Verify DNS, reverse-proxy routes, service discovery, and environment-specific base URLs.
4. Confirm the target is an approved Spend, administrator, or NRS origin rather than a Gateway public or service address.
5. Correct configuration and egress policy, restart safely, then prove one read-only request before restoring mutation traffic.

NRS contract versioning

NRS behavior is isolated behind an explicit, state-specific adapter. The current supported contract is v1; TEST and LIVE select their versions independently. An unsupported version prevents the affected integration from starting or serving rather than silently guessing a schema.

Gateway preserves the status and bytes of ordinary successful NRS relay responses after safe-header filtering, so additive response fields remain compatible. It validates only fields needed for a Gateway decision, including onboarding authentication and entity selection, sign success, and IRN/business ownership. A sign is successful only when HTTP is 2xx, body code is 201, and data.ok is true; a malformed, empty, renamed, or otherwise unrecognized success envelope remains an ambiguous sign outcome.

A new NRS path, method, API version, credential header, or redirect requires a new Gateway adapter/route release. Redirects remain disabled. Deploy a new adapter alongside the existing one, prove golden fixtures and the NRS sandbox in TEST, then canary LIVE before changing the whole state. Credential rotation may stage current and next values, but a mutation must never be retried automatically with a second credential after an uncertain first dispatch.

Identity responses are durable snapshots. Additive name, template, or active-state drift can be reviewed; NRS business-ID or TIN drift requires an audited migration. Never silently rewrite the local identity or break a previously published public identifier alias.

Idempotency and safe retry rules

The current Pass-through transmission version accepts a missing idempotency key for compatibility and derives a legacy operation identity from the request binding. New mutation versions can require Idempotency-Key. Absence of a key never makes an uncertain external mutation safe to repeat.

Slack alert delivery

Gateway sends alerts directly to the configured Slack HTTPS webhook; it does not use the Slack SDK, Alertmanager, or a cloud monitoring delivery service. Request handlers publish a small allowlisted FailureSignal to a Redis stream and return without waiting for Slack. A background worker aggregates across replicas by environment, dependency, operation, and failure class.

The default sustained-failure trigger is either five consecutive failures or at least 20 calls in five minutes with a failure ratio above 20%. A 15-minute cooldown deduplicates an active incident. Five consecutive successes send one recovery notification.

Immediate signals bypass the sustained threshold for rejected internal service/provider credentials, unconfirmed billed sign or reversal work, routing loops, capacity/circuit safety incidents, missing post-sign recovery state, and overdue reconciliation work. Slack delivery has a three-second timeout and up to five exponential-backoff attempts. Exhausted deliveries go to the alert dead-letter stream and emit slack_alert_delivery_failed; Slack failures never recursively alert Slack.

An alert contains only allowlisted context: environment, dependency, operation, normalized failure class, first/last occurrence, count and ratio, up to three request IDs, optional operation/provider IDs, public error code, and configured runbook/log-search links. If Redis is unavailable, an immediate or Redis-related incident can use a per-pod best-effort direct webhook with the same 15-minute cooldown.

Incident procedures

Spend registry or authorization

• Confirm whether the public code is registry or authorization unavailable; a warm domain cache never permits stale authorization.
• Search the request ID across Gateway and Spend, then inspect contract version and deployed service credential.
• Confirm NRS received no request.
• Restore Spend, close the incident only after five successful observed calls, and let callers retry according to their Problem Details.

Administrator and billing recovery

• Before debit: confirm NRS was not called and instruct a same-key retry only when the response permits it.
• After debit: inspect the billing attempt, reversal queue, reconciliation marker, and operation ID. Never issue a manual compensating debit or replay NRS from the customer request.
• Escalate immediately when a reversal or reconciliation result cannot be confirmed.

NRS outage or contract drift

• Separate throttling, timeout, transport/5xx, provider-credential rejection, and invalid-contract responses.
• Compare the active TEST/LIVE adapter version and provider correlation ID with the last known good call.
• For ordinary reads, restore service or allow bounded retries. For sign/transmit, reconcile the original operation before any new dispatch.
• If a payload field or success envelope changed, capture a sanitized fixture, update the versioned adapter and golden tests, test in the NRS sandbox, then canary LIVE.

Stored processor backlog

• Treat 202 Accepted as durable intake acknowledgement, not sign success.
• Use Spend's processor heartbeat, oldest queue age, and invoice state. Infrastructure outages stay pending/validated and use dependency backoff; they must not consume an invoice's business retry budget or terminally reject a valid invoice.
• Spend, not Gateway, owns alerts occurring after the synchronous 202 response.

Ambiguous sign or transmission

• Freeze automatic mutation retries for the IRN and represented business.
• Preserve request/operation IDs, the original idempotency key, request hash or exact bytes, target state, and UTC time.
• Inspect authoritative NRS state plus the Gateway/Spend recovery record. A missing optional NRS field is not proof that nothing committed.
• Resolve billing and provider state before allowing a deliberate new attempt.

Before closing an incident

• The dependency is healthy through the same contract version and credentials used by production traffic.
• Circuits have closed through a successful half-open probe; queues and recovery work are within their service-level objective.
• A safe read and, when authorized, one controlled idempotent workflow succeed end to end.
• No customer was instructed to replay an unconfirmed mutation.
• Slack emitted one recovery notification after the configured success threshold.
• The incident record contains request IDs and operation IDs, not secrets or raw invoice data.