Skip to content

Signer disappears

A signer outage can make application metrics look quiet precisely because the client stopped sending paid creation requests. The application does not call the signer during /shorten; the load client does. This failure scenario tests isolation and signer recovery, and its interpretation must preserve that distinction.

The bounded hypothesis is that a selected staging signer pod becomes unavailable for 60s, the application avoids restarts and /shorten 5xx responses attributable to the observed run, no application replay increment is observed, and the signer target recovers. That is narrower than a guarantee of uninterrupted paid throughput.

A paid client attempts signer authorization before the application request, with an early failure return
Application isolation and continuous successful paid traffic are different claims during a signer fault.

In paymentFlow the client requests /sign-permit2 before /shorten. Non-200 or malformed/missing authorization fields mark client outcomes false and return without a creation call. Successful signing permits the application request, whose201 still requires settlement and durable persistence. The signer scorecard checks target outage/recovery, app restarts, app 5xx and replay symptoms but does not require meaningful traffic during the fault window. An optional independent redirect probe is a separate observation.

Source-derived explanation; not a live observation.
Read the diagram step by step
  1. The paid load client requests a fresh authorization from the signer first.
  2. A failed or malformed signing response ends that iteration before /shorten.
  3. Quiet application errors can therefore coexist with failed paid traffic.
  4. After signing succeeds,201 still requires application settlement and persistence.
  5. The signer gate has outage/recovery checks but no fault-window useful-traffic rule.
Public implementation sources

kubernetes/jobs/scripts/loadgen.js · signer/main.py · kubernetes/chaos-experiments/score_experiment.py · tests/gate/test_signer_scoring.py · tests/loadgen/test_loadgen_contract.py

When signing fails, this iteration can end before /shorten. Quiet application errors alone do not establish successful paid throughput.

The signer owns up to three configured payer wallet slots. At startup it validates settings, verifies the configured RPC chain, reads balance and Permit2 allowance, and sends an approval transaction only for slots whose allowance is below the configured payment amount. A successful slot remains available even when another slot’s bootstrap fails.

Service readiness is true when at least one slot has bootstrapped. The paid load setup imposes a stronger condition: enough configured and bootstrapped slots for its virtual users, plus cached balance sufficient for the bounded run. A 200 /health by itself therefore does not establish that a three-user payment workload is ready.

POST /sign-permit2 selects a slot, enforces the configured amount and allowed deadline, and creates a local EIP-712 signature. Its random uint256 Permit2 nonce needs no request-time RPC lookup. Signing does not settle a payment. An RPC outage after successful startup is therefore not equivalent to taking away the signer service itself.

/health and /wallets return boot-cached state rather than querying balances or allowances anew. They do not continuously verify funds, external chain availability, or successful settlement. Signer API tests verify that health, wallet inspection, and signing make no additional RPC calls after bootstrap.

Before fault injection, the current run’s load Job completes paid creation and redirect and emits the safe readiness marker. Its configuration prechecks the app and signer. This proves useful initial work without publishing a wallet, signature, payment envelope, or transaction identifier.

During a paid iteration, the load client calls /sign-permit2 before /shorten. If the signer response is not 200, or its JSON lacks the required authorization fields, the client marks signing, settlement, and end-to-end success as false and returns. It does not send an application creation request for that iteration.

Consequently, a zero increase in app /shorten errors can coexist with unsuccessful paid traffic. The same is true for a low payment replay counter: no creation request may have reached the replay boundary. These metrics are evidence about application isolation under the supplied workload, not automatic proof of sustained payment service.

The load source also offers an optional independent redirect probe. If configured with an existing short URL, it can keep exercising resolution while signing is unavailable. Its independent_redirect_ok_rate provides a different observation from paid end-to-end success. The scenario cannot assume the probe was enabled in every historical run; check the actual run configuration and summaries.

Apply one fault and inspect the six checks

Section titled “Apply one fault and inspect the six checks”

The signer fault selector requires staging namespace plus stable signer workload labels. Its mode: one and 60s duration bound the requested fault. The deployment separates the key-bearing process from the app and load generator.

The scorer requires an observed signer readiness minimum of zero and a final readiness value of one. It also requires application restart increase, app /shorten 5xx increase, and application payment replay increase each below 0.5. Release identity validation is the sixth check.

Unlike PostgreSQL and Redis checks, signer checks contain no meaningful-traffic rule for the fault window. The three low-counter queries also include or vector(0). It would therefore be incorrect to claim the signer card independently proves continuous paid traffic or that every missing error series causes failure. Pre-injection paid readiness and strict measurement validity are useful protections, but they do not add a missing per-window throughput contract.

For a broader operational assessment, compare target readiness with sign_success_rate, end_to_end_success_rate, signing duration, app request volume, and any independently configured redirect success. Those describe separate client and server events. The repository’s gate thresholds remain unchanged; this explanation makes their implemented scope clear.

When a replacement signer process starts, bootstrap runs again. It can recognize an existing sufficient allowance without submitting another approval. A failed chain verification prevents approval attempts, and failed slots stay unavailable. Bootstrap is idempotent for successful slots, but there is no HTTP retry-bootstrap endpoint or continuous background bootstrap loop in the service.

The retained historical signer scorecard records passing checks in its 1 October window and binds the same candidate as that gate’s other cards. It observes a target outage followed by final readiness, zero observed application restarts, zero observed /shorten 5xx, and zero observed application replay increments. It does not add a fault-window paid-success measurement that its schema did not collect.

Bootstrap tests cover allowance reuse, partial bootstrap, chain verification failure, and missing configuration without opening RPC. Permit2 tests recover signatures and verify binding to payment terms. Signer scoring tests exercise its exact rules. These are credential-free mechanism checks, separate from the historical live record.

Continue with the paid request, application design, or measurements. Isolation protects component responsibility; measuring useful work is what tells you how much service survived that isolation boundary.

Follow the request that reaches the application

Section titled “Follow the request that reaches the application”

What happens after the client obtains a usable authorization? The paid-request walkthrough follows facilitator verification, settlement, and persistence, keeping each side effect separate from signing.

REVIEWED EVIDENCE / CAPTURE 20

Where can signer loss stop paid traffic?

The historical signer panel must be read alongside the load-generator dependency: failure can occur before a request reaches the application.

Observed window: 2026-10-02 05:28:00 UTC–05:30:15 UTC

Signer Ready state falls from 1 to 0 and returns to 1 while PostgreSQL and Redis remain ready; application traffic recovers and 5xx remains zero

Original available · recorded screenshot · numerical verdict not assessed from pixels

Campaign identity and provenance
Captured
3 Oct 2026, 13:14:11 UTC
Source identity
platform dashboard 7ea5e911aba159966b2a54657c501a8e643ee18f; release source 0a98c08ccab5fa7b05efcecf877c714e5c89f025
AnalysisRun
staging.01m3xgybahhk42nh7jd01xyc4k.e7171a5 — Successful
Image / OCI identity
sha256:a5beda836336de1677e27c3c2ff09a1ab27d9366040cd985c347d700de6f476b
Rendered revision
bc388eaff8ff294e1f59b3c417a18c6b96dd5d39
Original SHA-256
66a5b5bb96f3e465f74af725e636b3298e840dd2e50e3984bb7c18c40c4baa74

Open original image

Maintained by Satyam Agnihotri · DevOps & Cloud Engineer