Skip to content

A green result can be insufficient

A green dashboard is an observation surface. A passing scorecard is a set of measured assertions. A successful gate Job says the orchestrated run finished with a successful process outcome. A successful staging verification supplies normal downstream eligibility. These statements connect, but each has its own boundary.

The release decision becomes credible when those boundaries agree for the same candidate and run. It becomes misleading when the reader jumps from one green layer directly to “promoted.”

Distinct scorecard, log, final gate Job, analysis, eligibility, requested promotion, and reconciliation outcomes
A favorable card is earlier than the final Job and analysis; eligibility is earlier than an operator’s promotion action.

Metric checks and release identity produce experiment scorecards. The runner also accounts for workflow/load outcome and exact cleanup; cleanup can force failure after a PASS log. A successful gate Job contributes to the enclosing staging analysis alongside service health. Normal downstream Freight eligibility follows successful verification, while Project policy disables automatic staging/prod promotion. An operator’s manual approval is a distinct override; prod-like health is not another staging paid-chaos campaign.

Source-derived explanation; not a live observation.
Read the diagram step by step
  1. A scorecard reports its defined experiment values and release identity.
  2. The runner’s final outcome also includes workflow, load and cleanup.
  3. The gate Job contributes one result to the enclosing staging analysis.
  4. Successful staging verification grants normal Freight eligibility.
  5. Prod-like promotion still requires an explicit operator request.
  6. Manual Freight approval is an override, not proof of successful verification.
Public implementation sources

kubernetes/chaos-experiments/orchestrate.sh · kubernetes/chaos-experiments/score_experiment.py · kubernetes/kargo/analysistemplate.yaml · kubernetes/kargo/stage-staging.yaml · kubernetes/kargo/stage-prod.yaml · kubernetes/kargo/project.yaml · tests/gate/test_cleanup.py · tests/gitops/test_prod_contract.py

Each favorable layer leaves a different question for the next owner. Cleanup can change the final process outcome after a PASS log line.

The scorer treats Prometheus response failures as failed checks. It validates response status, vector or matrix kind, sample structure, numeric finiteness, ordering, freshness, and coverage before aggregation. An HTTP 200 response is not enough. An empty returned series is not implicitly zero.

That rule has query-level exceptions. Several error, replay, and restart expressions end in or vector(0), deliberately allowing absent left-hand series to become a returned zero. This can be appropriate for a counter that appears only when a symptom occurs, but it prevents the parser from distinguishing an absent counter from a real observed zero. Request totals, cache misses, latency, and target readiness require their own returned evidence. Measurement interpretation explains this distinction.

For each experiment, the scorer asks whether the required set of observations jointly supports the hypothesis. PostgreSQL and Redis require more than 50 non-probe requests over the fault-plus-settle lookback. Redis also requires a positive cache-miss increase and maximum range p95 below 1.2 seconds. Dependency readiness must show an unavailable sample and end at one. Other checked symptoms must stay below their explicit bounds. The reference lists exact expressions and units.

The signer set checks signer outage/recovery, application restarts, /shorten 5xx, and payment replays. It has no separate meaningful-traffic check. A signer failure can stop the client’s signing step before it calls /shorten. Consequently, quiet application errors support limited isolation assertions; they do not prove uninterrupted successful payment traffic.

Then, inspect the scorecard’s candidate and window

Section titled “Then, inspect the scorecard’s candidate and window”

The CLI requires source/Freight revision, application digest, and bounded run identifier. It appends a release-identity check; an invalid identity makes the card fail. It evaluates all defined metric checks rather than short-circuiting on the first problem, so the JSON retains useful diagnoses.

The resilience-gate.scorecard/v1 record includes experiment, verdict, generated time, release fields, injection/start/end times, fault/settle durations, and every check’s expression, query kind, observed value, comparison, unit, sample count, and reason or evidence error. A failed invocation can have no valid window. Scorecard tests and the schema check this contract.

Do not substitute generated_at for the observation window. Scoring occurs after workflow completion; the returned samples describe the earlier fault interval. Do not substitute the card’s source/Freight revision for the output branch revision. Evidence metadata retains that output separately.

A passing card can coexist with a failing gate

Section titled “A passing card can coexist with a failing gate”

The runner joins results that the scorer cannot know. A scorecard has no Kubernetes client and cannot determine whether the overall workflow failed, the generator terminated, the fault actually exposed an Apply timestamp for every dependency, or cleanup completed.

Observation Narrow conclusion Additional layer required
One scorecard passes That experiment’s defined returned evidence and identity checks passed Other experiments and orchestration outcome
All three scorecards pass All scored experiment assertions passed Workflow/load outcome and cleanup
GATE VERDICT: PASS log appears Main orchestration reached a pass before its exit trap Final process and Kubernetes Job outcome
Gate Job succeeds The Job-backed chaos metric completed successfully Staging service-health and AnalysisRun status
Staging verification succeeds Normal upstream verification requirement is satisfied Explicit downstream promotion and reconciliation
Prod readiness/liveness succeeds Prod-like target passed those post-deploy checks Historical staging link and scope-correct evidence claim

Cleanup illustrates the difference. The runner can emit passing cards and a PASS line, then fail to delete a run-scoped object or verify cleanup before the deadline. Its exit trap changes the final exit code to failure. An accepted load/workflow request is even earlier: it proves an API accepted work, not that useful traffic or faults occurred.

The single-object cleanup absence helper also treats a failed read as absence without classifying its reason. Do not amplify the implementation into a guarantee that every API outage is independently detected. Recorded readback and the cleanup tests establish the specific evidence available.

The enclosing analysis decides staging verification

Section titled “The enclosing analysis decides staging verification”

The staging Stage references service-health and chaos-gate. The AnalysisTemplate configures the chaos verdict as one Job-backed measurement with zero allowed failures and no Job retry. Grafana annotation remains optional and non-fatal. A dashboard annotation is therefore not the decision source.

The AnalysisRun’s status must be inspected alongside the Job and scorecards. A screenshot of an AnalysisRun can illustrate that state at a captured moment, but its candidate, source, and window still need the linked records. A screenshot capture time may be later than the graph’s displayed observation window.

Once staging verification succeeds, Freight gains normal eligibility for prod. The Project policy still sets autoPromotionEnabled: false for prod. No automatic production-like promotion follows from the score. A manual Freight approval is a distinct override and cannot be cited as proof of staging verification.

Keep the recorded pass narrower than a service guarantee

Section titled “Keep the recorded pass narrower than a service guarantee”

The public historical gate retains scorecard output and cleanup observations for its named execution. The later 3 October release report summarizes fresh staging and prod-like results, with detailed bundles held in a private external archive. Public summary and public raw scorecards are different evidence availability states.

These records establish selected bounded behavior for exact identities in an owned testnet lab. They do not define a general availability SLO, prove traffic continuity through every signer fault, or establish every runtime signature was independently admitted. Earlier failed candidates remain valuable negative evidence and must keep their own dates and identities.

When reviewing a release, ask in order: did the query return valid evidence; did the bounded rules pass; did the candidate binding match; did orchestration and cleanup finish; did the enclosing analysis succeed; and was the next promotion explicitly requested and reconciled? That order makes “green” an inspectable result instead of an unqualified conclusion.

Request a transition only after inspecting its evidence

Section titled “Request a transition only after inspecting its evidence”

How is the next transition planned and requested? Promotion and re-verification distinguishes a read-only plan, fresh verification, named-Freight promotion, and manual approval, with their execution scopes.

REVIEWED EVIDENCE / CAPTURE 11

Did this staging candidate complete verification?

The reviewed Kargo capture shows the fresh October 3 staging verification as Successful. It does not expose the privately retained raw metrics bundle.

Observed window: 2026-10-03 14:37:29 UTC–14:46:28 UTC

Exact AnalysisRun title, green Analysis passed summary, and green chaos-verdict plus readiness; selected Freight history card shows hoping-warthog. Kargo's table value 853b0a1 is the verification-history ID, live-mapped to this Freight

Original available · recorded screenshot · numerical verdict not assessed from pixels

Campaign identity and provenance
Captured
3 Oct 2026, 15:08:49 UTC
Source identity
v1.0.0 / 3b70835335462f1b6f9b1dd17ab20d1108724f89; chart source 0a98c08ccab5fa7b05efcecf877c714e5c89f025
AnalysisRun
staging.01m4135pfqh94e76vzzn7fpbs4.d58be07 — Successful; verification ID 853b0a11-2953-4801-835f-5caba72e314c
Image / OCI identity
sha256:ab88d89c20ccf1a79b9f47f90aa417ac0f54d7852c39b444b7fb788975fdd6a1
Rendered revision
64d2ddedb1493e0c59aef9ecad0ad0a2ff4196cf
Original SHA-256
ace8e91b625aa69414f756762f7f5cf3c005bcfe0e432e760c324caebe6eff83

Open original image

Maintained by Satyam Agnihotri · DevOps & Cloud Engineer