Skip to content

Why a passing scorecard can end in a failed gate

A scorecard can say that a workload met its measured fault hypothesis while the temporary fault object or load Job remains behind. That is an unfinished run. Resilience Gate makes cleanup part of the orchestrator’s final process outcome, then carries that outcome through Kubernetes analysis and stage verification.

Design decision examined. The linked implementation and tests are attributed to devSatym. This case examines the decision boundary without claiming every Kubernetes error is classified correctly or that this website ran a new cleanup experiment.

A release decision passes through measurement, scorecard, orchestration and cleanup, Job outcome, analysis, eligibility, and an explicit promotion
A favorable card is earlier than the final Job and analysis; eligibility is earlier than an operator’s promotion action.

Metric checks and release identity produce experiment scorecards. The runner also accounts for workflow/load outcome and exact cleanup; cleanup can force failure after a PASS log. A successful gate Job contributes to the enclosing staging analysis alongside service health. Normal downstream Freight eligibility follows successful verification, while Project policy disables automatic staging/prod promotion. An operator’s manual approval is a distinct override; prod-like health is not another staging paid-chaos campaign.

Source-derived explanation; not a live observation.
Read the diagram step by step
  1. A scorecard reports its defined experiment values and release identity.
  2. The runner’s final outcome also includes workflow, load and cleanup.
  3. The gate Job contributes one result to the enclosing staging analysis.
  4. Successful staging verification grants normal Freight eligibility.
  5. Prod-like promotion still requires an explicit operator request.
  6. Manual Freight approval is an override, not proof of successful verification.
Public implementation sources

kubernetes/chaos-experiments/orchestrate.sh · kubernetes/chaos-experiments/score_experiment.py · kubernetes/kargo/analysistemplate.yaml · kubernetes/kargo/stage-staging.yaml · kubernetes/kargo/stage-prod.yaml · kubernetes/kargo/project.yaml · tests/gate/test_cleanup.py · tests/gitops/test_prod_contract.py

The orchestrator binds a run identifier, release revision, and immutable image digest to the sequence. It checks the target and traffic preconditions, uses a Lease to limit concurrent runners, creates run-scoped load and Workflow objects, records actual injection anchors, and invokes scoring.

Cleanup deletes exact objects it created: the named Workflow, Workflow-owned PodChaos objects, its load Job, and the held Lease. It does not sweep unrelated experiment objects. Calls share a bounded cleanup deadline; failed deletes or remaining Workflow-owned PodChaos objects mark cleanup failed.

The intended invariant is a successful final process outcome requires the scored run and its cleanup to finish successfully. Exact ownership keeps recovery from turning into an indiscriminate namespace cleanup.

main() logs GATE VERDICT: PASS before the EXIT trap runs. The trap preserves the current exit code, attempts cleanup, and changes the code to 1 if cleanup reports failure. The earlier PASS text remains in the log. A consumer must inspect the final process and enclosing records rather than search for one green line.

Signal What it establishes What comes next
Passing scorecard Configured checks passed for its supplied candidate/window Orchestration and cleanup still need completion
PASS log line Main scoring path reached its passing branch The exit trap can still fail
Final exit / Job outcome The process and its enclosing Job finished with an outcome Analysis combines the configured metrics
Successful staging verification The stage’s verification contract succeeded Normal downstream eligibility is available
Eligible Freight A candidate can be selected through the normal path The protected stage still requires a requested promotion
Reconciled downstream revision The selected render reached the target Its stage-specific health checks still have their own scope

The Job-backed AnalysisTemplate allows no chaos-verdict failure and no Job retry. The staging Stage also requires service health. Grafana annotation is optional diagnostic output, not the authority deciding the release. The Project policy leaves staging and production-like testnet promotion manual.

Exact-object absence has a remaining ambiguity

Section titled “Exact-object absence has a remaining ambiguity”

verify_absent() attempts a named-object GET. It returns failure if the GET succeeds, because the object still exists. Any nonzero GET is treated as absence. An API outage, permission denial, or timeout can therefore be interpreted the same way as not found.

The surrounding delete and PodChaos-list paths do record failures, but this helper does not distinguish every readback uncertainty. Describing the runner as universally fail-closed on all cleanup reads would exceed its implementation. A future improvement would classify not-found separately from an unsuccessful observation; that improvement is not part of this website task.

Commit 2d000a1c introduces run-scoped traffic, fault orchestration, and scoring. Commit 0840ea42 adds locking, bounded recovery, cleanup verification, permissions, and fake-kubectl tests.

The current cleanup tests check exact object ownership, failed-score cleanup, timing/budget contracts, and related orchestrator behavior using a hermetic command stub. They are inspectable implementation evidence; they do not establish every cluster failure mode.

The recorded short-deadline result is deliberately narrower

Section titled “The recorded short-deadline result is deliberately narrower”

The 2 October timeout-and-cleanup record runs with a one-second Workflow deadline and an inert temporary load source. It fails before the ordinary 90-second baseline reaches any injected fault. The log records FAIL and cleanup; a post-run check finds none of the named manual Job, temporary CronJob, Workflow, PodChaos, load Job, or Lease.

That result establishes the shortened-deadline boundary path. It does not establish default timeout behavior, a naturally occurring Chaos Mesh failure, real paid traffic, normal fault recovery, or Kargo promotion. The October 3 report describes the separate release campaign and its identities.

The next question is which enclosing record a reviewer uses as the final decision, and whether its candidate and run match the card being cited. Score to promotion walks that chain; promotion ownership separates selection from reconciliation.

Open original image

Maintained by Satyam Agnihotri · DevOps & Cloud Engineer