When missing telemetry must stop a release
A release gate can make the wrong decision without receiving a visibly wrong number. An empty query, a stale scrape, or a zero supplied by the query itself can resemble a healthy result. Resilience Gate separates the validity of returned measurements from the hypothesis those measurements are meant to test.
Design decision examined. Repository implementation and tests are attributed to devSatym through the linked history. This case explains the mechanism and its limits; it does not claim sole original design or a new live experiment.
The scorer validates transport/JSON status and result kind, nonempty sample structures, numeric finiteness, strictly ordered range timestamps, freshness and range coverage. Only then are values aggregated and compared. Several symptom queries contain or vector(0); their present zero may originate from an absent raw counter. The validity parser cannot recover that missing left-side series.
Source-derived explanation; not a live observation.Read the diagram step by step
- A successful HTTP response must still have the expected Prometheus result shape.
- Empty or malformed returned evidence fails rather than silently becoming zero.
- Samples must be finite, ordered, fresh and sufficiently cover the range.
- An explicit or vector(0) expression can return zero for an absent raw symptom series.
- Valid returned data must still meet the scenario’s own comparison rules.
Public implementation sources
kubernetes/chaos-experiments/score_experiment.py · tests/gate/test_no_data.py · tests/gate/test_queries.py · tests/gate/test_scorecard.py
Reject unusable returned data before comparing a threshold
Section titled “Reject unusable returned data before comparing a threshold”The scorer accepts an injectable Prometheus reader. The HTTP adapter obtains responses; parsing, window validation, aggregation, and rule evaluation remain separately testable. That boundary makes it possible to supply an empty matrix or stale vector in a deterministic test without needing a cluster.
parse_prometheus_result() requires a successful API response and the expected result kind: an instant query returns a vector, a range query a matrix. It rejects empty returned series, malformed sample structures, non-finite values, and non-increasing range timestamps. A finite number is necessary but not sufficient. Instant queries must produce exactly one sample within 60 seconds of evaluation; each returned range series needs at least three samples, acceptable start/end coverage, and no gap longer than 75 seconds.
The rule evaluator converts an evidence error into a failed check with an error classification and no observed value. It continues evaluating later checks, so one invalid series does not hide useful diagnostic results. The experiment still fails if a required check fails.
The invariant is narrow: a comparison uses evidence that satisfies the parser and coverage contract. It does not establish that every useful raw metric existed before PromQL evaluated the expression.
A returned zero may have been created by the query
Section titled “A returned zero may have been created by the query”Some symptom queries deliberately end with or vector(0): application restarts, PostgreSQL /shorten 500 responses, signer /shorten 5xx responses, and payment replay counts. If the left-hand expression has no series, Prometheus can return a synthetic zero with a current timestamp. The parser sees present, finite evidence and cannot reconstruct the missing raw series.
| Situation | What the scorer receives | Supported interpretation |
|---|---|---|
| No matching returned series | Empty result | Reject as missing evidence |
| Stale or non-finite returned sample | Invalid sample | Reject before rule comparison |
| Absent left-hand series with a zero default | Finite zero vector | The configured expression returned zero; raw-series presence remains unproven |
| Valid target readiness reaches zero then one | Covered range | The selected target’s measured outage and recovery meet that check |
Presence, useful traffic, fault observation, and symptom absence therefore need different assertions. A zero error counter cannot establish that requests reached the application. A restart check cannot replace the target readiness series. The parser’s docstring also mentions unsafe warnings, but its current body does not inspect Prometheus’s warnings field; the implementation supports the specific checks described here.
Implementation and test history
Section titled “Implementation and test history”Commit 0894f0fe introduces the injectable query/result contracts, scorecard schema, fixtures, and query tests. Commit 16622bc0 adds explicit absent, NaN, stale, and insufficient-data controls. Later 57c23f2c corrects scoring around confirmed injection and recovery.
The current query tests and negative evidence tests demonstrate the returned-data contract with supplied fixtures. They were inspected for this explanation; this article does not report a newly executed platform test.
What the recorded boundary test establishes
Section titled “What the recorded boundary test establishes”The 2 October scorer-only record supplies deliberately unreachable http://127.0.0.1:1. Its five PostgreSQL telemetry checks fail with transport errors, and its final scorecard is fail.
This is evidence that the supplied unreachable endpoint does not become a pass. It is a manual direct Job, not an outage of shared Prometheus, a Workflow/load/fault exercise, Kargo verification, or promotion. A directory label and a screenshot cannot extend that scope.
The next review question is whether every zero-default expression has enough independent evidence to distinguish “nothing happened” from “the raw signal was absent.” Measurements and validity explains the query boundary; cleanup and final outcome follows a measured result into the enclosing release decision.
Maintained by Satyam Agnihotri · DevOps & Cloud Engineer