Skip to content

Redis disappears

A cache can make an application faster while making a dependency failure harder to interpret. A redirect that succeeds during Redis failure may have fallen back correctly, or it may simply have arrived after recovery. A growing cache-miss counter may reflect an outage, or ordinary first-time reads. Resilience Gate asks for several observations because no single one proves its Redis hypothesis.

The hypothesis is bounded: once the application has achieved initial readiness, a selected staging Redis pod can become unavailable for 60s while the application resolves mappings through PostgreSQL, stays within a redirect latency limit, avoids application restarts, and observes Redis recovery.

Redis outage removes the fast cache path while redirects use PostgreSQL and later cache fills resume
A cache hit ends early; fallback uses PostgreSQL.

GET code reads Redis. A hit returns 302 immediately. Miss or cache exception counts a miss and reads PostgreSQL. A durable mapping returns 302 after an optional cache-fill attempt; database failure returns 503. Cache misses alone do not prove outage.

Source-derived explanation; not a live observation.
Read the diagram step by step
  1. A Redis hit returns the redirect immediately.
  2. A miss or Redis exception counts a cache miss and reads PostgreSQL.
  3. A durable mapping returns a redirect; an optional cache-fill failure can be tolerated.
  4. Database failure is a different boundary; cache misses alone do not prove Redis outage.
Public implementation sources

app/main.py · tests/app/test_cache.py

Readiness matters before the fault. /ready always checks PostgreSQL and checks Redis until the process’s first successful readiness. After that success, has_been_ready permits steady-state service without Redis. This allows the cache fault to exercise degradation while the application remains usable.

A cold application process behaves differently. If it starts during Redis failure, it cannot pass that initial cache check even if PostgreSQL is working. The startup probe also calls /ready, so initial unavailability can exhaust its startup budget. The resilience claim is therefore about the established application under this scenario, not an assertion that every new replica starts without Redis. The health explanation covers the lifecycle trade-off.

The orchestration checks ready targets and requires useful paid traffic from its own load Job before injection. Its safe marker follows a successful paid creation and redirect. That precondition establishes that the traffic source exercised the path before the experiment; it does not guarantee every later iteration succeeds throughout the fault.

Remove the cache path and follow one redirect

Section titled “Remove the cache path and follow one redirect”

The Redis PodChaos source requests pod-failure, mode: one, and duration: 60s. Its selector requires the staging namespace, the staging Helm instance, and Redis’s master component. It targets neither every Redis pod nor the production-like namespace. The environments still share a lab cluster, so these selectors do not imply complete infrastructure isolation.

The redirect route attempts GET url:<code>. A truthy cached destination increments url_shortener_cache_hits_total and returns 302. If no usable value is present, or the cache wrapper raises CacheUnavailable, the route increments url_shortener_cache_misses_total and queries PostgreSQL.

After a matching database row is found, a SETEX attempts to populate the cache. Cache write errors are ignored for the response; the authoritative lookup has already succeeded. The route returns 302 with Location. An unknown mapping returns 404, and a failed required database lookup returns 503.

The fallback is deliberately simple. Redis socket timeouts are short, and cache operations translate failures into one bounded application exception. PostgreSQL bears the additional work. This saves availability for the exercised request path while sacrificing cache speed and increasing load on essential storage.

A cache miss proves fallback, not an outage

Section titled “A cache miss proves fallback, not an outage”

Normal first redirects are expected misses because URL creation does not populate Redis. Expiry, an absent code, and read failures also take the miss path. The counter’s help text explicitly says “Redis misses or unavailable cache fallbacks.” No separate miss label identifies the cause.

This matters especially for paid load: the client creates a new destination for each iteration and then resolves its new code. Even healthy Redis therefore produces many first-read misses. The scorer names its miss check redis-fallback-observed, and correctly establishes only that the fallback path was exercised.

Proof of the fault must come from the target’s readiness series and applied workflow state in the same run. The cache-miss increment is an additional behavioral observation. The app’s Redis dependency gauge is also insufficient: after initial readiness, /ready stops checking Redis, so that gauge may retain an earlier 1 through an outage.

Read the seven-check scorecard as one argument

Section titled “Read the seven-check scorecard as one argument”

The scorer adds release identity validation to six Redis behavior checks:

Check Required observation Interpretation
Meaningful traffic Non-probe HTTP increase > 50 across fault plus settle duration. Requests reached the application; the count alone does not establish every request succeeded.
Fallback path Cache-miss increase > 0 over that duration. Database fallback was exercised; it does not establish why Redis missed.
Redirect latency Maximum observed one-minute histogram p95 < 1.2 seconds. The degraded resolution path stays below the chosen measured latency bound.
Application restarts Maximum container restart increase < 0.5. No application restart is observed by the configured query.
Target outage Minimum Redis pod readiness == 0 in the range. The selected dependency was observed unavailable.
Target recovery Final Redis readiness sample == 1. The dependency recovered before the observation ended.
Release identity Immutable candidate plus bounded run identity validate. The result belongs to a specific candidate and run.

The latency rule queries the /{code} handler and aggregates histogram buckets across matching application series before calculating the p95. The scorer takes the maximum of the resulting time samples. It is not the single worst request, an end-to-end payment duration, or a promise that every redirect completes under 1.2 seconds.

The restart expression deliberately includes or vector(0). An absent raw restart series can therefore produce a valid zero result. Required readiness and fallback series do not receive the same fallback. Collection validity and the exact query expression both matter; see metrics and scoring.

The retained 1 October staging record contains a passing Redis scorecard. Its observation starts at 17:49:27 UTC, uses an actual injection anchor at 17:49:47, and ends at 17:51:47, combining 60s of fault and 60s settle time after injection.

That card records approximately 293.714 non-probe requests, 98.286 fallback increments, a maximum observed p95 of 0.32 seconds, and zero observed application restarts. Readiness reaches zero and its final sample reaches one. Counter increases are fractional because Prometheus extrapolates increase; they are not literal fractional HTTP requests. The scorecard binds the full image digest, revision, and run identifier.

This is historical staging evidence for the named candidate. It is not the later v1.0.0 release record, a production-like Redis test, or permission to transfer the pass to any new image. The historical cases and recorded release keep those campaigns separate.

Recovery and the remaining capacity question

Section titled “Recovery and the remaining capacity question”

Target readiness returning to one establishes the scorer’s recovery condition. Subsequent cache fills and hits support renewed acceleration, but they are not separate required scored checks. Cleanup and the enclosing Job must also succeed before the gate result is usable; a passing Redis card alone does not approve promotion.

Fallback demonstrates a chosen degradation path, not unlimited database capacity. A larger workload, longer cache outage, simultaneous PostgreSQL failure, or a wave of cold replicas could behave differently. The repository’s cache tests establish deterministic fallback and cache population; Redis scoring tests establish rule behavior. Neither supplies a new live capacity measurement.

Would a fresh process remain ready while Redis is unavailable? Health, startup, and recovery follows the process-local readiness flag. The degradation study connects that design choice to its tested and recorded scope.

REVIEWED EVIDENCE / CAPTURE 19

What supports the Redis fallback story?

The historical recovery campaign pairs Redis outage evidence with fallback activity and a bounded redirect-latency check. This is one tested window, not an availability promise.

Observed window: 2026-10-02 05:25:30 UTC–05:28:15 UTC

Redis Ready state alone falls from 1 to 0 and returns to 1; redirect p95 rises during database fallback then recovers while traffic continues and 5xx remains zero

Original available · recorded screenshot · numerical verdict not assessed from pixels

Campaign identity and provenance
Captured
3 Oct 2026, 13:10:50 UTC
Source identity
platform dashboard 7ea5e911aba159966b2a54657c501a8e643ee18f; release source 0a98c08ccab5fa7b05efcecf877c714e5c89f025
AnalysisRun
staging.01m3xgybahhk42nh7jd01xyc4k.e7171a5 — Successful
Image / OCI identity
sha256:a5beda836336de1677e27c3c2ff09a1ab27d9366040cd985c347d700de6f476b
Rendered revision
bc388eaff8ff294e1f59b3c417a18c6b96dd5d39
Original SHA-256
b1d286a93888755b7e1b7d80c4e3dcc92909203376a04824a09bb008d6a9c46d

Open original image

Maintained by Satyam Agnihotri · DevOps & Cloud Engineer