Auto Recovery

Auto-recovery is the process of restoring a system to a known-good state after a failure, without human intervention. Take an illustrative case: at 3 AM a container in a Kubernetes pod stops responding because its process has deadlocked. Its liveness probe fails, the kubelet restarts the container, and the readiness probe keeps the pod out of the Service's endpoints until it can take traffic again. The on-call engineer wakes up to a resolved alert, not a page demanding immediate action.

Auto-recovery handles the symptoms - it restarts the process, replaces the instance, or fails over to a backup. It does not investigate why the failure happened. That is the domain of auto-remediation, which identifies root causes and applies targeted fixes. Recovery is the fire sprinkler system; remediation is the fire investigator. Most organizations need both, but they solve different problems and operate at different levels of the stack.

What Auto-Recovery Does (and Does Not Do)

Auto-recovery has a narrow, well-defined scope. It detects that something is broken, and it restores service. It does not ask why the thing broke, whether it will break again, or whether the underlying architecture needs to change. This narrowness is a feature, not a limitation - it makes recovery fast, predictable, and safe to automate.

Think of it through a concrete analogy. When a web server process crashes, auto-recovery restarts it. When an EC2 instance fails a health check, the Auto Scaling Group terminates it and launches a replacement. When a database primary goes down, a replica is promoted in its place. In every case, the system returns to a working state - but the underlying cause (memory leak, hardware failure, network partition) remains unaddressed until someone or something investigates.

The Four Phases

Auto-recovery follows a consistent lifecycle across platforms and implementations. Each phase has a clear input, a clear output, and a handoff to the next.

Phase 1: Detect. A liveness probe sends an HTTP request to the application every 10 seconds, the Kubernetes default period. The probe returns a 503. After three consecutive failures, the default threshold, the kubelet treats the container as unhealthy. Detection also happens through metric thresholds (CPU above 95% for 5 minutes), log pattern matching (repeated OOM errors), or synthetic transactions (a test purchase flow that times out).

Phase 2: Diagnose. The system determines what failed and how severe it is. Is this a single pod in a ten-replica deployment (minor) or the only instance behind the load balancer (critical)? Is the failure isolated to one availability zone or affecting the entire region? Diagnosis determines which recovery playbook to execute - a single-pod restart requires different action than a zone-level failover.

Phase 3: Recover. The system executes the appropriate recovery action. For a crashed or unhealthy container, the kubelet restarts it in place, with an exponential back-off between repeated restarts; if a whole pod is lost, its Deployment creates a replacement from the same template. For an unhealthy EC2 instance in an Auto Scaling Group, the group terminates the instance and launches a replacement behind the load balancer. For a database primary failure, a replica is promoted and the application reconnects. The recovery action is selected from a pre-defined playbook - there is no improvisation at this stage.

Phase 4: Validate. After recovery, the system confirms the fix worked. The health check probe returns 200. Response latency drops back to normal ranges. Error rates fall below threshold. If validation fails, the system escalates - either retrying recovery with a different approach or alerting a human. Validation prevents the recovery system from declaring success while the application is still broken.

Recovery Patterns

Four patterns handle the majority of automated recovery scenarios. Each applies to a different failure mode.

Circuit breaker. When a downstream dependency starts failing, the circuit breaker stops sending requests to it. Instead of every request timing out (degrading the caller), the breaker returns a fallback response immediately. After a configured interval, it sends a single test request. If the dependency is back, the breaker closes and normal traffic resumes. If not, it stays open. This helps stop a single failing service from cascading into a wider outage. Netflix's Hystrix library brought the pattern into wide use; Hystrix is now in maintenance mode, and Netflix points new projects to resilience4j.

Retry with exponential backoff. Transient failures - a momentary network blip, a connection pool exhaustion that clears itself - often resolve on their own. In an illustrative policy, the system retries the failed operation after 1 second, then 2, then 4, then 8, with random jitter added to each interval. The jitter spreads retries out: without it, 500 clients that all failed at the same time would all retry at the same time, overloading the recovering service.

Bulkhead isolation. Critical resources - thread pools, connection pools, memory allocations - are partitioned so that one runaway consumer cannot exhaust them for everyone. If the reporting service starts consuming all available database connections, the bulkhead ensures that the checkout service retains its own dedicated pool. The failure is contained to the reporting service rather than taking down the entire platform.

Failover. When a primary component fails entirely, traffic shifts to a standby. Database failover promotes a read replica to primary. DNS failover routes traffic to a different region. Load balancer failover removes unhealthy targets and distributes traffic to healthy ones. The key requirement is that the standby must be genuinely ready - warm, synchronized, and tested through regular failover drills.

Auto-Recovery in Practice

Three illustrative examples show how recovery patterns combine in practice.

Kubernetes deployment with liveness probes. A payment service runs as a Deployment with three replicas, each configured with a liveness probe that hits /healthz every 15 seconds. One replica's JVM enters a GC death spiral - it is technically running but not responding within the 3-second probe timeout. After three consecutive failures (about 45 seconds), the kubelet restarts the container. A failing readiness probe takes the pod's address out of the Service's endpoints and keeps it out until the restarted application has fully initialized and connected to its database, so the other two replicas carry the traffic in the meantime. A PodDisruptionBudget does not help here: it limits voluntary disruptions such as node drains, and involuntary disruptions cannot be prevented by it. Spare replicas are what keep the service up during the restart.

AWS Auto Scaling Group replacing an unhealthy EC2 instance. An application runs behind an Application Load Balancer (ALB) on three EC2 instances in an Auto Scaling Group (ASG). The ALB health check sends an HTTP request to each instance every 30 seconds, the default for instance targets. One instance's EBS volume develops latency issues, causing the application to respond slowly and eventually time out on health checks. After two consecutive failures, the default unhealthy threshold, the ALB stops routing traffic to the unhealthy instance. Because Elastic Load Balancing health checks are turned on for the group (they are optional; the built-in checks are EC2 status checks), the ASG marks the instance unhealthy, terminates it, and launches a replacement from its launch template. The new instance boots, runs its user data script to configure the application, registers with the ALB, passes its health check, and starts receiving traffic. How long that takes depends mostly on boot and configuration time, and none of it needs a human.

A data pipeline that retries and repairs. Orchestrators carry the same ideas into data work. In Airflow, a failed task with retry attempts left goes to the up_for_retry state and is rescheduled, up to its retries count. In Databricks Lakeflow Jobs, a retry policy is set per task, and for most configurations the default is not to retry at all; a repair run reruns only the failed tasks and their dependents. dbt retry re-executes the last invocation from the point of failure. When the data itself is damaged, Snowflake Time Travel can query or clone a table as it was, or restore a dropped one with UNDROP, within the retention period (1 day by default, up to 90 days on Enterprise Edition). Retrying is only safe when the task is idempotent: Databricks warns that a repaired task restarts from the beginning and can duplicate data it had partly written.

What Goes Wrong

Auto-recovery is not risk-free. Poorly configured recovery automation creates its own category of incidents.

Recovery loops. A pod crashes because of a configuration error - a missing environment variable, a bad database connection string. Auto-recovery restarts it. It crashes again immediately. Kubernetes reports CrashLoopBackOff and backs off between restarts (10 seconds, then 20, then 40, capped at five minutes), but the container will never succeed because the underlying cause is not transient. A Deployment's pods keep restarting indefinitely, so without an alert and an escalation policy the loop can run for hours before anyone notices. The mitigation: alert on restart counts and CrashLoopBackOff status and hand the problem to a human; for batch work, cap retries explicitly, for example with a Kubernetes Job's backoffLimit or an orchestrator task's retry count.

Cascading failovers. Region A fails. Auto-recovery shifts all traffic to Region B. But Region B was provisioned for 50% of total capacity - it was the backup, not a full mirror. The doubled load overwhelms Region B, and now both regions are down. The mitigation: capacity planning that ensures every failover target can handle the full load, combined with load shedding that drops low-priority traffic rather than accepting all of it.

Thundering herd. A widespread issue causes 200 application instances to crash simultaneously. Auto-recovery starts all 200 at once. Every instance's startup routine queries the configuration database, floods the cache layer with cold reads, and establishes connection pools. The downstream systems, already stressed from the original failure, collapse under the startup load. The mitigation: rolling restarts with staggered intervals, connection pool warm-up, and circuit breakers that limit the rate of simultaneous recovery actions.

Recovery that masks problems. Auto-recovery restarts a service that crashed due to a memory leak. The service works fine for another 6 hours, then crashes again. The recovery system restarts it again. This cycle repeats for weeks, with recovery masking a genuine bug that never gets investigated because the pager never fires long enough for someone to notice the pattern. The mitigation: track recovery frequency per service and alert when a service is being recovered more often than its baseline.

Measuring Recovery Effectiveness

Four metrics tell you whether auto-recovery is working as intended.

Mean Time to Detection (MTTD) measures how quickly the system identifies a failure. A health check with 10-second intervals and a 3-failure threshold detects a hard failure in roughly 30 seconds. Synthetic monitoring that runs critical paths every minute adds another detection layer. Set the target per service tier, tighter for the services that matter most.

Mean Time to Recovery (MTTR) measures the elapsed time from detection to validated restoration. It varies widely by mechanism: a container restart can complete in seconds, an instance replacement takes as long as the new instance needs to boot and pass its checks, and database failover time depends on the engine and its configuration. For software delivery, DORA (the DevOps Research and Assessment program, not the EU regulation) now tracks failed deployment recovery time, a metric that replaced its earlier MTTR. If MTTR is consistently long, the bottleneck is usually in the validation phase - slow health checks or readiness probes that take too long to pass.

Recovery success rate is the percentage of automated recovery attempts that resolve the issue without human intervention. Track it per failure type rather than against a universal benchmark. When it is low, either the recovery playbooks are too simplistic for the failure modes they encounter, or the underlying infrastructure has chronic issues that recovery cannot solve.

False positive rate measures how often recovery triggers unnecessarily - killing a healthy pod because a health check endpoint was temporarily slow, or replacing an instance that was actually fine. High false-positive rates erode trust in the recovery system and, worse, can cause outages: unnecessarily killing pods during a traffic spike reduces capacity exactly when it is needed most. Review every false trigger and tune the probe or threshold behind it.

How Dawiso Supports Recovery Operations

Recovery playbooks need context to work correctly. When a data pipeline fails at 3 AM, the recovery automation needs to know: which services depend on this pipeline? What is the downstream impact of an outage? Who owns the affected data assets? Is this a Tier 1 regulatory pipeline that requires immediate escalation, or a Tier 3 internal report that can wait until morning?

Dawiso does not detect failures, restart pipelines or roll data back; those jobs belong to the orchestrators and platforms described above. What Dawiso's data catalog holds is the context those questions need. Admins can assign a data owner and data steward to a data asset, so the people accountable for it are on record. Assets in data products can carry PII and confidentiality classifications, set by users as descriptive metadata. Data lineage, built from supported connectors, dbt, SQL parsing (not on by default) and manually added relations, shows the downstream reports and tables a failed load can reach, as far as that lineage has been captured. In an illustrative case, if the "Customer" load fails, lineage shows that "Order Summary," "Churn Prediction" and "Regulatory Report" sit downstream, and the team can decide what to restore and check first.

Through the Model Context Protocol (MCP), the Dawiso MCP server lets an AI agent or an engineer's tooling search the catalog, read an object's details and user relations, and trace lineage upstream or downstream. A runbook step or an incident assistant can use it to look up who owns an affected asset and what depends on it, so a non-critical batch job is not handled with the same urgency as a regulatory feed, and a downstream table is not reloaded before the upstream source that feeds it.

Auto-recovery keeps systems running. Governed metadata tells the people and tools around it what is being recovered, why it matters, and who is accountable for it.

Sources

• Kubernetes, Pod Lifecycle (container restart back-off, CrashLoopBackOff, liveness and readiness probes).
• Kubernetes, Configure Liveness, Readiness and Startup Probes.
• Kubernetes, Disruptions (PodDisruptionBudgets and involuntary disruptions).
• AWS, Health checks for instances in an Auto Scaling group.
• AWS, Health checks for Application Load Balancer target groups.
• Microsoft Learn, Azure Architecture Center: Circuit Breaker pattern, Retry pattern and Bulkhead pattern.
• Netflix, Hystrix (maintenance-mode notice).
• Google, Site Reliability Engineering, Chapter 1: Introduction.
• DORA, DORA's software delivery metrics.
• Apache Airflow, Tasks (retries and task states).
• Databricks, Configure and edit tasks in Lakeflow Jobs (retry policy) and Troubleshoot and repair job failures.
• dbt, retry command.
• Snowflake, Understanding and using Time Travel.
• Dawiso Help, Role of ownership in data governance.
• Dawiso Help, PII and confidentiality.
• Dawiso Help, Data lineage.
• Dawiso Help, Sources of data lineage.
• Dawiso Help, Dawiso Model Context Protocol (MCP).