Auto-Remediation

Auto-remediation goes beyond restarting failed services. It detects a problem, determines why it happened, and fixes the underlying cause — automatically. A database connection pool is exhausted. Auto-recovery would restart the service. Auto-remediation detects the pool exhaustion, identifies that a recent configuration change reduced the pool size below what the workload requires, adjusts the pool configuration back to the correct value, restarts the affected connections, and files a ticket for the team to review the root cause in the morning.

This distinction matters because recovery without remediation leads to recurring failures. If the system restarts a service every 6 hours because of a memory leak, but never addresses the leak itself, the organization is paying the cost of repeated outages and the engineering cost of ignoring the root cause. Auto-remediation is the smarter, more surgical sibling — but it requires significantly more context about the system to operate safely.

How Auto-Remediation Differs from Recovery and Manual Response

Three approaches to incident response sit on a spectrum from fully manual to fully intelligent. Understanding where each fits prevents organizations from either under-investing in automation or trusting it beyond its capabilities.

Manual incident response is the baseline. An alert fires, a human investigates, runs diagnostic commands, identifies the cause, executes a fix, and verifies the result. This works when incidents are rare and novel, but it scales poorly — the human is the bottleneck, and 3 AM response quality is measurably worse than midday response quality.

Auto-recovery handles the symptom. The system detects a failure, executes a predefined action (restart, failover, scale), and validates that the service is working again. It is fast and reliable but does not address root causes. A service with a memory leak will be restarted repeatedly without the leak ever being fixed.

Auto-remediation handles the cause. The system detects the failure, analyzes why it happened (configuration drift, resource exhaustion, expired credential, bad deployment), and applies a targeted fix. It requires richer context — understanding not just that something is broken but why it broke and what the safe corrective action is.

The Five Phases

Auto-remediation follows a five-phase cycle. Each phase builds on the previous one, and the quality of remediation depends heavily on the context available at the Analyze and Decide phases.

Walk through a concrete scenario: disk usage hits 90% on a data pipeline node.

Phase 1: Detect. A monitoring agent reports that /data/logs has exceeded the 90% disk usage threshold. The alert fires with metadata: which host, which mount point, current usage, growth rate.

Phase 2: Analyze. The remediation system inspects the disk. It finds that log files in /data/logs/pipeline/ account for 85% of the consumed space. The logs are growing at 2 GB per hour. The log rotation configuration is missing — logs have been accumulating for three weeks since a recent deployment removed the rotation config file.

Phase 3: Decide. The system evaluates remediation options. Option A: delete old log files (quick fix, but does not prevent recurrence). Option B: restore the log rotation configuration and compress existing logs (addresses root cause). Option C: expand the disk volume (addresses symptoms, expensive). The decision engine selects Option B based on the policy that root-cause fixes are preferred when available and the blast radius is low (single node, non-production-impacting).

Phase 4: Execute. The system applies the log rotation configuration (retain 7 days, compress after 1 day, max 5 GB per file). It compresses existing logs older than 24 hours. Disk usage drops to 45%.

Phase 5: Verify. After 30 minutes, the system confirms: disk usage is stable, logs are being rotated as configured, the pipeline continues processing normally. The remediation is marked successful, and a ticket is filed documenting the missing rotation config so the team can prevent recurrence in future deployments.

Remediation Patterns in Practice

Three categories cover the majority of automated remediation scenarios. Each requires different context and carries different risk.

Configuration drift correction. Infrastructure-as-code tools like Terraform define the desired state. When actual state diverges — a security group rule was modified manually, a database parameter was changed outside the change management process, a load balancer timeout was adjusted during an incident and never reverted — the remediation system detects the drift and reapplies the declared state. This is among the safest remediation patterns because the desired state is explicitly defined and version-controlled. The risk is low as long as the declared state itself is correct.

Resource scaling. A message queue depth exceeds its threshold for 5 consecutive minutes, indicating that consumers cannot keep up with producers. The remediation system adds two consumer nodes, waits for them to register with the queue, and monitors whether queue depth begins decreasing. If the added capacity resolves the backlog within 15 minutes, the remediation is validated. If not, the system escalates to a human — the problem might not be capacity but rather a stuck consumer or a poison message blocking the queue.

Security response. A TLS certificate is 72 hours from expiration. The remediation system detects the impending expiry, requests a new certificate from the CA, deploys it to the affected services using a rolling restart (not all at once), and validates that each service is serving the new certificate. This pattern prevents the dreaded "expired certificate" outage that has taken down major services — a failure mode that is entirely preventable through automation.

Building a Remediation Strategy

Organizations should not automate everything on day one. A phased approach builds confidence and safety incrementally.

Start with low-risk, high-frequency issues. Disk cleanup, log rotation, stale connection clearing, temporary file purging. These actions are safe (worst case: something gets cleaned up too aggressively), well-understood (the playbooks are trivial), and frequent enough to deliver measurable time savings immediately. This is where the remediation system builds its track record.

Expand to medium-risk automation. Scaling decisions (adding nodes, adjusting resource limits), configuration correction (reverting drift to declared state), and scheduled maintenance (certificate rotation, credential refresh). These actions carry moderate risk and require validation: did the scaling actually help? Did the configuration correction break anything? Safety mechanisms at this tier include blast-radius limits (only affect one node at a time) and mandatory validation windows.

Graduate to high-risk automation cautiously. Failover orchestration, deployment rollback, cross-service recovery sequences. These actions have wide blast radius and require deep system understanding. At this tier, human approval gates are appropriate for actions above a certain severity threshold. The automation proposes the action and provides its analysis; a human approves or modifies before execution.

At every tier, two safety mechanisms are non-negotiable. Circuit breakers on remediation actions prevent the system from applying the same fix repeatedly if it is not working — an anti-pattern where remediation oscillates between scaling up and scaling down, or between applying and reverting a configuration change. Blast-radius limits cap how many systems a single remediation action can affect — a runbook that restarts services should target one at a time, not all at once.

When Remediation Goes Wrong

Auto-remediation is more dangerous than auto-recovery because it changes system state rather than merely restoring it. When it goes wrong, the damage is typically larger.

Blind remediation. A remediation script detects high CPU on a node and automatically scales the service horizontally. But the high CPU was caused by a runaway analytics query, not by legitimate traffic. Adding more nodes does nothing because the load is not distributed — the query runs on a single node. Worse, the added nodes consume budget and create the illusion that the problem is being addressed. The root cause: the remediation system did not have metadata about what was running on the node, so it applied a generic fix to a specific problem.

Over-remediation. The system detects that response times are elevated and scales up. Costs rise. Then it detects that utilization is low (because of the over-scaling) and scales down. Response times rise again. The system oscillates between scaling up and down, never stabilizing. This happens when remediation lacks hysteresis — a minimum stability period after each action before the next action is permitted. The fix: require a cooldown window (15-30 minutes) after any scaling action before the system evaluates again.

Privilege escalation through automation. Remediation scripts need permissions to modify infrastructure: restart services, change configurations, scale resources, rotate credentials. If those permissions are too broad, a compromised remediation system becomes an attack vector. A script that can modify security group rules could be exploited to open network access. The mitigation: least-privilege permissions scoped to exactly the actions each remediation runbook needs, with separate credentials for each tier of remediation severity.

Measuring Remediation Maturity

Remediation maturity follows a four-level model. Each level builds on the capabilities of the previous one.

Level 1: Manual. An alert fires. A human reads a runbook, connects to the system, runs commands, and verifies the fix. MTTR depends entirely on human availability and expertise. This is the starting point for most organizations.

Level 2: Scripted. Known issues have automated scripts. A cron job cleans up disk space. A monitoring hook restarts a service when it detects a specific error pattern. The scripts are static — they run the same commands regardless of context. They work well for predictable, repeating issues but cannot handle novel failures.

Level 3: Context-aware. The remediation system queries metadata before acting. Before scaling a service, it checks: what tier is this service? What downstream systems depend on it? Is there a maintenance window in progress? The decision changes based on context — a Tier 1 production service gets immediate remediation; a Tier 3 development service gets a ticket filed for morning review. This level requires integration with a data catalog or service registry that provides ownership, classification, and dependency information.

Level 4: Predictive. The system identifies problems before they manifest. A disk usage trend predicts exhaustion in 48 hours; the system provisions additional storage now. A certificate expiration in 30 days triggers renewal today. A model predicts that tomorrow's traffic spike will exceed current capacity; the system pre-scales. Predictive remediation requires historical data, trend analysis, and high confidence in the prediction — acting on a false prediction wastes resources or causes unnecessary disruption.

Four metrics track remediation maturity across these levels. MTTR (mean time to resolution) should decrease as automation handles more issues. Remediation success rate measures how often automated fixes actually resolve the issue without human follow-up. Escalation rate tracks how often automation gives up and pages a human — a high rate means the automation is not capable enough for the issues it encounters. Cost per incident drops as automation replaces manual toil.

How Dawiso Supports Auto-Remediation

Remediation scripts need context to act safely. Before fixing a problem, the system should know: which team owns this data asset? What downstream processes depend on it? Is this asset governed under compliance policies that restrict automated changes? What is the blast radius if the remediation goes wrong?

Dawiso's data catalog provides the ownership and classification metadata that separates context-aware remediation (Level 3) from blind scripting (Level 2). Every data asset has a named owner, a classification tier, and tags that indicate regulatory sensitivity. When a remediation system detects a failing data pipeline, it queries the catalog to determine: is this a GDPR-regulated pipeline that requires human approval before any changes? Is it a Tier 1 asset feeding real-time dashboards, or a Tier 3 batch job that can wait?

Data lineage shows the blast radius. If the remediation system needs to restart a pipeline, lineage reveals every downstream consumer that will be affected. A pipeline that feeds three internal reports can be restarted with low risk. A pipeline that feeds regulatory submissions to the central bank needs a different escalation path entirely.

Through the Model Context Protocol (MCP), remediation tools query Dawiso's catalog programmatically before every action. The tool checks ownership, classification, dependencies, and current quality status — transforming remediation from "run this script and hope it helps" to "run this script because the metadata confirms it is safe and appropriate."

Auto-remediation without metadata is automation without judgment. Dawiso provides the judgment layer that makes remediation trustworthy.