Incident Report: Docker Service Degradation Due to Disk Exhaustion

Summary

The server experienced Docker service degradation caused by critical disk space exhaustion on the host system.

As available disk space dropped below a safe operational threshold (<5% free), Docker Desktop (WSL2 backend) failed to start reliably. When it did start, Docker Engine became unstable, Docker CLI commands hung intermittently, and container-to-container communication over Docker networks degraded.

The issue was traced to disk exhaustion within Docker Desktop’s WSL2 virtual disk (ext4.vhdx), which does not automatically shrink after data deletion.

---

Impact

• Docker Desktop failed to start initially
• docker ps, docker stats, and other CLI commands intermittently hung
• Containers entered inconsistent states (e.g., stuck in Created)
• Intermittent networking failures between containers on user-defined bridge networks
• Development and deployment workflows were blocked
• Estimated productivity impact: ~3 hours

---

Timeline

Detection (≈ 13:00)

• Docker Desktop failed to start
• No explicit error surfaced in UI

Observation (≈ 13:30)

• Docker CLI commands became unresponsive after partial startup
• Containers failed health checks

Investigation (≈ 14:15)

• Host disk usage identified at ~98%
• WSL2 Docker disk (ext4.vhdx) consuming majority of space
• docker system df showed significant accumulation of unused images, volumes, and build cache

Mitigation (≈ 15:00)

• Performed Docker cleanup
• Executed WSL shutdown and VHD optimization

Resolution (≈ 16:00)

• Docker services stabilized
• CLI responsiveness restored
• Container networking functioning normally

Overall Incident Duration: ~3 hours

---

Root Cause

The primary root cause was disk space exhaustion on the host system, specifically within Docker Desktop’s WSL2 virtual disk (ext4.vhdx).

When disk utilization exceeded safe thresholds:

• Overlay filesystem writes began failing
• Docker Engine operations blocked on I/O
• Container initialization stalled
• Network namespace creation intermittently failed

Since Docker Desktop (WSL2 backend) stores all Docker data inside a dynamically growing virtual disk that does not auto-shrink, space reclamation required explicit VHD optimization.

---

Contributing Factors

• Accumulation of unused Docker images, containers, volumes, and build cache
• No automated cleanup routine in place
• Docker Desktop (WSL2 backend) does not automatically shrink the virtual disk after deletions
• Lack of disk usage monitoring and alert thresholds
• No guardrail preventing Docker operation under critically low disk conditions

---

Mitigation & Resolution

Disk space was reclaimed and Docker storage was optimized using the following steps:

WSL2 Docker Disk Optimization

wsl --shutdown
Optimize-VHD -Path "$env:LOCALAPPDATA\Docker\wsl\data\ext4.vhdx" -Mode Full

Docker Cleanup Commands

docker system df
docker system prune -a --volumes
docker image prune -f

These actions reclaimed significant disk space and reduced the size of ext4.vhdx after compaction.

After cleanup and optimization:

• Docker Desktop started normally
• Docker CLI commands became consistently responsive
• Container networking stabilized
• No further I/O blocking observed

---

Preventive Actions

1. Implement Regular Docker Cleanup

Frequency: Weekly

Automation Plan:

Schedule automated execution of:

docker system prune -a --volumes
docker image prune -f

• Execute during off-peak hours
• Capture reclaimed space metrics

Notification:

• Send Slack notification on completion
• Include reclaimed disk space and execution duration
• Notify on both success and failure states

---

2. Monitor Host and Docker Disk Usage

Configure alerts at:

• 80% utilization (warning)
• 90% utilization (critical)

Monitor both:

• Host disk capacity
• ext4.vhdx file size growth

Trigger escalation if free space <10%.

---

3. Periodic WSL2 Virtual Disk Optimization

Frequency: Monthly

Automation Plan:

Optimize-VHD -Path "$env:LOCALAPPDATA\Docker\wsl\data\ext4.vhdx" -Mode Full

• Execute during scheduled maintenance windows
• Log before/after disk size

Alert & Notification:

• Slack notification on success or failure
• Alert if optimization does not reclaim expected space

---

4. Operational Guardrail

• Avoid running Docker services when disk utilization is near capacity
• Enforce minimum free disk threshold before builds or large image pulls
• Document disk capacity requirements for development environments

---

Lessons Learned

• Disk exhaustion manifests as Docker instability before explicit failure messages appear
• Docker cleanup does not automatically reduce WSL2 virtual disk size
• Developer environments require the same disk observability discipline as production systems
• Proactive disk monitoring would have reduced recovery time significantly

---

Status

Resolved – Docker services operating normally after disk cleanup and WSL2 disk optimization.