Root Cause Analysis
Issue
On March 4, 2026, a storage capacity condition in one of the storage clusters within the Amsterdam region resulted in temporary write operations being blocked. This caused virtual machines relying on the affected storage system to experience disk I/O failures and service disruption.
The issue occurred during a maintenance activity involving a storage platform upgrade. While the upgrade completed successfully, a combination of background data redistribution and unusually high write activity caused storage utilization to exceed a critical threshold, triggering a protective mechanism that temporarily blocked new write operations.
Service functionality was restored after emergency actions were taken to reclaim storage capacity.
Timeline (UTC)
March 4, 2026 – 15:50 : Storage platform upgrade completes and background data redistribution begins
March 4, 2026 – 16:20 : Storage utilization reaches critical threshold; cluster transitions to write-blocking state
March 4, 2026 – 16:21 : Monitoring system becomes unavailable as dependent virtual machines experience storage errors
March 4, 2026 – 16:23 : Multiple customer reports indicate widespread virtual machine availability issues
March 4, 2026 – 17:25 : Investigation identifies storage capacity exhaustion as the underlying cause
March 4, 2026 – 17:28 : Storage engineering team initiates emergency capacity recovery procedure
March 4, 2026 – 17:40 : Write operations restored and affected virtual machines begin recovering
March 4, 2026 – 18:00 : Platform functionality verified through system tests
March 4, 2026 – 18:59 : Remaining virtual machines recovered and incident closed
Root-cause
The incident occurred due to a temporary spike in storage utilization caused by two simultaneous conditions. After a storage system restart during maintenance, an automated data redistribution process began, temporarily increasing storage usage while data was being rebalanced across nodes.
At the same time, unusually high write activity from a workload consumed additional storage capacity. Because the cluster was already operating at relatively high utilization, the combined effect pushed storage usage beyond the system’s safety threshold. As a result, the storage platform automatically blocked write operations to protect data integrity, causing virtual machines using the affected storage to experience disk I/O failures.
Action-Items
Implement a mandatory pre-maintenance validation step to ensure automated data redistribution mechanisms are disabled or controlled during storage service restarts or upgrades.
Introduce capacity safeguards in maintenance procedures to prevent upgrades from being executed when storage utilization exceeds safe operational thresholds.
Increase available storage capacity within the affected region to maintain sufficient headroom for background data operations and workload spikes.
Review operational procedures for scheduling maintenance activities to ensure adequate system capacity and visibility during upgrades.
