Overview

On February 6, 2024, our Content Delivery Network (CDN) experienced partial delivery performance degradation in Korea. This incident was primarily caused by a bug in the DDoS protection module.

Incident Timeline

• February 6, 2024, 16:26 (UTC): Initiation of the DDoS attack, leading to the overloading of edge servers at the Korean datacenter.
• February 6, 2024, 16:33 (UTC): The engineering team initiated an investigation into the issue.
• February 6, 2024, 17:10 (UTC): A problem with the DDoS protection module was identified as a contributing factor.
• February 6, 2024, 17:35 (UTC): Preparation of a fix for the identified bug in the DDoS protection module.
• February 6, 2024, 17:53 (UTC): Implementation of the fix, resolving the CPU usage issue at the Korean datacenter.

Root Cause Analysis

Excessive CPU utilization on the CDN edge servers located in the Korean datacenter. This was the result of a bug in the DDoS protection module, which led to heightened CPU consumption during the attack, thereby severely affecting our delivery services.

Impact

This incident led to intermittent 5xx errors and timeouts affecting CDN delivery within Korea, persisting for 1 hour and 27 minutes.

Action items

In response to this incident and to bolster our defense against future DDoS attacks, we are implementing the following action items:

• Performance Testing Enhancements: introduce additional performance tests to preemptively identify and rectify similar issues.
• Infrastructure Strengthening: expand the CDN edge infrastructure within the Korean datacenter, thereby enhancing our capacity to manage escalated DDoS attack scenarios.
• Balancer Optimization: refine the settings of the CDN balancer to more efficiently redistribute traffic away from the datacenter during DDoS attacks and periods of excessive demand.

We are committed to continuously improving our systems and processes to ensure high availability and performance of our CDN services, particularly in the face of evolving cybersecurity threats.