Summary: The network experienced significant disruptions due to downed links across several locations in Mexico. The resultant isolation of network equipment led to widespread service impacts.
Details:
Notifications were made to the fiber supplier to initiate the investigation and address the connectivity issues.
Backbone services reported multiple impacts, with subsequent events causing additional isolation of routers within the network.
Management teams were engaged in troubleshooting efforts; however, a clear resolution timeline was not immediately established.
Field engineers were dispatched to address faults at two critical locations. One location encountered extended delays due to severe damage to the local sewage system, necessitating extensive repairs before restoration could commence.
Restoration efforts were executed in both San Luis Potosí and Aguascalientes, with crews working diligently to reinstate services.
Resolution:
Progressive restoration of services was noted, and ongoing discussions with the fiber supplier were conducted to ensure the reliability and permanency of the implemented fix.
Posted Dec 12, 2024 - 08:20 UTC
Resolved
We'd like to inform you that the issue has been resolved, and we are closely monitoring the performance to ensure there are no further disruptions. We will provide a Root Cause Analysis (RCA) report in the coming days to help you understand what caused the incident and the steps we have taken to prevent it from happening again in the future.
We apologize for any inconvenience this may have caused you, and want to thank you for your patience and understanding throughout this process.
Posted Nov 22, 2024 - 07:48 UTC
Monitoring
We are pleased to inform you that our engineering team has implemented a fix to resolve full outage in SQR location. However, we are still closely monitoring the situation to ensure stable performance.
We will provide you with an update as soon as we have confirmed that the issue has been completely resolved.
Posted Nov 22, 2024 - 07:41 UTC
Investigating
We are currently experiencing a degradation in performance, which may result in service unavailability. We apologize for any inconvenience this may cause and appreciate your patience and understanding during this time.
We will provide you with an update as soon as we have more information on the progress of the resolution. Thank you for your understanding and cooperation.
Posted Nov 22, 2024 - 03:47 UTC
This incident affected: Hosting (Santiago de Querétaro (SQR)).