Gcore - Streaming | Live RTMP Ingestion Retrospective Incident Details – Incident details

Incident Postmortem: RTMP Ingress Initialization Degradation

Incident Window: October 20, 2026, 18:15 UTC – 23:20 UTC

Impacted Services: New RTMP Stream Ingestion & Stream Recording

Unaffected Services: Active Live RTMP Streams, All SRT Ingestion, Edge Transcoding, Global CDN Delivery

Executive Summary
On October 20, 2026, we updated perimeter security policies across our media ingest gateways to shield customer streaming infrastructure from aggressive automated AI scrapers and bot scanning. A configuration discrepancy caused ingress bandwidth controls to saturate, preventing newly establishing RTMP streams from completing media data transfer after the initial handshake.
All pre-existing live broadcasts, the entire SRT pipeline, and downstream viewer delivery remained completely uninterrupted. To protect active customer streams from sudden disconnects, engineering avoided disruptive system-wide restarts, deploying isolated traffic pathways and non-destructive reroutes across North America and Europe until normal operations were safely confirmed and restored.

Customer Impact

• New RTMP Broadcasts: Publishers establishing new connections completed network and protocol handshakes normally, but experienced media transmission stalls prior to first-frame delivery.

• Recording Services: Automated recording routines dependent on initial media frame arrival did not trigger for affected new sessions.

• Ongoing Live Streams: Zero disruption or frame drops across all live broadcasts initiated prior to the event.

• Viewer Delivery (CDN): Global egress caching and playback distribution operated with 100% availability.

Incident Timeline (UTC)

• 2026-10-20 18:15 — Updated transport protection and gateway validation policies take effect across global ingress regions.

• 2026-10-20 19:00 — Initial reports received regarding new publishing connection stalls.

• 2026-10-20 19:01 — Technical operations and platform engineering initiate multi-region diagnostic triage.

• 2026-10-20 22:40 — Root mechanism identified within perimeter policy quota calculations; conflicting validation parameters immediately disabled. Residual ingress backlog begins draining.

• 2026-10-20 23:20 — Isolated ingress routing pathways brought online for North American regions, enabling new RTMP streams to bypass legacy ingress backlogs.

Root Cause Analysis
While deploying hardened edge rate-limiting to protect customer endpoints against automated vulnerability scanning and AI bot activity, an algorithmic budget miscalculation occurred. High-frequency synthetic boundary verification traffic was unintentionally tallied against public ingress resource limits.

Once this shared quota was reached:

1. Standard network handshakes (TCP and RTMP negotiation) succeeded normally.

2. Ingress queues severely throttled subsequent media packet ingress.

3. Because connections remained open rather than returning a connection drop or socket error, publishing encoders remained paused waiting for transfer clearance, delaying standard client-side automatic retries.

Standard socket-level uptime checks continued reporting endpoints as fully operational because initial handshakes completed successfully, obscuring the payload-level starvation.