After the initial process recovered, the bview and update files since 2 October were produced successfully.
The incident trigger was a high volume of updates from one peer. This caused load in a single kafka partition. When that process restarted, it hit a latent bug where it tried to re-scan a large volume of data on startup. This scan repeatedly timed out, which prevented the process from making progress. Additionally, due to a misconfiguration in the code, this scan took an order of magnitude longer than it should. After fixing this configuration issue the process recovered.
We will work on a structural fix for the latent bug in the next sprint.
Posted Oct 06, 2026 - 08:44 CEST
Update
We continue to work on this issue. We have identified the root cause of the pipeline stall and are developing a workaround.
Posted Oct 05, 2026 - 14:46 CEST
Identified
At 1.30UTC, on October 2nd, 2026, a DE-CIX peer started sending an unusually high number of BGP messages to RRC12.
Our processing pipeline consuming from kafka did not keep up with this volume of updates, causing a process to time out. When this process restarts, it times out again before making enough progress.
We are now observing an impact on the production of other RRCs updates files, that are being delayed.
We are troubleshooting this issue.
Posted Oct 02, 2026 - 13:26 CEST
Investigating
The latest available MRT update file for RRC12 has a timestamp of 01:00 UTC, October 2nd, 2026. The latest available BVIEW file for RRC12 has a timestamp of 00:00 UTC , October 2nd, 2026. We are investigating.
Posted Oct 02, 2026 - 09:42 CEST
This incident affected: Non-Critical Services (RIS (Routing Information Service)).