RIS collector rrc15 is unavailable

Incident Report for RIPE NCC

Postmortem

One process on our collectors has a slow memory leak that gradually consumes system resources. When configuration management applied changes, this exhausted system resources. In this case, the system became completely unavailable when resources were exhausted.

We use PyPy instead of Python 3 for this process. We recently evaluated Python 3, and confirmed it reduces memory usage for our processes. While we had already planned a migration to Python3 to reduce memory consumption, we had only deployed the fix to the most memory constrained collectors.

We will move forward this transition to Python 3 on all RRCs.

Posted Jul 22, 2025 - 10:13 CEST

Resolved

The collector is available again. No BGP messages were stored between 04:05 UTC and 06:30 UTC.
Posted Jul 22, 2025 - 08:41 CEST

Investigating

One of the route collectors, rrc15, is unavailable at the moment. No BGP updates were processed for this collector since ~04:00 UTC. We are investigating the issue.
Posted Jul 22, 2025 - 08:08 CEST
This incident affected: Non-Critical Services (RIS (Routing Information Service)).