RPKI CA and Publication as a Service unavailable

Incident Report for RIPE NCC

Postmortem

At 10:29 UTC, the RPKI core systems and Publication as a Service (PaaS) were unavailable. Around 11:18 UTC, both systems were fully operational again.

Any changes made during the outage on the Publicaion as a Service are lost. Since these are changes from automated systems, recovery should be automatic.

Due to an issue with an NFS volume share, both RPKI CA and PaaS were unavailable.
The RPKI on-call engineer first began troubleshooting the RPKI CA issue, hence why troubleshooting the issue with the Publication as a Service (PAAS) was delayed. Unlike the issue with RPKI CA systems, the backup of the PaaS service was also unreachable, which meant restoring directly on the filesystem was not a possible workaround. Around 11:18 UTC, the service was restored, and Publication as a Service was again operational.

Following this event, the RPKI team has planned effort to move away from using NFS storage and implement a more stable solution.

Posted Jul 29, 2025 - 15:42 CEST

Resolved

This incident has been resolved.
Posted Jul 29, 2025 - 13:18 CEST

Update

We are continuing to monitor for any further issues.
Posted Jul 29, 2025 - 13:10 CEST

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Jul 29, 2025 - 13:03 CEST

Investigating

We are currently investigating this issue.
Posted Jul 29, 2025 - 12:50 CEST
This incident affected: RPKI (RPKI Dashboard, Publication as a Service (API endpoints)).