At 02:20 UTC, the RPKI core systems and Publication as a Service (PaaS) were unavailable. Around 06:00 UTC, Publication as a Service was fully operational again. Any changes made during the outage are lost. Since these are changes from automated systems, recovery should be automatic.
Both the RPKI core and PaaS were unavailable due to an issue with an NFS volume share.
The RPKI on-call engineer first began troubleshooting the RPKI core issue.
Subsequently, as the engineer started troubleshooting the PaaS, the backup was unreachable, which meant restoring it directly on the filesystem was not a possible workaround. At 04:00 UTC, the owners of the NFS service were contacted, and they started investigating the unavailability of the NFS. Around 06:00 UTC, the service was restored, and Publication as a Service was operational again.
During the outage of the NFS service, the Krill instance that publishes objects for our PaaS had no data, and therefore, a custom startup script mistakenly initialized an empty repository. This caused the RRDP and rsync repositories for PAAS to temporarily contain no objects.