At approximately 09:20 UTC, the RPKI RRDP repositories were temporarily unavailable for about 20 minutes. This caused a number of RPKI validators to fall back to retrieving RPKI content via rsync. The RPKI team was alerted almost immediately about the issue and was able to restore the RRDP repository service quickly. No data was lost as a result of this incident.
The RPKI dashboard was also briefly affected, preventing users from making changes to their ROAs during the incident.
This incident occurred during patching. Normally, patching causes no downtime, as nodes that are in maintenance are not used by our loadbalancer. However, this time the loadbalancer did not drop the connections to these nodes, resulting in traffic being sent to a node that is unavailable.
The root cause of the issue appeared to be that the loadbalancer did not terminate TCP connections once a node was declared offline. We have a work-around for this issue that can be used when patching, and are investigating a permanent solution.
Apologies for any inconvenience this has caused.