On 7 January 2026 at 15:11 UTC, all objects were removed from both our RRDP and rsync RPKI repositories. At 15:20 UTC, the repository state was restored, and all objects were back.
At 15:08 UTC, we did a production deployment of our CA software on the first application node. We deploy one node at a time so that we have zero downtime during deployments. We carried out a database DDL change on the table storing published objects, which took roughly 40 seconds to execute. During those 40 seconds, the online node queried the published objects in order to publish them. This transaction had to wait for the table lock held by the DDL change to be released. When the lock was released, PostgreSQL returned zero rows. We learned that this is a known caveat [1]: after rewrite commits, the table will appear empty to concurrent transactions.
We will implement circuit breakers between all phases of our publication process to prevent this situation from happening in the future.
[1] - https://www.postgresql.org/docs/18/mvcc-caveats.html
Apologies for the inconvenience this has caused.