Whois Update Service Incident
Incident Report for RIPE NCC
Postmortem

When:
Between approximately 20:00 UTC on Sunday, 16 June and 01:00 UTC on Monday 17 June, there was an intermittent interruption to the Whois Update service. On 24 June, between 08:03 and 08:21 UTC there was a subsequent incident.

Why:
The interruption occurred due to several unusual update messages containing many maintainers. Aside from this, there were a number of spurious update notification emails sent to uninvolved maintainers. Bounced mail messages from those maintainers were processed as updates which caused a lot more emails to be generated.

Action:

At the time of the incidents, we took measures to mitigate the impact to our users.

On 24 June, we deployed the 1.112.1 release to production to mitigate this issue (validate max references). On 1 July, another release of whois 1.113.2 was made which improved our handling of multipart/mixed bounced and auto-submitted messages.

The release notes are on the website:
https://apps.db.ripe.net/docs/Release-Notes/#ripe-database-release-1-113-2
 
In the following weeks, we continued to monitor the whois updates and we did not observe any further instances since 24 June.
 
We apologise to our users for the inconvenience this interruption may have caused.

Posted Jul 31, 2024 - 09:00 UTC

Resolved
Between approximately 20:00 UTC on Sunday 16th June and 01:00 UTC on Monday 17th June, there was an intermittent interruption to the Whois Update service, due to a number of unusual update messages. In addition, many spurious update notification emails were generated to uninvolved maintainers. We have taken steps to mitigate the impact of these updates, and will publish a more detailed post-mortem later.
Posted Jun 16, 2024 - 20:00 UTC