degraded RIPEstat performance due to maintenance preparations

Incident Report for RIPE NCC

Resolved

We have decided to postpone the move of our largest datasets (HBase) to a new environment by one month. Due to the complexity of this migration, we wanted to ensure that RIPEstat would continue operating, and did not feel confident going ahead when this could result in the service being severely disrupted.

We plan to publish a RIPE Labs article with more details soon, including which historical data will be delayed when the migration is fully completed.

You can read the announcement at:
https://mailman.ripe.net/archives/list/[email protected]/thread/4CMKBRORQ5WTDCTTN3WHBEUCOJZEZTPV/
Posted Sep 26, 2024 - 11:11 CEST

Update

We are unable to export historical data for four of our data sets, out of a total of twenty. As a result, we will not be able to provide aggregated historical data for the following API calls and their corresponding widgets for some time after we complete our migration. The migration for the remaining data is ongoing:

* as-path-length
* as-routing-consistency
* asn-neighbours
* asn-neighbours-history
* bgp-update-activity
* bgp-state
* bgp-updates
* bgplay
* ris-peerings
* routing-status
* visibility


A more detailed statement is available at: https://mailman.ripe.net/archives/list/[email protected]/thread/I2IXTP47JLMFSI4VFLPQP5DYBX3WWP7P/
Posted Sep 23, 2024 - 17:42 CEST

Monitoring

We continue to face both software issues and degraded hardware performance during the data export from our HBase database. However, application error rates have returned to normal levels.
Posted Sep 20, 2024 - 18:33 CEST

Update

We continue to face issues during the export of data.
Posted Sep 19, 2024 - 17:34 CEST

Identified

The performance of multiple RIPEstat data sources is currently degraded due to maintenance preparations. We are exporting RIS-related datasets from our database (HBase) to prepare for a migration to a new environment. This export causes a high load on our production environment that noticeably impacts performance.

We apologise for the inconvenience this may cause as the preparation continues until the end of the month when we plan to have all datasets migrated.
Posted Sep 18, 2024 - 11:14 CEST
This incident affected: Non-Critical Services (RIPEstat).