We are continuing to work on solving this issue. We have taken some measures that have improved matters and reduced the amount of delays, but we are still investigating the root cause and need to arrive at a longer-term solution. We will update when we have resolved this issue.
Posted Sep 22, 2023 - 14:37 UTC
We have finished adding capacity to HBase. Results processing has restarted at 17:20 UTC. We will continue to monitor the cluster.
Posted Sep 20, 2023 - 19:17 UTC
We have been seeing repeated crashes of nodes in the HBase backend that is used to store Atlas measurement results. Yesterday we have increased the amount of memory allocated to HBase. Since 21:00 UTC processing proceeded again, and processing delays were significantly reduced. Around 11:00 UTC today we started seeing several more crashing nodes, and delays are increasing again.
We are working on allocating more memory to HBase. In addition, we are adding more nodes to the cluster to spread the load.
Posted Sep 20, 2023 - 12:48 UTC
We have made some adjustments to the cluster configuration, and delays are slowly decreasing.
Posted Sep 20, 2023 - 06:15 UTC
We are experiencing issues with the RIPE Atlas backend and currently are investigating this issue.
Posted Sep 18, 2023 - 14:07 UTC
This incident affects: Non-Critical Services (RIPE Atlas).