Update - We are continuing to work on solving this issue. We have taken some measures that have improved matters and reduced the amount of delays, but we are still investigating the root cause and need to arrive at a longer-term solution. We will update when we have resolved this issue.
Sep 22, 2023 - 14:37 UTC
Update - We have finished adding capacity to HBase. Results processing has restarted at 17:20 UTC. We will continue to monitor the cluster.
Sep 20, 2023 - 19:17 UTC
Update - We have been seeing repeated crashes of nodes in the HBase backend that is used to store Atlas measurement results. Yesterday we have increased the amount of memory allocated to HBase. Since 21:00 UTC processing proceeded again, and processing delays were significantly reduced. Around 11:00 UTC today we started seeing several more crashing nodes, and delays are increasing again.
We are working on allocating more memory to HBase. In addition, we are adding more nodes to the cluster to spread the load.
Sep 20, 2023 - 12:48 UTC
Identified - We have made some adjustments to the cluster configuration, and delays are slowly decreasing.
Sep 20, 2023 - 06:15 UTC
Investigating - We are experiencing issues with the RIPE Atlas backend and currently are investigating this issue.
Sep 18, 2023 - 14:07 UTC