Wednesday, 19th August 2020

Storage Cluster at RMA Region: Software Upgrade, scheduled 5 months ago

From Wednesday, 2020-08-19, 21:00 CEST, we will install a new release of our storage cluster software at our RMA (Rümlang) region. During this maintenance we expect short periods of slightly degraded performance of volumes (SSD and bulk storage) and our object storage. No further impact is expected. The storage cluster at our LPG region is not affected by this maintenance.

Date / Time
Wednesday, 2020-08-19, 21:00 CEST to Thursday, 2020-08-20, 03:00 CEST

Expected Impact
We expect short periods of slightly degraded SSD, bulk, and object storage performance at our RMA region during this maintenance.

We apologize for any inconvenience this may cause and thank you for your understanding.


Addendum (2020-08-24 18:55 CEST)
The new storage cluster software includes an improved mechanism for optimizing distribution of data within the storage cluster. As compensating for uneven data distribution between individual storage nodes and disks (aka cluster rebalancing) is something we have been routinely including in everyday system maintenance in the past already, we enabled the new mechanisms on Thursday afternoon following the scheduled maintenance.

However, the amount of data to be rebalanced in the process was much larger than expected, which led to further periods of degraded storage performance. Rebalancing of our NVMe SSD storage finally concluded on Saturday, 2020-08-22, at 21:16 CEST. After this point, no further impact on our NVMe SSD storage was observed.

Drawing from our previous experience, this optimization process was supposed to be a routine task running in the background. We apologize for any unforeseen impact it might have had on our customers’ systems in this instance.