All systems are operational

Past Incidents

Saturday, 9th November 2019

Network Infrastructure (RMA) Connectivity Issues

Since last night's early network upgrade maintenance and subsequent upgrade of the firmware back to the version we have been using right from the start, the errors did not show up in the log files anymore. We do not believe that the root cause has been found or fixed, yet, but we are confident that the situation will remain stable for now. We will continue to find the root cause in close collaboration with the vendor and provide you with an update once further information is available.

Network Infrastructure (RMA) Connectivity Issues

Upgrading back to the firmware release we were using initially was successful. Together with the vendor we were able to collect performance data that will be analyzed during the next few hours. Based on that data the vendor will try to set up a reproduction in the lab. We do not expect an update before Saturday evening CET.

Network Infrastructure (RMA) Connectivity Issues

After a long and in-depth discussion with the vendor and because the issues still persist, we have decided to go back to the firmware release that we have been using right from the start and continue debugging this specific release. Unfortunately, the emergency upgrade will once more lead to short periods of packet loss. We are in permanent contact with the vendor and are monitoring the situation closely.

We are sorry for the inconvenience this may cause.

Friday, 8th November 2019

Network Infrastructure (RMA) Connectivity Issues

The downgrade of the firmware on all switches in location RMA1 was successful. However, CPU utilization is still at a significantly high level. We are in touch with the engineering and development team of the vendor in order to reproduce the issue in their or our lab. We will follow up with an update once available.

Network Infrastructure (RMA) Connectivity Issues

The vendor suggested to downgrade the firmware on all switches in order to try and mitigate the issue while they are still working on a local repro. We will start with the downgrade now after having validated it in the lab. We will keep you posted.

Network Infrastructure (RMA) Connectivity Issues

We were able to establish an emergency call and are now discussing options. Unfortunately, further packet loss is expected until we have a workaround in place.

Network Infrastructure (RMA) Connectivity Issues

We are currently facing partial packet loss again and are trying to get the vendor on an emergency call. We will keep you posted.

Early Network Upgrade: Partial Migration to New Network Hardware, scheduled 1 month ago

Tonight, in an early scheduled maintenance, we will try to further stabilize the network by live-migrating a subset of the virtual servers at cloudscale.ch to compute nodes that are attached to new network equipment. It is possible that your virtual servers will be live-migrated multiple times during this maintenance work. You may also experience short periods of packet loss (up to 1-2 minutes) or higher RTTs for connections from and towards the Internet as well as between virtual servers (even when using the private network interface).

Date / Time
Friday, 2019-11-08, 21:00 CET to Saturday, 2019-11-09, 03:00 CET

Expected Impact
Short periods of packet loss (up to 1-2 minutes) or higher RTTs for connections towards the Internet as well as between virtual servers.

We apologize for any inconvenience this may cause and thank you for your understanding.

Thursday, 7th November 2019

Network Infrastructure (RMA) Connectivity Issues

After performing a controlled reboot of all switches, the situation could be stabilized. We have raised a high-priority case with our vendor and it already got management awareness there.

The issue we are facing seems to be triggered by high CPU load on the switches themselves, which leads to instabilities in control plane traffic. We have relaxed several control plane timers hoping to mitigate the issue for the time being. However, the decrease in CPU utilization was only minimal.

We will continue investigating this issue together with the vendor and keep you up to date on this status page.

Please accept our sincere apologies and rest assured that we treat this case with the utmost priority.

Network Infrastructure (RMA) Connectivity Issues

After collecting support data together with the vendor, we are currently facing partial packet loss between virtual servers and to/from the Internet. We are investigating and will keep you updated.

We are sorry for the inconvenience this may cause.

Wednesday, 6th November 2019

Network: High CPU Issue (Emergency Maintenance Window), scheduled 1 month ago

After successful tests in our lab, we will deploy a fix for a high CPU issue that we have been facing on our new networking equipment during the last couple of days. During this maintenance work (lasting from 23:00 CET to 23:59 CET) you may experience short periods of packet loss (up to 1-2 minutes) or higher RTTs for connections from and towards the Internet as well as between your virtual servers.

We apologize for any inconvenience this may cause and thank you for your understanding.

Cloud Control Panel and API: Major Update, scheduled 1 month ago

On Wednesday morning, 2019-11-06, we will deploy a new major version of our cloud control panel and the API. This involves a migration of the existing data, during which our cloud control panel and the API will be unavailable. No impact on already running servers or other services is expected.

Date / Time
Wednesday, 2019-11-06, 10:00 to 13:00 CET

Expected Impact
We expect several short service interruptions of our cloud control panel and the API during this maintenance window.

We apologize for any inconvenience this may cause and thank you for your understanding.

Tuesday, 5th November 2019

No incidents reported

Monday, 4th November 2019

Network Upgrade: Partial Migration to New Network Hardware, scheduled 1 month ago

In the evening and night from Monday, 2019-11-04, to Tuesday, 2019-11-05, we will live-migrate a subset of the virtual servers at cloudscale.ch to compute nodes that are attached to new network equipment. It is possible that your virtual servers will be live-migrated multiple times during this maintenance work. You may also experience short periods of packet loss (up to 1-2 minutes) or higher RTTs for connections from and towards the Internet as well as between virtual servers (even when using the private network interface).

Date / Time
Monday, 2019-11-04, 20:00 CET to Tuesday, 2019-11-05, 03:00 CET

Expected Impact
Short periods of packet loss (up to 1-2 minutes) or higher RTTs for connections towards the Internet as well as between virtual servers.

We apologize for any inconvenience this may cause and thank you for your understanding.

Sunday, 3rd November 2019

No incidents reported