Some systems are experiencing issues

Past Incidents

Friday, 22nd November 2019

Network Infrastructure Network Issues

Situation is stabilizing. Several services need to be taken care of due to split brain situations. We will keep you posted.

Network Infrastructure Network Issues

In order to ease the load on the CPU we have disabled a control protocol (BFD) completely hoping to stabilize the situation again. Investigating.

Network Infrastructure Network Issues

We are facing a complete outage of the network in RMA1. Investigating.

Network Infrastructure Network Issues

We are currently facing partial packet loss again and are investigating the situation. We will keep you posted.

Thursday, 21st November 2019

No incidents reported

Wednesday, 20th November 2019

Network Infrastructure Connectivity Issues

Unfortunately, the workaround provided by the vendor has only had a short-term effect. CPU utilization on the network devices is back at the (high) level it was before. We are continuing to investigate what is causing this and are also looking into other options, such as replacing the vendor or moving the routing to different equipment.

Network Infrastructure Connectivity Issues: Workaround Deployed

We have deployed the workaround to production without further impact to customer traffic. CPU utilization has decreased significantly and is continuing to decrease as we monitor the situation.

We will continue investigating this issue together with the vendor and keep you up to date on this status page.

Please accept our sincere apologies and rest assured that we will continue to treat this case with the utmost priority.

Network Infrastructure Connectivity Issues: Potential Workaround

We have received another potential workaround from the vendor that has been tested in their lab. Given the situation we are in right now, we have decided to deploy the workaround in an emergency maintenance window immediately.

In case we encounter any unexpected issues we should be able to roll back the changes in a matter of minutes.

Network Infrastructure Connectivity Issues

We were able to establish an emergency call with the vendor and are now discussing options. Unfortunately, further packet loss is expected until we have a workaround in place.

Network Infrastructure Connectivity Issues

We are currently facing partial packet loss again and are trying to get the vendor on another emergency call. We will keep you posted.

Cloud Control Nodes (RMA1): Operating System Security Updates, scheduled 2 days ago

On Wednesday 2019-11-20, from 08:00 CET to 13:00 CET we will perform security updates on our cloud control nodes. Thanks to the redundant setup of these nodes no customer impact is expected. Our Cloud Control Panel will remain active during this maintenance work. Already running virtual servers will not be affected by this work.

Date / Time
Wednesday 2019-11-20, 08:00 CET to 13:00 CET

Expected Impact
Thanks to the redundant setup no impact is expected. Short periods of degraded performance of our Cloud Control Panel may be observed. Already running virtual servers are not affected by this work as the compute nodes running these servers and the network infrastructure are not changed.

We apologize for any inconvenience this may cause and thank you for your understanding.

Cloud Compute Nodes (RMA1): Security Updates to Address iTLB Multihit Vulnerability, scheduled 2 days ago

From Wednesday 2019-11-20 to Friday 2019-11-22 we will perform security updates on our cloud compute nodes at RMA1 to address the iTLB multihit vulnerability. At the same time we will also install additional minor operating system updates. All running virtual servers will be live-migrated to other compute nodes at least once during this maintenance work. This is a standard procedure during compute node maintenance.

Our hardware platform is not affected by the recently announced TAA TSX Asynchronous Abort vulnerability.

Date / Time
Wednesday 2019-11-20, 08:00 CET to Friday 2019-11-22 24:00 CET

Expected Impact
All running virtual servers will be live-migrated to other compute nodes prior to the installation of the updates on those nodes. However, it is possible that you may experience degraded server performance and/or short interruptions of network connectivity during live-migration. Servers running at our recently opened second location LPG1 are not affected.

We apologize for any inconvenience this may cause and thank you for your understanding.

Tuesday, 19th November 2019

No incidents reported

Monday, 18th November 2019

No incidents reported

Sunday, 17th November 2019

No incidents reported

Saturday, 16th November 2019

No incidents reported