Mitigation of CVE-2023-20593 (Zenbleed) from 2023-07-24 23:33 CEST to 2023-07-25 01:33 CEST

All compute hosts in both our RMA1 (Rümlang) and LPG1 (Lupfig) zones have been patched successfully and are now running the new CPU microcode version which includes a mitigation for CVE-2023-20593 (Zenbleed).

This concludes the emergency maintenance.

All the compute hosts in zone RMA1 (Rümlang) have been patched successfully.

We will now start rolling out the new microcode version in our LPG1 (Lupfig) zone as well.

Emergency maintenance

We are performing a CPU microcode update on our compute hosts in both our RMA1 (Rümlang) and LPG1 (Lupfig) zones. The new microcode version fixes a 0-day vulnerability which has been disclosed recently. Due to the criticality of this vulnerability, we decided to immediately start patching our systems without informing our customers in advance.

Expected impact

There is some risk of a negative performance impact that might come with the new microcode. We suggest to closely monitor your workloads and scale your virtual servers if necessary.

Timeline

Earlier today, we have learned of a 0-day vulnerability affecting AMD CPUs based on the Zen2 architecture. The exploit allows an attacker to read the contents of CPU registers of other virtual servers as well as the compute host itself and therefore requires immediate mitigation. More information regarding this vulnerability can be found here: https://lock.cmpxchg8b.com/zenbleed.html

Immediately after reproducing the exploit in our environment, our engineers started testing different approaches to mitigate the vulnerability.

By 22:00 CEST, a subset of our lab servers were patched with a new microcode version and we started running our acceptance test suite to confirm stable operations on the patched hypervisors.

Once we were able to confirm that the patch does not negatively affect the stability of our hypervisors, we immediately started and are still engaged in rolling out the mitigation in our RMA1 (Rümlang) zone.

Due to the criticality of this vulnerability, we will seamlessly transition to rolling out the new microcode version in our LPG1 (Lupfig) zone as well.

We will keep you posted by updating this incident ticket and apologize for any inconvenience this situation may cause. Thank you for your understanding.