Spectre and Meltdown – the patching of the leaks to be more specific – occupied the mind and the to do list of every system administrator the last couple of days. Now that the dust has settled and the most important patches are in place, we can try to estimate the real impact for cloud providers such as Nucleus.
Was this the biggest/most important patch ever?
There is a comparison with Heartbleed a few years ago. To be brief: everything that has a CPU needs a software- and/or firmware-upgrade. That means both hypervisors and virtual machines. So yes, this was one of the biggest/most important patches we ever did.
Was this the most dangerous leak ever?
In terms of risk, Heartbleed was worse. Spectre and Meltdown are hard to exploit remotely: you have to have access or be able to make a user execute a certain command (click a link, visit a website, etc.). You can’t just scan for servers that are not yet patched and then use the exploit remotely. That means the danger is not as imminent (read: we are not saying there is no danger) since it is limited to persons and machines that already have access to a server. Still dangerous, but less than Heartbleed was. With Heartbleed, we saw remote scanning for vulnerable servers hours after the leak was published. That is not the case now.
How difficult is it to patch the leak?
Installing patches is not that hard, but Spectre and Meltdown came with their own problems.
- Not all vendors had patches yet
Because the embargo was lifted earlier than planned, some vendors had to come up with quick patches that to be replaced by new patches a few days later (“Oops, we forgot a few things”). What meant two rounds of patching for us.
- EVERYTHING had to be patched
In a virtual environment there are three sorts of patches: BIOS upgrades, hypervisors (KVM/VMware) and the operating system in the virtual machines (VMs). On top of that, a regular kernel update & reboot in VMs was not sufficient. The VM had to get a “cold boot”: turn the VM in VMware off and on to load the microcode in the virtual CPU. That obviously makes remote patching a lot harder and more time consuming.
Are the performance loss numbers that were mentioned correct?
There were mentions of a performance loss of 5 to 30 percent. We currently measure performance losses that are closer to 5 than to 30 percent. Servers that are impacted most severely are database servers: MySQL/MariaDB & PostgreSQL. On those, we measure CPU increases of 10 to 20 percent. For webservers it’s harder to say, but we think performance loss might not be as bad as was feared.
Which of both leaks is most dangerous?
Meltdown was the most dangerous one. That’s why all of the attention went to that one at first. Spectre is harder to exploit, but we’re sure that it won’t take long for public exploits to emerge as well. In the end these are still local attacks: you need to have access to a server before you can use the exploit. And it’s read-only, meaning you can read information in memories you are not allowed to access, but you can’t manipulate the data.