Virtuozzo 7 - Meltdown/Spectre Patch Performance desaster

Latest response

Hey there,
I know it's about Virtuozzo 7 (already got Support Ticket #18222 because of this) and not RHEL directly - but as Virtuozzo is built on top of RHEL 7 - and the Meltdown/Spectre Patches are those from RHEL - I want hear if anyone else also observed similiar Performance Problems with those Patches.

Since Meltdown/Spectre Patches the Performance dropped to unuseable levels. I patched one of our Root Servers which is running 1 (ONE) productive Container with EZ CMS (Apache 2.4.6, PHP 5.6.32) and MySQL/MariaDB DB (5.5.56) to latest VZ Kernel (3.10.0-693.11.6.vz7.40.4)
Root Server is HPE Gen9 Blade Server (Xeon CPU E5-2640 v3 @ 2.60GHz), Storage is Virtuozzo Storage running on SSD only (1-2GB/s Performance) - so rather good Hardware Specs ... ;-)

So here what happened when I bootet to patched Kernel:
Load AVG

completely unusable ... Load AVG spiked up to 150 and more (peaks up to over 200)

Disabling the Security Patches brings the Load down to normal:
Quote:

tee /sys/kernel/debug/x86/*enabled <<< 0

Answer from Virtuozzo Support:
Virtuozzo SupportVirtuozzo Support

Essentially this means I can either patch Virtuozzo against Spectre and cripple the Performance that much that the Server is unuseable - or I decide to not patch the Server - keep good Performance but stay vulnerable to Spectre ...
Both options not really satisfactory ...

Anyone else observed similiar issues with those Patches? Or got a good Tip for me? ;-)

thx, bye from sunny Austria
Andreas Schnederle-Wagner

Responses

Hi Andreas,

The information you've provided are more or less already known facts - the performance impact depends extremely on the individual setup and usage. You can find comprehensive information and all instructions from Red Hat in the knowledebase article Kernel Side-Channel Attacks - CVE-2017-5754 CVE-2017-5753 CVE-2017-5715. Fighting the security flaw, especially concerning Spectre is not a trivial thing and thus a work in progress which will take some time. As of now we are not aware of any exploits regarding Meltdown and Spectre. In the end it is your own decision to disable the mitigations when they make your working environment unusable, until improved software (kernel, microcode etc.) updates are available.

Regards,
Christian

Hi Christian,

I read every KB Article / Whitepaper I found the last Days. Most of them speak about Performance degradation of 5 - 30% - but here we speak about a rise in Load Average from 5-8 up to 160-200 LOAD AVG - on a System running fairly widespread Software --> Apache, PHP, MySQL - which means absolutely unuseable System which leaves one with either patch System to unuseable State or not patch System at all leaving it vulernable ... both variants in themselves indiscutable

I could live with 5-30% Perf degradation ... but this massive degradation brings our whole Hosting Business into massive Troubles ... I know ... not much that can be done here right now ... but I thought such massive Performance degradation (way above what is stated anywhere) is worth mentioning ...

Andreas

Hi Andreas,

I have read about similar cases on the internet in the last days and I absolutely can understand your frustration.
And yes, you're right to report the huge impact. But what does it help ? We all are victims in a way - aren't we ?

Cheers :)
Christian

Hi Andreas and the others in the forum,

We all offer our sympathies for your problem. You are in a terrible state of making two bad decisions, no mater which way you go.

However, your reporting and public awareness do help because eventually:

a) Businesses will turn away from vendors who do not show much higher duty of care and transparency as soon as viable alternative happens.

b) A money-rich or powerful organisation will take vendors in a large-scale legal case for mistakes like this one of these days. It is just a matter of time and the question of who and where.

c) More comprehensive testing of "new" hardware and software is needed. The crazy schedules for new versions and releases is unsustainable UNLESS much bigger testing environment is involved AND proprietary codes and designs are more open to reliable reviews.

d) One of these days, fine-print in licensing for software/hardware will be significantly modified and the cost to the business for losses and/or time spent to fix issues will be passed to vendors.

e) Internet, with its good and bad sides, still offers enough quality-based reports that many of us can use to make educated decisions based on other people's experiences. Whenever I am planning new purchases, be it for private use, or companies I work for, I spend significant time searching Internet to find out what others think about the product I am interested in...

I have been in IT for 34 years now and have always wondered why we accept such solutions. If I was in search for a new car, I would run away from salesperson who would tell me something along these lines:

"This car comes with four wheels today, but that is not guaranteed even during warranty period in the future...".

Looking at various customers, I see that the unhappiness, panic, and amount of work required to fix Spectre/Meltdown is more than massive.

Regards,

Dusan Baljevic, amateur radio VK2COT

Hey there,

seems like Google found a Way to mitigate those massive Performance Issues ... especially for the Spectre Variant 2 which is causing the Troubles ... (Software fix instead of disabling CPU Features)

https://blog.google/topics/google-cloud/protecting-our-google-cloud-customers-new-vulnerabilities-without-impacting-performance/

But as far as I understand - this would require everything to be recompiled?!? Guess RHEL won't do this in foreseeable future? (And Virtuozzo is based on RHEL ...)