Meltdown/Spectre patches for E2E Networks Cloud & Dedicated servers

Meltdown-Spectre-Image-E2E

What are Meltdown and Spectre?

Meltdown breaks the mechanism that keeps applications from accessing arbitrary system memory. Consequently, applications can access system memory. Spectre tricks other applications into accessing arbitrary locations in their memory. Both attacks use side channels to obtain the information from the accessed memory location. For a more technical discussion, we refer to the papers (Meltdown and  Spectre)

Several microarchitectural (hardware) implementation issues affecting many modern microprocessors have surfaced recently. As explained in Red Hat’s security advisory, fixing these requires

“updates to the Linux kernel, virtualization-related components, and/or in combination with a microcode update.  An unprivileged attacker can use these flaws to bypass conventional memory security restrictions in order to gain read access to privileged memory that would otherwise be inaccessible. There are 3 known CVEs related to this issue in combination with Intel, AMD, and ARM architectures. All three rely upon the fact that modern high performance microprocessors implement both speculative execution, and utilize VIPT (Virtually Indexed, Physically Tagged) level 1 data caches that may become allocated with data in the kernel virtual address space during such speculation.

 

Overview

  • CVE-2017-5753 (variant #1/Spectre) is a Bounds-checking exploit during branching. This issue is fixed with a kernel patch. Variant #1 protection is always enabled; it is not possible to disable the patches. Red Hat’s performance testing for variant #1 did not show any measurable impact.
  • CVE-2017-5715 (variant #2/Spectre) is an indirect branching poisoning attack that can lead to data leakage. This attack allows for a virtualized guest to read memory from the host system. This issue is corrected with microcode, along with kernel and virtualization updates to both guest and host virtualization software. This vulnerability requires both updated microcode and kernel patches. Variant #2 behavior is controlled by the ibrs and ibpb tunables (noibrs/ibrs_enabled and noibpb/ibpb_enabled), which work in conjunction with the microcode.
  • CVE-2017-5754 (variant #3/Meltdown) is an exploit that uses speculative cache loading to allow a local attacker to be able to read the contents of memory. This issue is corrected with kernel patches. Variant #3 behavior is controlled by the pti tunable (nopti/pti_enabled).

 

Patching instructions for Customers using E2E Cloud or VIRTUAL MACHINES –

Current status: E2E Cloud Infrastructure utilizes Xen Paravirtualization for the best possible performance. Virtual machine kernels running in 64-bit PV mode are not directly vulnerable to attack using Meltdown, because 64-bit PV guests already run in a KPTI-like mode.

[CentOS Users] – The currently released patched kernel from Red Hat causes the virtual machines to not boot on Xen PV. This has been separately confirmed by people in the AWS and Citrix communities –

https://forums.aws.amazon.com/thread.jspa?messageID=823179

https://discussions.citrix.com/topic/392239-new-centos-6-kernel-fails-to-boot-on-xenserver-65/

We are awaiting revised kernel packages from Red Hat which will be suitable for use by our cloud customers. We will send out an update when they become available. For now, please continue with the older stable non-patched kernel in your CentOS virtual machines.

[Ubuntu and Debian Users] Please follow the same instructions as provided for users of dedicated machines below:-

 

Patching instructions for Customers using DEDICATED MACHINES –

The following sections give information pertaining to available updates for CentOS, Ubuntu and Debian distributions.

Update all affected packages. Update your kernel and reboot into the same. You may ignore qemu-kvm and libvirt packages unless you are using virtualization packages.

For more information on optionally disabling the fixes while using the new kernels, see the Red Hat article in the Notes section at the end.

 

 

Fix on CentOS

[Note] If you are a CentOS user using cloud/virtual machines, _do not_ proceed with the kernel upgrades. Please see patching instructions for CentOS virtual machines in the previous section of this document.

$ sudo yum update kernel microcode_ctl linux-firmware qemu-kvm libvirt

Edit /boot/grub/grub.conf on CentOS 6 such that default=0 is set, signifying that the latest kernel (mentioned at the top of the list of boot entries) should be booted.

On CentOS 6, the first 8 uncommented lines of grub.conf should look like this  –

default=0

timeout=5

splashimage=(hd0,0)/grub/splash.xpm.gz

hiddenmenu

title CentOS (2.6.32-696.18.7.el6.x86_64)

       root (hd0,0)

       kernel /vmlinuz-2.6.32-696.18.7.el6.x86_64 ro root=/dev/mapper/storage-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=85d9e5f1:57836183:aebaae46:2601caca SYSFONT=latarcyrheb-sun16 crashkernel=auto rd_LVM_LV=storage/root  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet

       initrd /initramfs-2.6.32-696.18.7.el6.x86_64.img

 

On CentOS 7, verify /boot/grub2/grub.cfg

 

grep -A1 “BEGIN /etc/grub.d/10_linux” /boot/grub2/grub.cfg

### BEGIN /etc/grub.d/10_linux ###

menuentry ‘CentOS Linux (3.10.0-693.11.6.el7.x86_64) 7 (Core)’ –class centos –class gnu-linux –class gnu –class os –unrestricted $menuentry_id_option ‘gnulinux-3.10.0-693.el7.x86_64-advanced-93c83fb8-fd60-445a-8f0b-3be17d41146b’ {

 

Boot into the new kernel: For CentOS dedicated machines, use the “reboot” command.

Fixed packages for CentOS

 

 

Fix on Ubuntu

Current patch will only address CVE-2017-5754 (aka Meltdown or Variant 3) for x86_64. A fix for “Spectre” variants will be available soon. Ubuntu 17.04 will not receive any fix.

$ sudo apt-get update

[ for Ubuntu 16.04 ]

$ sudo apt-get install linux-generic

[ for Ubuntu 14.04 ]

$ sudo apt-get install linux-image-4.4.0-108-generic

Edit /boot/grub/menu.lst such that default=0 is set, signifying that the latest kernel (mentioned at the top of the list of boot entries) should be booted. The first 8 uncommented lines of menu.lst should look like this –

default=0

timeout=10

 

title vmlinuz-4.4.0-108-generic

 root (hd0,0)

 kernel /boot/vmlinuz-4.4.0-108-generic root=/dev/xvda console=hvc0 ro

 initrd /boot/initrd.img-4.4.0-108-generic

 

Boot into the new kernel: For Ubuntu cloud/virtual machines, use the reboot button on the cloud console and for dedicated machines, use the “reboot” command.

Fixed packages for Ubuntu

Package Version Series
linux 4.4.0-108.131 Xenial 16.04
linux 4.13.0-25.29 Artful 17.10
linux-aws 4.4.0-1047.56 Xenial 16.04
linux-aws 4.4.0-1009.9 Trusty 14.04
linux-azure 4.13.0-1005.7 Xenial 16.04
linux-euclid 4.4.0-9021.22 Xenial 16.04
linux-gcp 4.13.0-1006.9 Xenial 16.04
linux-hwe-edge 4.13.0-25.29~16.04.1 Xenial 16.04
linux-kvm 4.4.0-1015.20 Xenial 16.04
linux-lts-xenial 4.4.0-108.131~14.04.1 Trusty 14.04
linux-oem 4.13.0-1015.16 Xenial 16.04

 

 

Fix on Debian

CVE-2017-5754 (aka Meltdown or Variant 3) is fixed. “Spectre” mitigations are a work in progress.

$ sudo apt-get update

$ sudo apt-get install linux-image-amd64

This will install the updated kernel release package linux-image-3.16.0-5-amd64 on Debian 8 and linux-image-4.9.0-5-amd64 on Debian 9.

 

Boot into the new kernel: For Ubuntu cloud/virtual machines, use the reboot button on the cloud console and for dedicated machines, use the “reboot” command. With the new kernel version, you should see 3.16.51-3+deb8u1 for Debian 8 Jessie and 4.9.65-3+deb9u2 for Debian 9 –

# uname -srv

Linux 3.16.0-5-amd64 #1 SMP Debian 3.16.51-3+deb8u1 (2018-01-08)

# uname -srv

Linux 4.9.0-5-amd64 #1 SMP Debian 4.9.65-3+deb9u2 (2018-01-04)

 

Vulnerable and fixed packages for Debian

Source Package Release Version Status
linux (PTS) wheezy 3.2.78-1 vulnerable
wheezy (security) 3.2.96-3 fixed
jessie 3.16.51-2 vulnerable
jessie (security) 3.16.51-3+deb8u1 fixed
stretch 4.9.65-3 vulnerable
stretch (security) 4.9.65-3+deb9u2 fixed
buster 4.14.7-1 vulnerable
sid 4.14.12-2 fixed

The information below is based on the following data on fixed versions.

Package Type Release Fixed Version Urgency Origin Debian Bugs
linux source (unstable) 4.14.12-1 medium
linux source jessie 3.16.51-3+deb8u1 medium DSA-4082-1
linux source stretch 4.9.65-3+deb9u2 medium DSA-4078-1
linux source wheezy 3.2.96-3 medium DLA-1232-1

 

 

Fix on Windows

Windows Server-based machines (physical or virtual) should get the Windows security updates that were released on January 3, 2018, and are available from Windows Update. The following updates are available:

Operating system version Update KB
Windows Server, version 1709 (Server Core Installation) 4056892
Windows Server 2016 4056890
Windows Server 2012 R2 4056898
Windows Server 2012 Not available
Windows Server 2008 R2 4056897
Windows Server 2008 Not available

 

Use these registry keys to enable the mitigations on the server and make sure that the system is restarted for the changes to take effect:

Switch | Registry Settings

To enable the fix

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverride /t REG_DWORD /d 0 /f

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

reg add “HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization” /v MinVmVersionForCpuBasedMitigations /t REG_SZ /d “1.0” /f

If this is a Hyper-V host and the firmware updates have been applied: fully shutdown all Virtual Machines (to enable the firmware related mitigation for VMs you have to have the firmware update applied on the host before the VM starts).

Restart the server for changes to take effect.

To disable this fix

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverride /t REG_DWORD /d 3 /f

reg add “HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management” /v FeatureSettingsOverrideMask /t REG_DWORD /d 3 /f

Restart the server for the changes to take effect.

(There is no need to change MinVmVersionForCpuBasedMitigations.)

Note For Hyper-V hosts, live migration between patched and unpatched hosts may fail: See https://docs.microsoft.com/en-us/virtualization/hyper-v-on-windows/CVE-2017-5715-and-hyper-v-vms for more information.

Verifying that protections are enabled

To help customers verify that protections have been enabled, Microsoft has published a PowerShell script that customers can run on their systems. Install and run the script by running the following commands:

   PowerShell Verification using the PowerShell Gallery (Windows Server 2016 or WMF 5.0/5.1)
   Install the PowerShell Module

   PS> Install-Module SpeculationControl

   Run the PowerShell module to validate the protections are enabled

   PS> # Save the current execution policy so it can be reset

   PS> $SaveExecutionPolicy = Get-ExecutionPolicy

   PS> Set-ExecutionPolicy RemoteSigned -Scope Currentuser

   PS> Import-Module SpeculationControl

   PS> Get-SpeculationControlSettings

   PS> # Reset the execution policy to the original state

   PS> Set-ExecutionPolicy $SaveExecutionPolicy -Scope Currentuser

   PowerShell Verification using a download from Technet (Earlier OS versions/Earlier WMF versions)
   Install the PowerShell Module from Technet ScriptCenter.

   Go to https://aka.ms/SpeculationControlPS

   Download SpeculationControl.zip to a local folder.

   Extract the contents to a local folder, for example C:\ADV180002

   Run the PowerShell module to validate the protections are enabled

   Start PowerShell, then (using the example above), copy and run the following commands:

   PS> # Save the current execution policy so it can be reset

   PS> $SaveExecutionPolicy = Get-ExecutionPolicy

   PS> Set-ExecutionPolicy RemoteSigned -Scope Currentuser

   PS> CD C:\ADV180002\SpeculationControl

   PS> Import-Module .\SpeculationControl.psd1

   PS> Get-SpeculationControlSettings

   PS> # Reset the execution policy to the original state

   PS> Set-ExecutionPolicy $SaveExecutionPolicy -Scope Currentuser

 

The output of this PowerShell script will resemble the following. Enabled protections appear in the output as “True.”

PS C:\> Get-SpeculationControlSettings

Speculation control settings for CVE-2017-5715 [branch target injection]

Hardware support for branch target injection mitigation is present: True

Windows OS support for branch target injection mitigation is present: True

Windows OS support for branch target injection mitigation is enabled: True

Speculation control settings for CVE-2017-5754 [rogue data cache load]

Hardware requires kernel VA shadowing: True

Windows OS support for kernel VA shadow is present: True

Windows OS support for kernel VA shadow is enabled: True

Windows OS support for PCID optimization is enabled: True

 

 

Notes and References

Performance impact (Linux): Speculative execution is a performance optimization technique. Thus, these updates (both kernel and microcode) may result in workload-specific performance degradation. Therefore, some customers who feel confident that their systems are well protected by other means (such as physical isolation), may wish to disable some or all of these kernel patches. If the end user elects to enable the patches in the interest of security, this article provides a mechanism to conduct performance characterizations with and without the fixes enabled. Controlling the Performance Impact of Microcode and Security Patches for CVE-2017-5754 CVE-2017-5715 and CVE-2017-575

https://meltdownattack.com/