apim
2018/02/21
21 Feb, 2018

The Impact of Meltdown, Spectre and JDK Patches on the Performance of WSO2 API Manager

  • WSO2 Team
  • Content Writer - WSO2

Authors: Malith Jayasinghe, WSO2 Director of Platform Architecture, and Milan Karunarathne, WSO2 Intern

Meltdown and Spectre exploit critical vulnerabilities in modern processors. Meltdown breaks the isolation between user applications and the operating system while Spectre breaks the isolation between different applications. OS level patches have been released to mitigate Meltdown and Spectre vulnerabilities. In addition to this, JDK 8u162 release provides fixes for certain Spectre and Meltdown for Intel processor vulnerabilities.

In this article, we will investigate the impact of the following on the performance of WSO2 API Manager (referred here as APIM):

  1. Meltdown Fixes (CVE-2017-5754)
  2. Spectre Fixes (CVE-2017-5753)
  3. Oracle JAVA JDK Updates (8u162)

This performance evaluation is carried out using different message sizes (50B, 1K, 10K), concurrent users (100, 200, 300, 1000, 2000) and back-end delays (0ms, 500ms) using the following two APIs:

  • Echo API: This is a secured API which directly invokes the back-end service.
  • Mediation API: This is a secured API which has a “sequence” as a mediation extension.

The back-end service is a simple “Netty HTTP Echo Service” which echoes back any request posted to the service. The performance is measured using two metrics; throughput and response time.

The performance evaluation is carried out in the following order:

  1. Test performance using (old) JDK 8u144 without OS level Meltdown/Spectre fixes
  2. Test performance using (old) JDK 8u144 with OS level Meltdown/Spectre fixes
  3. Test performance using (new) JDK 8u162 with OS level Meltdown/Spectre fixes

Deployment Architecture

Figure 1 below shows the deployment architecture we will be using.

Figure 1: Deployment architecture

As shown in the above diagram, WSO2 API Manager, and the databases were deployed on two separate machines. The back-end service which is the HTTP echo service was deployed on a separate machine. The two JMeter servers and the JMeter client were deployed on 3 separate machines. The bandwidth of the network is 100Mbps. The performance tests were run on physical computers and the specifications of the machines are provided in the following table.

Node Number of cores Processor details OS Kernel version - before patching Kernel version - after patching
JMETER 1 4 Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz debian 9 4.9.0-5-amd64 4.9.0-5-amd64
JMETER 2 4 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz debian 9 4.9.0-5-amd64 4.9.0-5-amd64
JMETER client 4 Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz debian 9 4.9.0-5-amd64 4.9.0-5-amd64
APIM 4 Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz Ubuntu 17.10 4.13.0-21-generic 4.13.0-32-generic
DB 4 Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz Ubuntu 17.10 4.13.0-21-generic 4.13.0-32-generic
Netty backend 4 Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz debian 9 4.9.0-5-amd64 4.9.0-5-amd64

The Impact of Meltdown/Spectre Fixes on Performance

We'll now summarize the impact of kernel fixes on the performance. We have done this performance evaluation using different message sizes, back-end delays, and concurrent users. In total there are 60 different scenarios (3 message sizes X 2 back-end delays X 5 concurrencies X 2 APIs). The performance results were collected with and without the kernel fixes. The JDK 8u144 was used. The detailed analysis results can be found in the performance results section.

The key observations of this performance evaluation are summarized below.

We define insignificant impact as less than 3% degradation or less than 3% improvement in the performance and high variation as more than 3% degradation or more than 3% improvement in performance.

  • Out of 60 scenarios tested 54 scenarios had an insignificant impact.
  • There is an insignificant impact on the performance when using mediation API.
  • There is an insignificant impact on the performance when using large message sizes (>1K).
  • There is an insignificant impact on the performance when using slow back-ends ( high response time > 500ms).
  • High variation in the results has been noticed for a number of scenarios in particular when testing with the Echo API with small message sizes and fast backends. For these scenarios, we have seen both improvements (up to 12.58%) and degradations (up to 8.37%) in performance. There were only 6 such scenarios.
  • The performance impact on the throughput and the average latency are very similar.

The Impact of JDK Patches on Performance

We'll now discuss the impact of both the kernel fixes and JDK patches on the performance. We have done this performance evaluation using different message sizes, back-end delays, and concurrent users. In total there are 60 different scenarios (3 message sizes X 2 back-end delays X 5 concurrencies X 2 APIs). The performance results were collected with the kernel fixes using the old and the new JDK.

The key observations of this performance evaluation are summarized below.

  • Out of 60 scenarios tested 56 scenarios had an insignificant impact.
  • There is an insignificant impact when using mediation API.
  • There is an insignificant impact when using large message sizes (>1K).
  • There is an insignificant impact if the back-end delay is greater than 500ms.
  • High variation in the results has been noticed for a number of scenarios in particular with the Echo API when using small message sizes and fast backends. For these scenarios, we have seen both improvements (up to 11.40%) and degradations (up to 10.61%) in performance. There were only 4 such scenarios.
  • The performance impact on the throughput and the average latency are very similar.

Performance Results

Let’s now look at a detailed comparison of throughput results.

Chart 1: TPS vs. Concurrency (message size = 50B and backend delay = 0ms)

Chart 2: TPS vs. Concurrency (message size = 1K and backend delay = 0ms)


Chart 3: TPS vs. Concurrency (message size = 10K and backend delay = 0ms)


Chart 4: TPS vs. Concurrency (message size = 50B and backend delay = 500ms)


Chart 5: TPS vs. Concurrency (message size = 1K and backend delay = 500ms)


Chart 6: TPS vs. Concurrency (message size = 10K and backend delay = 500ms)

In this article, we investigated the performance impact of Spectre, Meltdown and JDK patches on the performance of WSO2 API Manager. Out of 120 scenarios we tested, 110 scenarios had an insignificant impact (< 3%). There was high variation in the results for a number of scenarios. There were 10 such scenarios and for these scenarios we noticed more than 3% improvement/degradation in performance.

 

About Author

  • WSO2 Team
  • Content Writer
  • WSO2