性能测试

概述

在使用网关和web服务器的时候,性能测试是一项常见的工作,性能测试可以用于多种目的和用途:

  • 确定性能基线
  • 度量系统的横向扩展能力
  • 疲劳测试,检验软件在持续高负载下的可靠性与稳定性
  • 对比测试不同的方案与性能的关联

这里我们分享一个我们常用的、基本的办法用于给类似的任务作参考。

目标

通过这个测试,我们希望实现如下目的:

  • 对比三种软件在作为web服务器时候的性能
  • 对比三种软件在作为网关时候的性能
  • 对比1cpu,2cpu,4cpu不同配置对性能的影响

关于三个软件

  1. Nginx,我们选择用CentOS7 EPEL自己带的1.12版本
  2. Kong,我们选择自己打包的0.14.1版本。自己打包的主要目的是包含了对MySQL DAO的支持
  3. Piped,这个是我们自己用C++开发的软件,用于作协议和格式转换;其中包含了http协议的处理

工具

在测试过程中,我们采用最常见的ab(apache bench)作为压力软件;用nmon搜集系统性能指标。这两个工具都是在centos7的epel里包含的,简单易用而且随手可得。

安装与准备

我们采用一个虚拟机作为测试环境,被测试软件(Nginx/Kong/Piped)、测试软件(ab)、数据库(MariaDB10)、性能采集软件(nmon)都运行在同一个虚拟机里。这样做的目的是简化环境的部署与配置;在实际使用当中,可以参考这个部署,把软件分布到不同的服务器(虚拟机)上,其核心原理和步骤是非常接近的。

安装操作系统

最小化安装CentOS。作为起步的测试环境,我们给虚拟机配备了1CPU,2G内存。整个虚拟机我们采用20G单一磁盘,实际上这类性能测试对磁盘没有很强的性能要求,只要磁盘空间可以满足测试时日志存储要求即可。

在安装系统之后,我们关闭防火墙,关闭selinux,配置基本的内核参数:

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
echo "fs.file-max = 1024000" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_recycle = 1" >> /etc/sysctl.conf
echo "net.ipv4.tcp_tw_reuse = 1" >> /etc/sysctl.conf
sysctl -p

同时,为了数据具备参考意义,我们收集下基本的CPU信息:

cat /proc/cpuinfo

其中和性能相关的几个信息如下:

vendor_id	: GenuineIntel
cpu family	: 6
model		: 94
model name	: Intel Core Processor (Skylake, IBRS)
stepping	: 3
cpu MHz		: 1992.000
cache size	: 16384 KB
cpu cores	: 1
bogomips	: 3984.00

主频、缓存大小、MIPS是最相关的数据。

需要注意的是,如果组件分开部署,网络性能也是至关重要的。但是这个演示测试里,我们是在同一个系统内测试,可以忽略网络性能的影响。

安装软件

  • 安装Nginx
yum -y install epel-release
yum -y install nginx
echo "ok" >> /usr/share/nginx/html/ok.html
systemctl start nginx
systemctl enable nginx
  • 设置Mariadb10的repo:
[root@localhost ~]# cat /etc/yum.repos.d/MariaDB.repo 
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.1/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1
  • 安装Kong-MySQL:
yum -y install http://repo.polaristech.io/kong-mysql-0.14.1.rpm
yum -y install Mariadb-server Mariadb-client
systemctl enable mariadb
systemctl start mariadb
 echo "create database kong" | mysql -uroot -p
kong migrations up

注意修改/etc/kong/kong.conf中数据库连接参数的设定,可以参考这个:

[root@localhost ~]# grep mysql /etc/kong/kong.conf
database = mysql
mysql_host = 127.0.0.1
mysql_port = 3306
mysql_user = root
mysql_password = 
mysql_database = kong

创建一个kong的service、route,并配置一个request termination插件:

curl -i -X POST http://localhost:8001/services --data 'name=demo' --data 'url=http://localhost/ok.html'
curl -i -X POST http://localhost:8001/services/demo/routes --data 'paths[]=/'
curl -X POST http://localhost:8001/plugins --data "name=request-termination" --data "config.status_code=200" --data "config.message=good
  • 安装piped:
yum -y install http://repo.polaristech.io/piped-0.1.1.rpm
systemctl enable piped
systemctl start piped

piped的配置文件在/etc/piped/config.ini,这里我们先作简单的http测试,这个文件的内容如下:

[root@localhost ~]# cat /etc/piped/config.ini 
[common]
log_level = info

[pipeline.1]
listen = 0.0.0.0:8080

[module.1]
name = http-request-decoder

[module.2]
name = echo

[module.3]
name = http-response-encoder

这个配置文件的意思是:在8080端口接受请求;对请求作http的decode,也就是分析http协议;把请求的内容echo回去;对返回值作http response的编码,也就是返回http回复。

确认功能

安装之后,nginx监听在80端口,提供http://localhost/ok.html的处理;kong运行在http://localhost:8000/;piped运行在http://localhost:8080/,我们可以用如下命令验证这些服务是正常工作的:

[root@localhost ~]# curl -i http://localhost/ok.html
HTTP/1.1 200 OK
Server: nginx/1.12.2
Date: Fri, 31 May 2019 13:51:35 GMT
Content-Type: text/html
Content-Length: 3
Last-Modified: Fri, 31 May 2019 03:49:22 GMT
Connection: keep-alive
ETag: "5cf0a442-3"
Accept-Ranges: bytes

ok
[root@localhost ~]# curl -i http://localhost:8000/
HTTP/1.1 200 OK
Date: Fri, 31 May 2019 13:52:14 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
Server: kong/0.14.1
Content-Length: 19

{"message":"good"}
[root@localhost ~]# curl -i http://localhost:8080/
HTTP/1.1 200 OK
Content-Length: 0

设定nmon

我们既可以在命令行直接用nmon启动来实时查看性能数据(按键c表示看处理器信息;按键n表示看网络信息;按键d表示看磁盘信息);也可以把nmon放在后台一直执行,nmon会把采集的性能数据写到文件中,这个文件后续可以用软件(比如:https://nmonvisualizer.github.io/nmonvisualizer/)打开来看图形展示。

yum -y install nmon
nmon -f -s 5 -c 100000

测试场景1:单处理器web服务

测试的时候,我们用50并发,100万请求。

Nginx单处理器

[root@localhost ~]# ab -c 50 -n 1000000 http://localhost/ok.html
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        nginx/1.12.2
Server Hostname:        localhost
Server Port:            80

Document Path:          /ok.html
Document Length:        3 bytes

Concurrency Level:      50
Time taken for tests:   72.975 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      232000000 bytes
HTML transferred:       3000000 bytes
Requests per second:    13703.26 [#/sec] (mean)
Time per request:       3.649 [ms] (mean)
Time per request:       0.073 [ms] (mean, across all concurrent requests)
Transfer rate:          3104.65 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       8
Processing:     0    3   0.4      3      12
Waiting:        0    2   0.2      2      10
Total:          2    4   0.3      4      12

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      4
  95%      4
  98%      4
  99%      5
 100%     12 (longest request)

核心测试结果是:Requests per second: 13703.26 [#/sec] (mean)

Kong单处理器

[root@localhost ~]# ab -c 50 -n 1000000 http://localhost:8000/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        kong/0.14.1
Server Hostname:        localhost
Server Port:            8000

Document Path:          /
Document Length:        19 bytes

Concurrency Level:      50
Time taken for tests:   132.681 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      182000000 bytes
HTML transferred:       19000000 bytes
Requests per second:    7536.85 [#/sec] (mean)
Time per request:       6.634 [ms] (mean)
Time per request:       0.133 [ms] (mean, across all concurrent requests)
Transfer rate:          1339.56 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.4      1       7
Processing:     1    6   1.3      5      25
Waiting:        1    5   1.2      4      24
Total:          2    7   1.3      6      25

Percentage of the requests served within a certain time (ms)
  50%      6
  66%      7
  75%      7
  80%      7
  90%      8
  95%     10
  98%     12
  99%     12
 100%     25 (longest request)

核心测试结果是:Requests per second: 7536.85 [#/sec] (mean)

Piped单处理器

[root@localhost ~]# ab -c 50 -n 1000000 -p echo.json http://localhost:8080/echo
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests


Server Software:        piped/0.1.1
Server Hostname:        localhost
Server Port:            8080

Document Path:          /echo
Document Length:        3 bytes

Concurrency Level:      50
Time taken for tests:   67.405 seconds
Complete requests:      1000000
Failed requests:        0
Write errors:           0
Total transferred:      41000000 bytes
Total body sent:        135000000
HTML transferred:       3000000 bytes
Requests per second:    14835.64 [#/sec] (mean)
Time per request:       3.370 [ms] (mean)
Time per request:       0.067 [ms] (mean, across all concurrent requests)
Transfer rate:          594.01 [Kbytes/sec] received
                        1955.87 kb/s sent
                        2549.88 kb/s total

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.3      1       4
Processing:     0    2   0.4      2       6
Waiting:        0    2   0.2      2       4
Total:          2    3   0.3      3       7

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      3
  80%      3
  90%      4
  95%      4
  98%      4
  99%      4
 100%      7 (longest request)

核心测试结果是:Requests per second: 14835.64[#/sec] (mean)