Dubbo网关(Dubbo Gateway)

Dubbo是来自淘宝的开源分布式框架,后来贡献给了Apache社区,https://github.com/apache/dubbo。主要实现了服务注册、服务发现和远程调用(RPC)。Dubbo在Spring Boot流行起来之前,是中国很流行的分布式开发框架,有很大的用量。

背景与概述

Dubbo在2.6版本以后,通过jboss resteasy提供了内置的REST支持,在之前版本有通过第三方办法支持REST的。关于在分布式系统里,尤其是微服务系统里,到底是用REST还是用RPC,一直存在不同的理解和认识。这里我们不讨论这个话题。从客观上说,这几年随着spring cloud的发展,以及硬件基础设施(网络、计算能力)等不停的增强,REST被越来越多的人接受。

Dubbo在早期并没有提供服务治理能力(早期的Dubbo就是一个分布式调用框架,并不是完整的微服务平台。实际上Dubbo是SOA时期向微服务过度过程中的一个形态),后来框架里引入了Sentinel实现服务治理,但是Sentinel目前没有大规模使用的案例,成熟度和可靠性有待验证。在Flomesh团队看来,Dubbo在近几年一直在学习和效仿Spring Cloud,Sentinel就是在模仿Hystrix。

无论Spring Cloud Hystrix还是Dubbo Sentinel,都是在框架(framework)层面实现服务治理,我们称为“基于框架的服务治理”。随着kubernetes的成熟,以及service mesh概念的普及和被社区广为接受,“基于网络的服务治理”成为新的选择。这里不去比较两种思路的优劣,Flomesh团队更认可基于网络的服务治理(或者我们称为“基于mesh的服务治理”)思路,主要原因是:

  • 支持多语言。基于网络协议的方式回避了不同开发语言对同一种框架的支持差异。虽然也有一些把A框架能力迁移到B语言栈的例子(比如在go里支持Dubbo),但是更多的还是框架体系内的服务治理
  • 支持同一框架多个版本。即使是一个框架,比如Dubbo,随着使用时间推移,在组织内会逐渐形成多版本共存的局面,跨版本之间的服务调用、服务治理,都是衍生出来的话题。多数的框架本身都没有把这类问题的解决放在自己的roadmap上。Mesh是在网络协议层和负载调度层实现了治理,通常即使版本升级,协议也都兼容的
  • 支持跨集群能力。无论是基于管理原因,还是技术原因,规模使用框架都会有多集群的情况,跨集群的服务治理(或者叫做联邦模式),都是已有框架的短板。基于Mesh的服务治理,由于是从网络切入解决问题,所以天生的具备跨集群能力
  • 支持异构服务的统一治理。存量的Dubbo和新增的Spring Boot共存共用互访互通是一种典型的异构服务治理

在这个文档中,我们演示Flomesh在这个方向上的两个场景:

  1. 在Dubbo RPC调用链路中插入Dubbo网关,实现限速
  2. 把Dubbo网关和Dubbo服务以Sidecar方式部署,对外提供REST能力(REST领域的服务治理工具和手段更丰富)

场景1是没有改变Dubbo的RPC方式,通过串路网关,实现功能

场景2通过Dubbo网关把Dubbo服务暴露成REST,主要是兼容存量的Dubbo服务。新的Dubbo服务如果需要REST可以直接用框架的REST能力

版权与免责

本文所述的piped程序为Flomesh团队开发并拥有版权(并非open source software或者freeware),这里作为测试提供。此文档及安装介质可以免费下载用于测试、学习、非盈利目的使用。对于非原文转载,请注明出处(http://flomesh.cn)。piped程序不得作为盈利目的使用。Flomesh团队会维护该文档确保有效性,维护piped程序确保稳定、可靠,但是Flomesh团队不对此文档和该程序的使用负有责任。

部署环境

在这两个演示里,我们使用如下的拓扑:

场景1:dubbo consumer --> consumer sidecar --> dubbo provider
场景2:ab(apache bench) --> provider sidecar --> dubbo provider

这里我们使用piped作为sidecar。

部署Dubbo Provider

这个虚拟机的IP是192.168.122.72

在最小安装的centos7上:

部署dubbo演示服务:

git clone https://github.com/flomesh-io/dubbo-demo.git
cd dubbo-demo
./mvnw package

然后启动Dubbo服务端:

cd dubbo-demo/dubbo-spring-boot-samples/dubbo-registry-zookeeper-samples/provider-sample
java -jar target/dubbo-spring-boot-registry-zookeeper-provider-sample-2.7.0.jar

默认除了启动Dubbo服务外,还启动了一个内置的zookeeper作为注册中心。Dubbo在端口20881,zookeeper在2181

使用iptables把8000端口请求转发到20881

iptables -t nat -A PREROUTING -p tcp --dport 8000 -j REDIRECT --to-port 20881

这一步的目的,我们稍后执行测试时候解释

安装piped:

wget http://54.92.105.113/piped/piped-0.1.0-55.el7_pl.x86_64.rpm
yum -y localinstall piped-0.1.0-55.el7_pl.x86_64.rpm

启动piped

我们启动piped,让他监听在8080端口,对外提供HTTP服务,把HTTP请求转换为Dubbo调用,发给127.0.0.1:20881也就是Dubbo服务。这个piped和Dubbo服务以sidecar模式部署在同一个虚拟机里(后续会容器化的pod部署方式,本质是一样的)。

编辑/etc/piped/rest-dubbo.ini,内容如下:

;
; REST -> Dubbo
;

[pipeline.http]
listen = 0.0.0.0:8080

[module.1]
name = http-request-decoder

[module.2]
name = json-decoder

[module.3]
name = hessian2-encoder

[module.4]
name = dubbo-request-encoder

[module.5]
name = proxy
upstream = 127.0.0.1:20881

[module.6]
name = dubbo-response-decoder

[module.7]
name = hessian2-decoder

[module.8]
name = json-encoder

[module.9]
name = http-response-encoder

启动piped:

cd /etc/piped
piped rest-dubbo.ini

验证

到这里,Dubbo服务端部署完成,Dubbo服务监听在20881、2181端口;piped监听在8080端口,提供HTTP服务,并把请求转换成Dubbo请求,转发到20881端口;iptables把发往8000端口的请求,转发到20881上。

[root@localhost ~]# netstat -anp | grep java
tcp        0      0 0.0.0.0:2181            0.0.0.0:*               LISTEN      1863/java 
tcp        0      0 0.0.0.0:20881           0.0.0.0:*               LISTEN      1863/java 
tcp        0      0 127.0.0.1:50184         127.0.0.1:2181          ESTABLISHED 1863/java 
tcp        0      0 127.0.0.1:50180         127.0.0.1:2181          ESTABLISHED 1863/java 
tcp        0      0 127.0.0.1:2181          127.0.0.1:50180         ESTABLISHED 1863/java 
tcp        0      0 127.0.0.1:2181          127.0.0.1:50184         ESTABLISHED 1863/java 
unix  2      [ ]         STREAM     CONNECTED     205032   1863/java            
unix  2      [ ]         STREAM     CONNECTED     205028   1863/java            

[root@localhost ~]# netstat -anp | grep piped
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      2107/piped  

[root@localhost ~]# iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8000 redir ports 20881

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 3 packets, 256 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 3 packets, 256 bytes)
 pkts bytes target     prot opt in     out     source               destination     

部署Dubbo Consumer

这个虚拟机的IP是192.168.122.10

在最小安装的centos7上:

部署dubbo演示服务:

git clone https://github.com/flomesh-io/dubbo-demo.git
cd dubbo-demo
./mvnw package

启动Consumer的方式:

cd dubbo-demo/dubbo-spring-boot-samples/dubbo-registry-zookeeper-samples/consumer-sample
java -jar target/dubbo-spring-boot-registry-zookeeper-consumer-sample-2.7.0.jar

默认情况下,Consumer会循环调用服务端100次,在我的演示环境里,这个过程大概需要10-12秒,也就是平均每秒8-10次调用

安装piped:

wget http://54.92.105.113/piped/piped-0.1.0-55.el7_pl.x86_64.rpm
yum -y localinstall piped-0.1.0-55.el7_pl.x86_64.rpm

启动piped

我们启动piped,让他监听在8000端口,然后对dubbo调用进行Throttle操作。

编辑/etc/piped/dubbo-throttle.ini,内容如下:

;
; Dubbo Throttle
;
[pipeline.proxy]
listen = 0.0.0.0:8000

[module.1]
name = dubbo-request-decoder

[module.tap]
name = tap
limit = 2

[module.2]
name = dubbo-request-encoder

[module.7]
name = proxy
upstream = 192.168.122.10:8000

启动piped:

cd /etc/piped
piped dubbo-throttle.ini

这里我们速度限制在每秒两次调用(limit = 2)。因此,100次循环调用大概需要50秒

注意这个配置最后一行,请求被piped转发到Dubbo服务所在虚拟机的8000端口。如果记得上边的配置,这个端口的请求会被iptables转发到20881端口,也就是Dubbo服务端口

配置iptables

到这里,我们把Dubbo consumer和piped做了sidecar的部署。我们还需要把Dubbo consumer发出的请求拦截,转发到piped上(默认情况是直接连接了Dubbo provider的20881端口)。通过这个iptables规则,我们把直接访问改变成了经过piped访问。实际上,kubernetes里实现sidecar拦截流量的方式也是一样的。

iptables -t nat -A OUTPUT -p tcp --dport 20881 -j DNAT --to-destination 127.0.0.1:8000

安装ab(apache bench)

我们还会从这个dubbo consumer的虚拟机发起REST请求,访问dubbo provider上的8080端口。安装ab:

yum -y install httpd-tools

场景1: Dubbo RPC限速

在Dubbo consumer的虚拟机里,启动dubbo consumer,会循环调用dubbo服务100次:

[root@localhost consumer-sample]# java -jar target/dubbo-spring-boot-registry-zookeeper-consumer-sample-2.7.0.jar | grep transfer
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/root/dubbo-spring-boot-project/dubbo-spring-boot-samples/dubbo-registry-zookeeper-samples/consumer-sample/target/dubbo-spring-boot-registry-zookeeper-consumer-sample-2.7.0.jar!/BOOT-INF/lib/logback-classic-1.2.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/root/dubbo-spring-boot-project/dubbo-spring-boot-samples/dubbo-registry-zookeeper-samples/consumer-sample/target/dubbo-spring-boot-registry-zookeeper-consumer-sample-2.7.0.jar!/BOOT-INF/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
log4j:WARN No appenders could be found for logger (org.apache.dubbo.common.logger.LoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2020-01-22 13:14:37.736  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:37.978  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:38.769  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:38.987  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:39.770  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:39.938  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:40.771  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:40.932  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:41.771  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:41.936  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:42.772  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:42.932  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:43.771  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:43.940  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:44.771  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:44.948  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:45.770  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:45.920  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:46.771  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:46.925  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:47.772  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:47.923  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:48.772  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:48.917  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:49.773  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:49.925  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:50.774  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:50.914  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:51.774  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:51.920  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:52.776  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:52.922  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:53.776  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:53.920  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:54.777  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:54.921  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:55.777  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:55.902  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:56.779  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:56.916  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:57.779  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:57.929  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:58.781  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:58.924  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:59.781  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:14:59.925  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:00.782  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:00.926  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:01.783  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:01.916  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:02.784  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:02.930  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:03.784  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:03.941  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:04.784  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:04.929  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:05.785  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:05.914  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:06.786  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:06.919  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:07.786  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:07.912  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:08.786  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:08.929  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:09.787  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:09.939  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:10.787  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:10.910  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:11.788  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:11.909  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:12.789  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:12.921  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:13.789  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:13.920  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:14.790  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:14.928  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:15.790  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:15.914  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:16.790  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:16.907  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:17.791  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:17.914  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:18.792  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:18.928  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:19.793  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:19.929  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:20.793  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:20.917  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:21.794  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:21.932  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:22.794  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:22.913  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:23.795  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:23.916  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:24.795  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:24.922  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:25.796  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:25.915  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:26.796  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer
2020-01-22 13:15:26.925  INFO 2236 --- [           main] .DubboRegistryZooKeeperConsumerBootstrap : [dubbo-registry-zookeeper-provider-sample] : transfer

可以看到,流量被限制到每秒2个请求,总执行时长为50秒。

回顾一下这个场景网络请求过程

  1. Dubbo consumer发起请求,请求服务端20881端口
  2. 请求被Dubbo consumer虚拟机上的iptables规则拦截,转发到sidecar piped的8000端口
  3. piped完成限速(Throttle)功能,并转发请求到Dubbo provider的8000端口
  4. Dubbo provider的8000端口,被iptables重定向到20881端口,也就是实际的dubbo服务端口
dubbo consumer --> iptables --> piped (throttle) --> dubbo provider iptables --> dubbo provider

场景2: 用REST访问Dubbo服务

这个场景里,我们从dubbo consumer的虚拟机发起REST请求(实际上从网络任何位置访问都可以)。请求端采用ab,10个并发,请求10000次。数据流程是这样的:

ab(apache bench, c10, n10000) --> piped(sidecar on dubbo provider) --> dubbo service

测试结果如下:

[root@localhost dubbo]
cd /etc/piped/test/dubbo
[root@localhost dubbo]# ab -c 10 -n 10000 -f dubbo.json http://192.168.122.10:8080/
This is ApacheBench, Version 2.3 <$Revision: 1430300 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.122.10 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        
Server Hostname:        192.168.122.10
Server Port:            8080

Document Path:          /
Document Length:        133 bytes

Concurrency Level:      10
Time taken for tests:   2.114 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      2520000 bytes
HTML transferred:       1330000 bytes
Requests per second:    4729.34 [#/sec] (mean)
Time per request:       2.114 [ms] (mean)
Time per request:       0.211 [ms] (mean, across all concurrent requests)
Transfer rate:          1163.86 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:     1    2   0.4      2       9
Waiting:        0    2   0.4      2       8
Total:          1    2   0.4      2       9

Percentage of the requests served within a certain time (ms)
  50%      2
  66%      2
  75%      2
  80%      2
  90%      2
  95%      3
  98%      3
  99%      3
 100%      9 (longest request)

在这个过程当中,piped使用的资源情况:

[root@localhost ~]# top | grep piped
 2107 root      20   0   92488   2332   1668 S 110.0  0.1   0:14.92 piped                 
 2107 root      20   0   92488   2332   1668 R 109.3  0.1   0:18.21 piped                 
 2107 root      20   0   92488   2332   1668 S 110.3  0.1   0:21.52 piped                 
 2107 root      20   0   92488   2332   1668 R 109.6  0.1   0:24.82 piped                 
 2107 root      20   0   92488   2332   1668 R 110.0  0.1   0:28.13 piped                 
 2107 root      20   0   92488   2332   1668 S 109.7  0.1   0:31.42 piped                 
 2107 root      20   0   92488   2332   1668 S 109.6  0.1   0:34.72 piped                 
 2107 root      20   0   92488   2332   1668 S 109.3  0.1   0:38.00 piped                 
 2107 root      20   0   92488   2332   1668 S  44.5  0.1   0:39.34 piped     

piped使用了2.3M内存

注意请求方POST了一个json文件,在dubbo consumer的/etc/piped/test/dubbo/dubbo.json。这个文件的格式需要符合dubbo服务的接口要求(这个接口实际是一个java的接口,piped完成了json到java对象的转换,序列化格式是henssion2)

从测试结果看,在给定的测试环境下,Dubbo服务在单一jvm下可以支持4700RPS的REST(通过piped sidecar处理)请求速度。和原生的Java HTTP服务相差无几。平均请求时长为2ms,和标准的REST服务能力相当。也就是通过piped的sidecar部署,把一个Dubbo RPC服务,无缝的转换成为REST服务;原生的Dubbo RPC服务无需做任何改动,就可以适配到REST模式。

结论

Dubbo网关的引入,对于Dubbo的服务治理提供了必要的前提条件。基于piped的Dubbo网关,以很小的开销(内存和CPU),为sidecar模式的Dubbo服务治理奠定了基础。