本文共 12482 字，大约阅读时间需要 41 分钟。
最近我这里docker单机平台正式上线使用，使用中有很多问题都一样解决，在给一个游戏项目做测试的时候，此项目由于8080端口对公网全部开放，并且安全策略没有做好（默认的tomcat模板没有删除），导致被人进行webshell，跑了很多的流量，为了解决此问题我针对tc与openvswitch本身的qos做了深入研究，最后选择openvswitch的qos作为容器的网络资源限制方法。
docker本身仅能对容器的cpu、内存做限制，而且必须是在容器运行前做，运行过程中未发现如何动态修改，并且不提供网络资源限制，所以只能使用其他软件做了。
我的docker网络没有使用默认bridge，使用none，然后绑定openvswitch的bridge，并使用pipework提供网络，所以我可以根据容器对于openvswitch的port来进行基于port的网络资源限制，好处是可以动态的修改，坏处是容器一重启还得重新做，但也可以通过其他方法来解决。
一、下面是我做测试的结果：
对于限速100m以下，其实硬盘的类型与读写速度没什么影响，但如何限速150m以上，或者无限速，那么硬盘肯定是ssd>sas>sata，所以建议进来使用sas的磁盘作为docker的挂载分区。
openvswitch默认官方文档提供的限速方法“rate limiting vm traffic using qos policing"，地址是http://openvswitch.org/support/config-cookbooks/qos-rate-limiting/，此限速仅能对上传做限速，下载没有办法，所以还得通过其他的qos来做限制。
二、下面是具体限速脚本
| 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | #!/bin/bash #filename:modify_docker_container_network_limit.sh #author:Deng Lei #email:dl528888@gmail.com op =$1 container=$2 limit=$3 # Mbits/s if [ -z $1 ] || [ -z $2 ]; then echo "Usage: operation container_name limit(default:5m)" echo "Example1: I want limit 5m in the container:test" echo "The command is: bash `basename $0` limit test 5" echo "Example2: I want delete network limit in the container:test" echo "The command is: bash `basename $0` ulimit test" exit 1 fi if [ -z $3 ]; then limit= '5m' fi if [ `docker inspect -- format "``.`State`.`Pid`" $container &>> /dev/null && echo 0 || echo 1` - eq 1 ]; then echo "no this container:$container" exit 1 fi ovs_prefix= 'veth1pl' container_id=`docker inspect -- format "``.`State`.`Pid`" $container` device_name=` echo ${ovs_prefix}${container_id}` if [ $ op == 'limit' ]; then for v in $device_name; do ovs-vsctl set interface $ v ingress_policing_rate=$((limit*1000)) ovs-vsctl set interface $ v ingress_policing_burst=$((limit*100)) ovs-vsctl set port $ v qos=@newqos -- -- id =@newqos create qos type =linux-htb queues=0=@q0 other-config:max-rate=$((limit*1000000)) -- -- id =@q0 create queue other-config:min-rate=$((limit*1000000)) other-config:max-rate=$((limit*1000000)) &>> /dev/null && echo 'modify success!' || echo 'modify fail!' done elif [ $ op == 'ulimit' ]; then for v in $device_name; do ovs-vsctl set interface $ v ingress_policing_rate=0 ovs-vsctl set interface $ v ingress_policing_burst=0 ovs-vsctl clear Port $ v qos &>> /dev/null && echo 'modify success!' || echo 'modify fail!' done fi |
此脚本使用的话，限速可以直接针对下载与上传，并且限制是统一生效的，比如我限制一个容器带宽为5m，那么下载与上传的限速都是5m，单位是bit不是byte。
三、下面是使用方法：
| 1 2 3 4 5 6 | [root@docker-test3 tmp] # sh modify_docker_container_network_limit.sh Usage: operation container_name limit(default:5m) Example1: I want limit 5m in the container: test The command is: bash modify_docker_container_network_limit.sh limit test 5 Example2: I want delete network limit in the container: test The command is: bash modify_docker_container_network_limit.sh ulimit test |
四、下面是测试过程：
测试的方法是：
找另外一个主机172.16.1.126，然后dd生成个100m的文件/tmp/test_client.iso，在本机下载这个文件来测试下载速度，在本机dd生成100m的文件/tmp/test_server.iso把此文件上传到172.16.1.126里测试上传速度。
sata 7.5k
1、没有限制的情况
下载速度
| 1 2 3 4 5 6 7 8 | 14:12:18 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 56.82MB /s 0:00:01 (xfer #1, to-check=0/1) sent 30 bytes received104892888 bytes 1226817.75 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:14:27 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 51.08MB /s 0:00:01 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 29969404.00 bytes /sec total size is 104857600 speedup is 1.00 |
2、限速5m的
下载速度
| 1 2 3 4 5 6 7 8 | 14:15:27 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 580.46kB /s 0:02:56 (xfer #1, to-check=0/1) sent 30 bytes received104892888 bytes 590946.02 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:22:10 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 616.19kB /s 0:02:46 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 571623.51 bytes /sec total size is 104857600 speedup is 1.00 |
3、限速为10m
下载速度
| 1 2 3 4 5 6 7 8 9 | 14:28:55 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 1.13MB /s 0:01:28 (xfer #1, to-check=0/1) sent 30 bytes received 104892888bytes 1109977.97 bytes /sec total size is 104857600 speedup is 1.00 root@fdc81b0d2508: /tmp |
上传速度
| 1 2 3 4 5 6 7 8 | 14:30:33 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 1.21MB /s 0:01:22 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 1133977.45 bytes /sec total size is 104857600 speedup is 1.00 |
4、限速为20m
下载速度
| 1 2 3 4 5 6 7 8 | 14:32:57 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 2.27MB /s 0:00:44 (xfer #1, to-check=0/1) sent 30 bytes received104892888 bytes 2305338.86 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:33:59 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 2.45MB /s 0:00:40 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 2305338.77 bytes /sec total size is 104857600 speedup is 1.00 |
5、限速为50m
下载速度
| 1 2 3 4 5 6 7 8 | 14:35:20 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 5.67MB /s 0:00:17 (xfer #1, to-check=0/1) sent 30 bytes received 104892888bytes 5379124.00 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:35:54 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 6.33MB /s 0:00:15 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 5116727.51 bytes /sec total size is 104857600 speedup is 1.00 |
6、限速100m
下载速度
| 1 2 3 4 5 6 7 8 | 14:37:18 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 11.35MB /s 0:00:08 (xfer #1, to-check=0/1) sent 30 bytes received104892888 bytes 8391433.44 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:37:39 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 13.19MB /s 0:00:07 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 11041359.37 bytes /sec total size is 104857600 speedup is 1.00 |
7、限速150m
下载速度
| 1 2 3 4 5 6 7 8 | 14:38:39 # rsync -avz --progress 172.16.1.126:/tmp/test_client.iso/tmp/ root@172.16.1.126's password: receiving incremental file list test_client.iso 104857600 100% 11.35MB /s 0:00:08 (xfer #1, to-check=0/1) sent 30 bytes received 104892888bytes 11041359.79 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 14:38:56 # rsync -avz --progress /tmp/test_server.iso172.16.1.126:/tmp/ root@172.16.1.126's password: sending incremental file list test_server.iso 104857600 100% 13.14MB /s 0:00:07 (xfer #1, to-check=0/1) sent 104892883 bytes received31 bytes 11041359.37 bytes /sec total size is 104857600 speedup is 1.00 |
下面是使用sas 7.5k硬盘的测试结果
8、无限速的
下载速度
| 1 2 3 4 5 6 7 8 | 17:11:07 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 2147483648 100% 106.42MB /s 0:00:19 (xfer #1, to-check=0/1) sent 30 bytes received 2148204633 bytes 95475762.80 bytes /sec total size is 2147483648 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:20:06 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 2147483648 100% 112.51MB /s 0:00:18 (xfer #1, to-check=0/1) sent 2148204628 bytes received 31 bytes 95475762.62 bytes /sec total size is 2147483648 speedup is 1.00 |
9、5m限速
下载速度
| 1 2 3 4 5 6 7 8 | 17:30:30 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 591.44kB /s 0:02:53 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 587635.39 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:38:57 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 590.35kB /s 0:02:53 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 574755.69 bytes /sec total size is 104857600 speedup is 1.00 |
10、限速10m
下载速度
| 1 2 3 4 5 6 7 8 | 17:42:54 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 1.15MB /s 0:01:26 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 1146370.69 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:44:31 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 1.16MB /s 0:01:26 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 1146370.64 bytes /sec total size is 104857600 speedup is 1.00 |
11、限速20m
下载速度
| 1 2 3 4 5 6 7 8 | 17:47:02 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 2.32MB /s 0:00:43 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 2162740.58 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:48:06 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 2.38MB /s 0:00:42 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 2255761.59 bytes /sec total size is 104857600 speedup is 1.00 |
12、限速50m
下载速度
| 1 2 3 4 5 6 7 8 | 17:52:52 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 5.84MB /s 0:00:17 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 5116727.71 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:53:15 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 6.10MB /s 0:00:16 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 5993880.80 bytes /sec total size is 104857600 speedup is 1.00 |
13、限速为100m
下载速度
| 1 2 3 4 5 6 7 8 | 17:55:16 # rsync -avz --progress172.16.1.53:/tmp/test_client.iso /tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 11.75MB /s 0:00:08 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 9989801.71 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:55:39 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 12.72MB /s 0:00:07 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 11041359.37 bytes /sec total size is 104857600 speedup is 1.00 |
14、限速150m
下载速度
| 1 2 3 4 5 6 7 8 | 17:56:58 # rsync -avz --progress 172.16.1.53:/tmp/test_client.iso/tmp/ root@172.16.1.53's password: receiving incremental file list test_client.iso 104857600 100% 17.69MB /s 0:00:05 (xfer #1, to-check=0/1) sent 30 bytes received 104892888 bytes 13985722.40 bytes /sec total size is 104857600 speedup is 1.00 |
上传速度
| 1 2 3 4 5 6 7 8 | 17:57:18 # rsync -avz --progress/tmp/test_server.iso 172.16.1.53:/tmp/ root@172.16.1.53's password: sending incremental file list test_server.iso 104857600 100% 20.44MB /s 0:00:04 (xfer #1, to-check=0/1) sent 104892883 bytes received 31 bytes 16137371.38 bytes /sec total size is 104857600 speedup is 1.00 |
如果大家有问题可以留言给我，我会及时回复。
本文转自 reinxu 51CTO博客，原文链接：http://blog.51cto.com/dl528888/1641569，如需转载请自行联系原作者