症状
所有通过 HTTPS 访问的服务均无法正常加载页面,云主机后台查看监控发现公网流出带宽使用率 (%)
在最近十几天内持续占用超过100%。
实时查看服务器公网网口总流量进出情况
ifstat -i eth0 1
1表示刷新间隔,每秒刷新一次
实时查看按进程/接口带宽
nethogs
实时查看每个 Docker 容器的网络使用情况
iftop -i br-3303658042e6
br-xxxxx 是 Docker 创建的 bridge 网桥,流量通过网桥转发到对应的容器IP
汇总访问 443 端口的 IP
netstat -tn | grep ':443' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -nr
查看防火墙日志
cat /var/log/ufw.log | grep 57.141.2
Apr 14 05:00:35 iZwz917qy40up8vmsmqzsdZ kernel: [10779478.799321] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.11 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=42412 DF PROTO=TCP SPT=51358 DPT=443 WINDOW=3356 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:00:37 iZwz917qy40up8vmsmqzsdZ kernel: [10779480.794548] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.3 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=49073 DF PROTO=TCP SPT=41796 DPT=443 WINDOW=1633 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:00:49 iZwz917qy40up8vmsmqzsdZ kernel: [10779492.660113] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.7 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=28052 DF PROTO=TCP SPT=53678 DPT=443 WINDOW=340 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:01:12 iZwz917qy40up8vmsmqzsdZ kernel: [10779515.694527] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.11 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=9083 DF PROTO=TCP SPT=45386 DPT=443 WINDOW=457 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:01:30 iZwz917qy40up8vmsmqzsdZ kernel: [10779534.213447] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.14 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=56071 DF PROTO=TCP SPT=60330 DPT=443 WINDOW=906 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:02:24 iZwz917qy40up8vmsmqzsdZ kernel: [10779588.403633] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.18 DST=172.26.190.121 LEN=64 TOS=0x14 PREC=0x00 TTL=48 ID=26538 DF PROTO=TCP SPT=46088 DPT=443 WINDOW=855 RES=0x00 ACK FIN URGP=0
Apr 14 05:04:17 iZwz917qy40up8vmsmqzsdZ kernel: [10779701.084531] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.29 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=36256 DF PROTO=TCP SPT=35484 DPT=443 WINDOW=1409 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:04:19 iZwz917qy40up8vmsmqzsdZ kernel: [10779702.930070] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.17 DST=172.26.190.121 LEN=76 TOS=0x14 PREC=0x00 TTL=48 ID=62191 DF PROTO=TCP SPT=53468 DPT=443 WINDOW=1208 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:04:20 iZwz917qy40up8vmsmqzsdZ kernel: [10779704.195132] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.21 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=59138 DF PROTO=TCP SPT=53064 DPT=443 WINDOW=511 RES=0x00 ACK PSH FIN URGP=0
Apr 14 05:04:21 iZwz917qy40up8vmsmqzsdZ kernel: [10779704.570289] [UFW BLOCK] IN=eth0 OUT= MAC=00:16:3e:0e:67:13:ee:ff:ff:ff:ff:ff:08:00 SRC=57.141.2.1 DST=172.26.190.121 LEN=88 TOS=0x14 PREC=0x00 TTL=48 ID=4986 DF PROTO=TCP SPT=35226 DPT=443 WINDOW=928 RES=0x00 ACK PSH FIN URGP=0
解决问题
从上面的 nethogs 输出发现大量 57.141.2.0/24 来源 IP 段访问本机 443 端口,ifstat 实时输出显示出口带宽已经被打满。尝试停止 443 监听端口,也就是停止 nginx 服务,一切恢复正常,所以需要先封堵来源IP。
由于 nginx 作为容器运行,Docker 默认使用 iptables NAT 转发规则来把流量从宿主机转发进容器,容器的网络流量通常绕过宿主机的防火墙,直接通过 Docker 自己的 iptables NAT 转发机制。因此,UFW 规则可能不会拦截 Docker 的流量,所以下面这些规则并没有效果。
ufw insert 1 deny proto tcp from 57.141.2.0/24 to any port 443
ufw route deny proto tcp from 57.141.2.0/24 to any port 443
ufw reload
通过 Docker 自带的 DOCKER-USER 链来拦截流量
iptables -I DOCKER-USER -s 57.141.2.0/24 -j DROP
sudo iptables -L DOCKER-USER -v -n
安全加固
可选方式:
- 写个简单脚本,每分钟记录一次带宽占用高的进程/IP,并自动告警/封锁。
- 使用 fail2ban 限制异常访问频率(推荐)。
Fail2ban 针对 SSH 爆破和 HTTPS 端口扫描的防护
apt install -y fail2ban
创建针对 UFW 的配置文件 /etc/fail2ban/jail.d/ufw-custom.local
[ufw-ssh]
enabled = true
filter = ufw-ssh
logpath = /var/log/ufw.log
maxretry = 5
findtime = 300
bantime = 3600
action = ufw
[ufw-https]
enabled = true
filter = ufw-https
logpath = /var/log/ufw.log
maxretry = 10
findtime = 60
bantime = 3600
action = ufw
创建 SSH 过滤文件 /etc/fail2ban/filter.d/ufw-ssh.conf
[Definition]
failregex = UFW BLOCK.* SRC=<HOST>.* DPT=22
ignoreregex =
创建 HTTPS 过滤文件 /etc/fail2ban/filter.d/ufw-https.conf
[Definition]
failregex = UFW BLOCK.* SRC=<HOST>.* DPT=443
ignoreregex =
确认 UFW 日志已经开启
ufw logging on
tail -f /var/log/ufw.log
加载配置
systemctl restart fail2ban
fail2ban-client status
fail2ban-client status ufw-ssh
fail2ban-client status ufw-https
评论区