用 shell 脚本制造连接频繁中断的场景

  • A+
所属分类：linux技术
摘要

最近在准备客户端的新版本，在内部灰度过程中，发现一类奇怪的 dump，通过查看日志和堆栈，可以确定是因为每次连上后台就被后台断开了、导致多次重连后随机发生的崩溃。dump 和日志都无法提供进一步的信息来定位问题，本地又不复现，也没有办法去联系用户查看现场 (windows 服务崩溃后自动重启，用户不感知)。于是想到能不能自己制造这样的场景 —— 当连接建立后立即断开该连接 —— 看是否会复现崩溃。
便宜好用的国外VPS推荐

问题的提出

最近在准备客户端的新版本，在内部灰度过程中，发现一类奇怪的 dump，通过查看日志和堆栈，可以确定是因为每次连上后台就被后台断开了、导致多次重连后随机发生的崩溃。dump 和日志都无法提供进一步的信息来定位问题，本地又不复现，也没有办法去联系用户查看现场 (windows 服务崩溃后自动重启，用户不感知)。于是想到能不能自己制造这样的场景 —— 当连接建立后立即断开该连接 —— 看是否会复现崩溃。

问题的解决

tcpview

在 windows 上最直观的解决方案就是手动断开连接啦，拿出 sysinternal 工具集，翻出 tcpview，就可以看到系统上所有的 tcp 连接了：

用 shell 脚本制造连接频繁中断的场景

 

除了能看到建立的 tcp 连接所属进程、本地地址/端口号、远端地址/端口号外、连接状态外，还可以看到一些连接上的统计信息，如收发包数和字节数等。高亮的那一行就是我想要杀掉的连接。在 tcpview 里杀连接很简单，直接右键菜单 ‘Close Connection’ 即可。但是这样做的问题是，每次从看到连接到杀死连接要经历一定时间 (手动操作)，甚至进程日志已经显示连接建立了，tcpview 还没有刷出来，总而言之就是一个字 —— 慢，杀了十几次，挂上调试器的进程纹丝不动，一点要崩溃的迹象也没有 (关键是还手疼)，汗~

tcpkill

都 2020 年了，自动化工具用起来，查了一下，tcpview 除了界面外没有提供类似命令行的功能，于是只能在网上搜 “什么命令能杀掉 tcp 连接” 了，百度到一个 tcpkill，这个命令是原生于 linux 的，需要先安装 dsniff 工具包

$ sudo yum install dsniff

安装成功后就可以实操一下了

$ tcpkill Version: 2.4 Usage: tcpkill [-i interface] [-1..9] expression

help 和 man 都太过简单，其实重点就是最后这个参数 expression，貌似是使用和 tcpdump 相同的格式，为了验证 tcpkill，我先搭建了一个简单的测试环境：

$ nc -4 -l -p 5555

使用 nc 创建一个在 5555 端口监听的进程；

$ nc -4 localhost 5555 -p 6666

在另外一个终端中创建一个进程去连接 5555 端口，它自己的端口是 6666；

$ netstat -antp (Not all processes could be identified, non-owned process info  will not be shown, you would have to be root to see it all.) Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name     ……             tcp        0      0 127.0.0.1:5555          127.0.0.1:6666          ESTABLISHED 2451/nc              tcp        0      0 127.0.0.1:6666          127.0.0.1:5555          ESTABLISHED 2470/nc  ……

可以通过 netstat 命令查看到建立的这个连接及其端口号。下面用 tcpkill 杀掉这个连接，这里使用指定 6666 端口的方式：

$ sudo tcpkill -i lo port 6666 tcpkill: listening on lo [port 6666] 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486680909:18446744073486680909(0) win 0 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486681251:18446744073486681251(0) win 0 127.0.0.1:6666 > 127.0.0.1:5555: R 18446744073486681935:18446744073486681935(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881223293:18446744071881223293(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881223635:18446744071881223635(0) win 0 127.0.0.1:5555 > 127.0.0.1:6666: R 18446744071881224319:18446744071881224319(0) win 0

 

打印了一堆莫名其妙的信息，看 netstat 输出的话，那个连接还是 ESTABLISHED 状态，但是通过在控制台输入一些字符 (例如 abc) 来发送数据后，连接中断、进程退出

$ nc -4 localhost 5555 -p 6666 hello world abc Ncat: Connection reset by peer.

在 5555 端口监听的进程由于连接中断也自动退出了

$ nc -4 -l -p 5555 hello world abc

此时再看 netstat 输出，就看不到这条连接的相关信息了。看相关文章，貌似是这个命令向连接发送了伪造的 rst 包，所以只有当下次客户端再请求时，才会发现连接已经中断了。而且 tcpkill 好像会一直运行，只要它发现在 6666 这个端口建立了连接，就会去尝试中断。虽然后面这个特性挺好，但是连接只有在下一次发送数据时才能检测到中断这事，实效性差那么点儿意思；最麻烦的是我在 windows 的 msys2 环境中，没有这个命令可用，看来这个命令依赖的一些 linux 底层机制在 win32 上不太好实现，于是果断放弃。

cports

下面百度的重点就放在了 “windows 上可以杀掉 tcp 连接的命令” 了，结果还真被找到一个 —— CurrPorts，它本身是个 UI 工具，界面和 tcpview 很类似：

用 shell 脚本制造连接频繁中断的场景

 

输出的信息大同小异，都是进程、协议、本地地址/端口、远程地址/端口、连接状态等 (其实还少了一些连接上的统计信息)。然后关闭连接也是通过选中项目后右键菜单来实现的：

用 shell 脚本制造连接频繁中断的场景

 

光看菜单的话，感觉比 tcpview 功能丰富多了，比如光选项就有这么多：

用 shell 脚本制造连接频繁中断的场景

 

确实比 tcpview 要好用一些，但是使用右键菜单来关闭连接，貌似和之前没有多大分别呢 (虽然可以使用 Ctrl+T 快捷方式)。不要急，下面着重说一下 CurrPorts 的命令行参数，这是区别于 tcpview 的一大优势：

/stext <Filename> Save the list of all opened TCP/UDP ports into a regular text file.
/stab <Filename> Save the list of all opened TCP/UDP ports into a tab-delimited text file.
/scomma <Filename> Save the list of all opened TCP/UDP ports into a comma-delimited text file.
/stabular <Filename> Save the list of all opened TCP/UDP ports into a tabular text file.
/shtml <Filename> Save the list of all opened TCP/UDP ports into HTML file (Horizontal).
/sverhtml <Filename> Save the list of all opened TCP/UDP ports into HTML file (Vertical).
/sxml <Filename> Save the list of all opened TCP/UDP ports to XML file.
/CaptureTime <Milliseconds> Specifies the capture time in milliseconds for the save command-line options (/stext, /stab, /scomma, and so on...)
Example:
cports.exe /RunAsAdmin /scomma c:tempports1.csv /CaptureTime 15000
/RunAsAdmin Runs CurrPorts as Administrator.
/sort <column> This command-line option can be used with other save options for sorting by the desired column. If you don't specify this option, the list is sorted according to the last sort that you made from the user interface. The <column> parameter can specify the column index (0 for the first column, 1 for the second column, and so on) or the name of the column, like "Remote Port" and "Remote Address". You can specify the '~' prefix character (e.g: "~Remote Address") if you want to sort in descending order. You can put multiple /sort in the command-line if you want to sort by multiple columns.

Examples:
cports.exe /shtml "f:temp1.html" /sort 2 /sort ~1
cports.exe /shtml "f:temp1.html" /sort "Protocol" /sort "~Remote Address"

/nosort When you specify this command-line option, the list will be saved without any sorting.
/filter <filter string> Start CurrPorts with the specified filters. If you want to specify more than one filter, use the ';' character as a delimiter.
/cfg <cfg filename> Start CurrPorts with the specified config file.
/MarkPorts
/DisplayUdpPorts
/DisplayTcpPorts
/DisplayClosedPorts
/MarkNewModifiedPorts
/SortOnAutoRefresh
/AlwaysOnTop
/AskBefore
/DisplayIPv6Ports
/DisplayListening
/DisplayEstablished
/DisplayNoState
/DisplayNoRemoteIP
/ResolveAddresses
/RememberLastFilter
/DisplayPortInAddress
/AutoRefresh,
/ShowInfoTip
/TrayIcon
/TrayIconOneClick
/StartAsHidden
/LogChanges
/LogFilename
/DisabledFilters
/AddExportHeaderLine 		You can use all these parameters to control the options that are available under the Options and View menus.
For example, if you want to start CurrPorts with 'Display UDP Ports' turned off and 'Display Closed' turned on:
cports.exe /DisplayUdpPorts 0 /DisplayClosedPorts 1

You can also use these parameters in conjunction with all save parameters. For example: If you want to save into tab-delimited file only the UDP ports:
cports.exe /DisplayUdpPorts 1 /DisplayTcpPorts 0 /stab "c:tempudp.txt"

 

看了一圈儿，好像都是将结果保存到文件的一些选项，还好下面有一段话是专门说明如何关闭连接的：

Closing a Connection From Command-Line Starting from version 1.09, you can close one or more connections from command-line, by using /close parameter. The syntax of /close command: /close <Local Address> <Local Port> <Remote Address> <Remote Port> {Process Name/ID}  For each parameter, you can specify "*" in order to include all ports or addresses. The process name is an optional parameter. If you specify a process, only the ports of the specified process will be closed. Examples:      Close all connections with remote port 80 and remote address 192.168.1.10:     /close * * 192.168.1.10 80     Close all connections with remote port 80 (for all remote addresses):     /close * * * 80     Close all connections to remote address 192.168.20.30:     /close * * 192.168.20.30 *     Close all connections with local port 80:     /close * 80 * *     Close all connections of Firefox with remote port 80:     /close * * * 80 firefox.exe     Close all connections of the process that its ID is 3276:     /close * * * * 3276

 

给出了丰富的示例，例如针对我们的场景，可以这样调用 cports (UI 叫 CurrPorts，但命令名为 cports.exe)：

 cports /close * * xxx.xxx.xx.xx 3570 gdphost.exe

其中 xxx 部分是连接的远端 IP 地址，3570 是远端端口，gdphost.exe 是发起连接的进程名。当 cmd 以管理员权限运行，上面的调用是可以杀死连接的，但是在 msys2 bash (其实就是 git bash 啦) 中运行却直接启动了 CurrPorts 工具的 UI 界面。问题可能出在 /close 被 bash 作了转义识别上，用双引号将它们包含一下：

cports "/close" "*" "*" "xxx.xxx.xx.xx" "3570" "gdphost.exe"

结果还是不行，经过一翻研究，改成下面这样就可以了：

cports "//close" "*" "*" "xxx.xxx.xx.xx" "3570" "gdphost.exe"

在 msys2 bash 中要对 win32 命令的选项开始符 '/' 使用转义符前缀，否则 bash 会认为 /close 是一个目录(?)而非参数，从而进行某种转换(?)。又研究了一下双引号的作用，发现对于星号还是必需加上的，其它的参数可以不加，于是最后版本就成了这样：

cports "//close" "*" "*" xxx.xxx.xx.xx 3570 gdphost.exe

注意 cports 所在目录我是添加到了 Path 环境变量，所以可以这样直接写命令，否则必需提供 cports 的完整路径。另外这个 bash 也必需以管理员权限启动，不然命令虽然可以返回，但是没有发生任何实质性影响。

完整脚本

有了 cports 的底层支持，我们就可以这样写脚本来自动断开连接“伪造”事发现场啦：

 1 #! /bin/sh  2 while true  3 do  4   n=$(netstat -ano | grep 3570 | wc -l)  5   if [ $n -gt 0 ]; then  6     cports "//close" "*" "*" "*" 3570 gdphost.exe  7     echo "close connection"  8   else   9     echo "no connection find" 10   fi 11 done

 

加了一点简单的逻辑，在运行循环中首先通过 netstat 判断有没有在 3570 端口建立的连接，如果有就调用 cports 去断开；如果没有就继续循环，直到 Ctrl+C 结束。脚本运行起来后，结合图形界面、可以看到这个进程到后台的连接在不停的断开重连，下面是观察到脚本的一些输出 (内容较多、展开慎重)：

$ ./kill.sh close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find no connection find close connection no connection find no connection find no connection find ^C

 

录取了大概一千行的输出，平均每 80 行可以观察到一次中断连接的记录，也就是说 netstat 运行 80 次左右才能轮到一次 cports，这和进程在连接中断后 5 秒进行第一次重试有关 (通过换算，netstat 一秒运行了 16 次?)，而为了“及时”杀掉进程这里也没有采用 sleep 去避免忙等待 (其实可以等待 4 秒)。

结语

今天通过一个实际场景来研究了一下如何使用 shell + cports 不断的杀死某个连接、进而构造一个 bug 复现的场景。虽然脚本写的很漂亮，但遗憾的是这个 bug 未能复现 (泪奔~)，我用调试器挂上 win32 进程跑了四个多小时也没有出现崩溃。再仔细对比 log 输出，发现伪造的场景下进程还是可以有机会输出更多信息，看来“杀”的还是不够快啊；于是我将 netstat 检测 3570 端口是否存在这步去掉了，直接在循环里调用 cports 不断的杀连接，这应该比之前快了吧，但是还是没有发生崩溃；所以我感觉下一步只能用 c++ 写个程序，模拟在 3570 端口侦听、并在连接一上来的时候就 close connection 试试了 (需要设置 host 以便将域名指向本地启动的这个模拟程序)。

虽然没能复现 bug，但是无意间得到了 cports 这个宝贝，另外它的作者 nir sofer 也是一个 windows 工具小达人，制作了一系列 win32 工具供网友免费下载使用，有兴趣的读者可以去他的网站上翻翻。

参考

[1]. Windows Sysinternals

[2]. Tcpkill

[3]. How to kill a particular tcp connection in windows?

[4]. CPorts

[5]. NirSoft

便宜好用的国外VPS推荐