linux上TCP connection timeout问题解决办法
<p><strong> linux上TCP connection timeout问题解决办法</strong></p>
<p>
最近在产线上经常出现connection timeout的问题，先看看Java 中关于connection timeout 的异常如何产生</p>
<p>
<strong>JAVA中的timeout</strong></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_471233">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">java.net.SocketTimeoutException: connect timed out </code>
</div>
<div class="line number2 index1 alt1">
<code class="java plain">客户端异常:connect timed out </code>
</div>
<div class="line number3 index2 alt2">
<code class="java spaces"> </code><code class="java plain">at java.net.PlainSocketImpl.socketConnect(Native Method) </code>
</div>
<div class="line number4 index3 alt1">
<code class="java spaces"> </code><code class="java plain">at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:</code><code class="java value">345</code><code class="java plain">) </code>
</div>
<div class="line number5 index4 alt2">
<code class="java spaces"> </code><code class="java plain">at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:</code><code class="java value">206</code><code class="java plain">) </code>
</div>
<div class="line number6 index5 alt1">
<code class="java spaces"> </code><code class="java plain">at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:</code><code class="java value">188</code><code class="java plain">) </code>
</div>
<div class="line number7 index6 alt2">
<code class="java spaces"> </code><code class="java plain">at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:</code><code class="java value">392</code><code class="java plain">) </code>
</div>
<div class="line number8 index7 alt1">
<code class="java spaces"> </code><code class="java plain">at java.net.Socket.connect(Socket.java:</code><code class="java value">589</code><code class="java plain">) </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
我们能经常看到的connect timed out异常产生，看一下java 是如何生成这个异常</p>
<p>
<strong>plainsocketimpl.c 中</strong></p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_739277">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
<div class="line number15 index14 alt2">
15</div>
<div class="line number16 index15 alt1">
16</div>
<div class="line number17 index16 alt2">
17</div>
<div class="line number18 index17 alt1">
18</div>
<div class="line number19 index18 alt2">
19</div>
<div class="line number20 index19 alt1">
20</div>
<div class="line number21 index20 alt2">
21</div>
<div class="line number22 index21 alt1">
22</div>
<div class="line number23 index22 alt2">
23</div>
<div class="line number24 index23 alt1">
24</div>
<div class="line number25 index24 alt2">
25</div>
<div class="line number26 index25 alt1">
26</div>
<div class="line number27 index26 alt2">
27</div>
<div class="line number28 index27 alt1">
28</div>
<div class="line number29 index28 alt2">
29</div>
<div class="line number30 index29 alt1">
30</div>
<div class="line number31 index30 alt2">
31</div>
<div class="line number32 index31 alt1">
32</div>
<div class="line number33 index32 alt2">
33</div>
<div class="line number34 index33 alt1">
34</div>
<div class="line number35 index34 alt2">
35</div>
<div class="line number36 index35 alt1">
36</div>
<div class="line number37 index36 alt2">
37</div>
<div class="line number38 index37 alt1">
38</div>
<div class="line number39 index38 alt2">
39</div>
<div class="line number40 index39 alt1">
40</div>
<div class="line number41 index40 alt2">
41</div>
<div class="line number42 index41 alt1">
42</div>
<div class="line number43 index42 alt2">
43</div>
<div class="line number44 index43 alt1">
44</div>
<div class="line number45 index44 alt2">
45</div>
<div class="line number46 index45 alt1">
46</div>
<div class="line number47 index46 alt2">
47</div>
<div class="line number48 index47 alt1">
48</div>
<div class="line number49 index48 alt2">
49</div>
<div class="line number50 index49 alt1">
50</div>
<div class="line number51 index50 alt2">
51</div>
<div class="line number52 index51 alt1">
52</div>
<div class="line number53 index52 alt2">
53</div>
<div class="line number54 index53 alt1">
54</div>
<div class="line number55 index54 alt2">
55</div>
<div class="line number56 index55 alt1">
56</div>
<div class="line number57 index56 alt2">
57</div>
<div class="line number58 index57 alt1">
58</div>
<div class="line number59 index58 alt2">
59</div>
<div class="line number60 index59 alt1">
60</div>
<div class="line number61 index60 alt2">
61</div>
<div class="line number62 index61 alt1">
62</div>
<div class="line number63 index62 alt2">
63</div>
<div class="line number64 index63 alt1">
64</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash keyword">while</code> <code class="bash plain">(1) { </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash spaces"> </code><code class="bash plain">jlong newTime; </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash comments">#ifndef USE_SELECT </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash spaces"> </code><code class="bash plain">{ </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash spaces"> </code><code class="bash plain">struct pollfd pfd; </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash spaces"> </code><code class="bash plain">pfd.fd = fd; </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash spaces"> </code><code class="bash plain">pfd.events = POLLOUT; </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash spaces"> </code><code class="bash plain">errno = 0; </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash spaces"> </code><code class="bash plain">connect_rv = NET_Poll(&pfd, 1, timeout); </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash comments">#else </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash spaces"> </code><code class="bash plain">{ </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash spaces"> </code><code class="bash plain">fd_set wr, ex; </code>
</div>
<div class="line number15 index14 alt2">
<code class="bash spaces"> </code><code class="bash plain">struct timeval t; </code>
</div>
<div class="line number16 index15 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number17 index16 alt2">
<code class="bash spaces"> </code><code class="bash plain">t.tv_sec = timeout / 1000; </code>
</div>
<div class="line number18 index17 alt1">
<code class="bash spaces"> </code><code class="bash plain">t.tv_usec = (timeout % 1000) * 1000; </code>
</div>
<div class="line number19 index18 alt2">
<code class="bash spaces"> </code>
</div>
<div class="line number20 index19 alt1">
<code class="bash spaces"> </code><code class="bash plain">FD_ZERO(&wr); </code>
</div>
<div class="line number21 index20 alt2">
<code class="bash spaces"> </code><code class="bash plain">FD_SET(fd, &wr); </code>
</div>
<div class="line number22 index21 alt1">
<code class="bash spaces"> </code><code class="bash plain">FD_ZERO(&ex); </code>
</div>
<div class="line number23 index22 alt2">
<code class="bash spaces"> </code><code class="bash plain">FD_SET(fd, &ex); </code>
</div>
<div class="line number24 index23 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number25 index24 alt2">
<code class="bash spaces"> </code><code class="bash plain">errno = 0; </code>
</div>
<div class="line number26 index25 alt1">
<code class="bash spaces"> </code><code class="bash plain">connect_rv = NET_Select(fd+1, 0, &wr, &ex, &t); </code>
</div>
<div class="line number27 index26 alt2">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
<div class="line number28 index27 alt1">
<code class="bash comments">#endif </code>
</div>
<div class="line number29 index28 alt2">
<code class="bash spaces"> </code>
</div>
<div class="line number30 index29 alt1">
<code class="bash spaces"> </code><code class="bash keyword">if</code> <code class="bash plain">(connect_rv >= 0) { </code>
</div>
<div class="line number31 index30 alt2">
<code class="bash spaces"> </code><code class="bash keyword">break</code><code class="bash plain">; </code>
</div>
<div class="line number32 index31 alt1">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
<div class="line number33 index32 alt2">
<code class="bash spaces"> </code><code class="bash keyword">if</code> <code class="bash plain">(errno != EINTR) { </code>
</div>
<div class="line number34 index33 alt1">
<code class="bash spaces"> </code><code class="bash keyword">break</code><code class="bash plain">; </code>
</div>
<div class="line number35 index34 alt2">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
<div class="line number36 index35 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number37 index36 alt2">
<code class="bash spaces"> </code><code class="bash plain">/* </code>
</div>
<div class="line number38 index37 alt1">
<code class="bash spaces"> </code><code class="bash plain">* The poll was interrupted so adjust timeout and </code>
</div>
<div class="line number39 index38 alt2">
<code class="bash spaces"> </code><code class="bash plain">* restart </code>
</div>
<div class="line number40 index39 alt1">
<code class="bash spaces"> </code><code class="bash plain">*/ </code>
</div>
<div class="line number41 index40 alt2">
<code class="bash spaces"> </code><code class="bash plain">newTime = JVM_CurrentTimeMillis(</code><code class="bash functions">env</code><code class="bash plain">, 0); </code>
</div>
<div class="line number42 index41 alt1">
<code class="bash spaces"> </code><code class="bash plain">timeout -= (newTime - prevTime); </code>
</div>
<div class="line number43 index42 alt2">
<code class="bash spaces"> </code><code class="bash keyword">if</code> <code class="bash plain">(timeout <= 0) { </code>
</div>
<div class="line number44 index43 alt1">
<code class="bash spaces"> </code><code class="bash plain">connect_rv = 0; </code>
</div>
<div class="line number45 index44 alt2">
<code class="bash spaces"> </code><code class="bash keyword">break</code><code class="bash plain">; </code>
</div>
<div class="line number46 index45 alt1">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
<div class="line number47 index46 alt2">
<code class="bash spaces"> </code><code class="bash plain">prevTime = newTime; </code>
</div>
<div class="line number48 index47 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number49 index48 alt2">
<code class="bash spaces"> </code><code class="bash plain">} /* </code><code class="bash keyword">while</code> <code class="bash plain">*/ </code>
</div>
<div class="line number50 index49 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number51 index50 alt2">
<code class="bash spaces"> </code><code class="bash keyword">if</code> <code class="bash plain">(connect_rv == 0) { </code>
</div>
<div class="line number52 index51 alt1">
<code class="bash spaces"> </code><code class="bash plain">JNU_ThrowByName(</code><code class="bash functions">env</code><code class="bash plain">, JNU_JAVANETPKG </code><code class="bash string">"SocketTimeoutException"</code><code class="bash plain">, </code>
</div>
<div class="line number53 index52 alt2">
<code class="bash spaces"> </code><code class="bash string">"connect timed out"</code><code class="bash plain">); </code>
</div>
<div class="line number54 index53 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number55 index54 alt2">
<code class="bash spaces"> </code><code class="bash plain">/* </code>
</div>
<div class="line number56 index55 alt1">
<code class="bash spaces"> </code><code class="bash plain">* Timeout out but connection may still be established. </code>
</div>
<div class="line number57 index56 alt2">
<code class="bash spaces"> </code><code class="bash plain">* At the high level it should be closed immediately but </code>
</div>
<div class="line number58 index57 alt1">
<code class="bash spaces"> </code><code class="bash plain">* just </code><code class="bash keyword">in</code> <code class="bash keyword">case</code> <code class="bash plain">we </code><code class="bash functions">make</code> <code class="bash plain">the socket blocking again and </code>
</div>
<div class="line number59 index58 alt2">
<code class="bash spaces"> </code><code class="bash plain">* </code><code class="bash functions">shutdown</code> <code class="bash plain">input & output. </code>
</div>
<div class="line number60 index59 alt1">
<code class="bash spaces"> </code><code class="bash plain">*/ </code>
</div>
<div class="line number61 index60 alt2">
<code class="bash spaces"> </code><code class="bash plain">SET_BLOCKING(fd); </code>
</div>
<div class="line number62 index61 alt1">
<code class="bash spaces"> </code><code class="bash plain">JVM_SocketShutdown(fd, 2); </code>
</div>
<div class="line number63 index62 alt2">
<code class="bash spaces"> </code><code class="bash keyword">return</code><code class="bash plain">; </code>
</div>
<div class="line number64 index63 alt1">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
这里可以看到在做connect的时候，是调用 NET_Poll 或者 NET_Select, 在linux 上就是使用 poll/select</p>
<p>
当发生timeout的时候connect_rv=0 ，这里有个注意点虽然在poll/select 是传入timeout的时间，但是这是会被打断的，connect_rv返回的值为-1 ，所以jvm里面重新计算了timeout , 确保timeout 的时间片已经运行完了，才推出循环。</p>
<div class="jb51code">
<div>
<div class="syntaxhighlightercpp" id="highlighter_632626">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="cpp plain">newTime = JVM_CurrentTimeMillis(env, 0); </code>
</div>
<div class="line number2 index1 alt1">
<code class="cpp spaces"> </code><code class="cpp plain">timeout -= (newTime - prevTime); </code>
</div>
<div class="line number3 index2 alt2">
<code class="cpp spaces"> </code><code class="cpp keyword bold">if</code> <code class="cpp plain">(timeout <= 0) { </code>
</div>
<div class="line number4 index3 alt1">
<code class="cpp spaces"> </code><code class="cpp plain">connect_rv = 0; </code>
</div>
<div class="line number5 index4 alt2">
<code class="cpp spaces"> </code><code class="cpp keyword bold">break</code><code class="cpp plain">; </code>
</div>
<div class="line number6 index5 alt1">
<code class="cpp spaces"> </code><code class="cpp plain">} </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
同时设置connect_rv 为0， 也是下面只有当connect_rv为0的时候才抛出connect timeout</p>
<p>
<strong>什么是connect timeout ? </strong></p>
<p>
也就是client 发出 syn 包，server端在你指定的时间内没有回复ack，poll/select 返回0</p>
<p>
server 端为什么没有回复ack, 因为syn包的回复是内核层的，要么网络层丢包，要么就是内核层back_log的queue满了，关于backlog在本片中就不详细描述了。</p>
<p>
当时查看产线上的连接最高能到1000多，同时查看了backlog 的queue的大小</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_109833">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">cat</code> <code class="bash plain">/proc/sys/net/ipv4/tcp_max_syn_backlog</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
有8192 在产线上没有这么多的客户端的连接，不可能backlog queue会满，虽然syn_backlog 的设置是8192 但并不代表服务器启动的时候设置成了8192，所以必须查这个端口所设置的backlog大小</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_132321">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain">ss -lt</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
看到Send-Q在8080端口是128 ，原来在服务器端启动listen 的时候设置了128的backlog</p>
<p>
查看tomcat 的配置，默认bio的设置</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_759673">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"><Connector executor=</code><code class="bash string">"tomcatThreadPool"</code>
</div>
<div class="line number2 index1 alt1">
<code class="bash spaces"> </code><code class="bash plain">port=</code><code class="bash string">"8080"</code>
</div>
<div class="line number3 index2 alt2">
<code class="bash spaces"> </code><code class="bash plain">protocol=</code><code class="bash string">"HTTP/1.1"</code>
</div>
<div class="line number4 index3 alt1">
<code class="bash spaces"> </code><code class="bash plain">acceptCount=</code><code class="bash string">"5000"</code>
</div>
<div class="line number5 index4 alt2">
<code class="bash spaces"> </code><code class="bash plain">connectionTimeout=</code><code class="bash string">"25000"</code>
</div>
<div class="line number6 index5 alt1">
<code class="bash spaces"> </code><code class="bash plain">maxHttpHeaderSize=</code><code class="bash string">"8192"</code>
</div>
<div class="line number7 index6 alt2">
<code class="bash spaces"> </code><code class="bash plain">useBodyEncodingForURI=</code><code class="bash string">"true"</code>
</div>
<div class="line number8 index7 alt1">
<code class="bash spaces"> </code><code class="bash plain">enableLookups=</code><code class="bash string">"false"</code>
</div>
<div class="line number9 index8 alt2">
<code class="bash spaces"> </code><code class="bash plain">redirectPort=</code><code class="bash string">"8443"</code>
</div>
<div class="line number10 index9 alt1">
<code class="bash spaces"> </code><code class="bash plain">URIEncoding=</code><code class="bash string">"UTF-8"</code>
</div>
<div class="line number11 index10 alt2">
<code class="bash spaces"> </code><code class="bash plain">maxThreads=</code><code class="bash string">"500"</code>
</div>
<div class="line number12 index11 alt1">
<code class="bash spaces"> </code><code class="bash plain">maxKeepAliveRequests=</code><code class="bash string">"1000"</code>
</div>
<div class="line number13 index12 alt2">
<code class="bash spaces"> </code><code class="bash plain">keepAliveTimeout=</code><code class="bash string">"30000"</code>
</div>
<div class="line number14 index13 alt1">
<code class="bash spaces"> </code><code class="bash plain">/> </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
产线上已经设置了acceptCount, 默认是100 但是这里设置了是5000 ，这与通过ss看到的send-q的结果严重不符合<br>
通过内核代码分析,发现原来内核参数不仅仅是通过tcp_max_syn_backlog控制，同时也受somaxconn控制<br>
查看</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_521015">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash functions">cat</code> <code class="bash plain">/proc/sys/net/core/somaxconn</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
发现值是128, OK 原因找到了，修改/etc/sysctl.conf 添加</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterjava" id="highlighter_843091">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="java plain">net.core.somaxconn = </code><code class="java value">8192</code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
sysctl -f /etc/sysctl.conf 重新加载一下，这样就能改变全局了</p>
<p>
<strong>问题</strong>：是1000多个连接，500个工作线程，因为backlog的大小是受socket.accept控制的，我们通常境况下会单独起一个线程去serversocket.accept(),而当前server的load并不高，不因该会出现back_log queue出现满的情况,更何况只有1000多个连接，代码就是真相，查看tomcat的源码。</p>
<p>
原来accptor 线程在accept 之前，会去countUpOrWaitConnection 发现接受到的的socket数目大于设置的work线程数目的时候，会停止accept.</p>
<div class="jb51code">
<div>
<div class="syntaxhighlighterbash" id="highlighter_357149">
<div class="toolbar">
<span>?</span>
</div>
<table border="0" cellpadding="0" cellspacing="0"><tbody><tr>
<td class="gutter">
<div class="line number1 index0 alt2">
1</div>
<div class="line number2 index1 alt1">
2</div>
<div class="line number3 index2 alt2">
3</div>
<div class="line number4 index3 alt1">
4</div>
<div class="line number5 index4 alt2">
5</div>
<div class="line number6 index5 alt1">
6</div>
<div class="line number7 index6 alt2">
7</div>
<div class="line number8 index7 alt1">
8</div>
<div class="line number9 index8 alt2">
9</div>
<div class="line number10 index9 alt1">
10</div>
<div class="line number11 index10 alt2">
11</div>
<div class="line number12 index11 alt1">
12</div>
<div class="line number13 index12 alt2">
13</div>
<div class="line number14 index13 alt1">
14</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2">
<code class="bash plain"><strong>countUpOrAwaitConnection<</code><code class="bash plain">/strong</code><code class="bash plain">>(); </code>
</div>
<div class="line number2 index1 alt1">
<code class="bash spaces"> </code>
</div>
<div class="line number3 index2 alt2">
<code class="bash spaces"> </code><code class="bash plain">Socket socket = null; </code>
</div>
<div class="line number4 index3 alt1">
<code class="bash spaces"> </code><code class="bash plain">try { </code>
</div>
<div class="line number5 index4 alt2">
<code class="bash spaces"> </code><code class="bash plain">//</code> <code class="bash plain">Accept the next incoming connection from the server </code>
</div>
<div class="line number6 index5 alt1">
<code class="bash spaces"> </code><code class="bash plain">//</code> <code class="bash plain">socket </code>
</div>
<div class="line number7 index6 alt2">
<code class="bash spaces"> </code><code class="bash plain">socket = serverSocketFactory.acceptSocket(serverSocket); </code>
</div>
<div class="line number8 index7 alt1">
<code class="bash spaces"> </code><code class="bash plain">} catch (IOException ioe) { </code>
</div>
<div class="line number9 index8 alt2">
<code class="bash spaces"> </code><code class="bash plain">countDownConnection(); </code>
</div>
<div class="line number10 index9 alt1">
<code class="bash spaces"> </code><code class="bash plain">//</code> <code class="bash plain">Introduce delay </code><code class="bash keyword">if</code> <code class="bash plain">necessary </code>
</div>
<div class="line number11 index10 alt2">
<code class="bash spaces"> </code><code class="bash plain">errorDelay = handleExceptionWithDelay(errorDelay); </code>
</div>
<div class="line number12 index11 alt1">
<code class="bash spaces"> </code><code class="bash plain">//</code> <code class="bash plain">re-throw </code>
</div>
<div class="line number13 index12 alt2">
<code class="bash spaces"> </code><code class="bash plain">throw ioe; </code>
</div>
<div class="line number14 index13 alt1">
<code class="bash spaces"> </code><code class="bash plain">} </code>
</div>
</div>
</td>
</tr></tbody></table>
</div>
</div>
<div class="codetool" id="codetool">
<div class="code_n">
<textarea></textarea>
</div>
</div>
</div>
<p>
也就是说当并发超过628个连接以上，就有可能出现backlog queue满的情况，而出现connect timeout的情况，一切皆清楚了。</p>
<p>
感谢阅读，希望能帮助到大家，谢谢大家对本站的支持！</p>
<p>
原文链接：http://blog.csdn.net/raintungli/article/details/37879907</p>
頁:
[1]