kubernetes核心组件kube-proxy
<p><span style="font-size: 18pt; color: rgba(0, 0, 255, 1)"><strong>一. kube-proxy 和 service </strong></span></p><p>kube-proxy是Kubernetes的核心组件，部署在每个Node节点上，它是实现Kubernetes Service的通信与负载均衡机制的重要组件; kube-proxy负责为Pod创建代理服务，从apiserver获取所有server信息，并根据server信息创建代理服务，实现server到Pod的请求路由和转发，从而实现K8s层级的虚拟转发网络。</p>
<p>在k8s中，提供相同服务的一组pod可以抽象成一个service，通过service提供的统一入口对外提供服务，每个service都有一个虚拟IP地址（VIP）和端口号供客户端访问。<span style="color: rgba(153, 51, 102, 1)">kube-proxy存在于各个node节点上，主要用于Service功能的实现，具体来说，就是实现集群内的客户端pod访问service，或者是集群外的主机通过NodePort等方式访问service。在当前版本的k8s中，kube-proxy默认使用的是iptables模式，通过各个node节点上的iptables规则来实现service的负载均衡，但是随着service数量的增大，iptables模式由于线性查找匹配、全量更新等特点，其性能会显著下降。<span style="color: rgba(0, 0, 128, 1)">从k8s的1.8版本开始，kube-proxy引入了IPVS模式，IPVS模式与iptables同样基于Netfilter，但是采用的hash表，因此当service数量达到一定规模时，hash查表的速度优势就会显现出来，从而提高service的服务性能。</span></span></p>
<p><span style="color: rgba(153, 51, 0, 1)">kube-proxy负责为Service提供cluster内部的服务发现和负载均衡，它运行在每个Node计算节点上，负责Pod网络代理, 它会定时从etcd服务获取到service信息来做相应的策略，维护网络规则和四层负载均衡工作。在K8s集群中微服务的负载均衡是由Kube-proxy实现的，它是K8s集群内部的负载均衡器，也是一个分布式代理服务器，在K8s的每个节点上都有一个，这一设计体现了它的伸缩性优势，需要访问服务的节点越多，提供负载均衡能力的Kube-proxy就越多，高可用节点也随之增多。</span></p>
<p><span style="color: rgba(153, 51, 0, 1)">service是一组pod的服务抽象，相当于一组pod的LB，负责将请求分发给对应的pod。service会为这个LB提供一个IP，一般称为cluster IP。<span style="color: rgba(153, 51, 102, 1)">kube-proxy的作用主要是负责service的实现，具体来说，就是实现了内部从pod到service和外部的从node port向service的访问。</span></span></p>
<p>简单来说: <br><span><strong>-></strong> kube-proxy其实就是管理service的访问入口，包括集群内Pod到Service的访问和集群外访问service。<br><strong><span>-><span> </span></span></strong>kube-proxy管理sevice的Endpoints，该service对外暴露一个Virtual IP，也成为Cluster IP, 集群内通过访问这个Cluster IP:Port就能访问到集群内对应的serivce下的Pod。<br><span><strong>-></strong> service是通过Selector选择的一组Pods的服务抽象，其实就是一个微服务，提供了服务的LB和反向代理的能力，而kube-proxy的主要作用就是负责service的实现。<br><span><strong>-> </strong>service另外一个重要作用是，一个服务后端的Pods可能会随着生存灭亡而发生IP的改变，service的出现，给服务提供了一个固定的IP，而无视后端Endpoint的变化。</span></span></span></p>
<p>举个例子，比如现在有podA，podB，podC和serviceAB。serviceAB是podA，podB的服务抽象(service)。那么kube-proxy的作用就是可以将pod(不管是podA，podB或者podC)向serviceAB的请求，进行转发到service所代表的一个具体pod(podA或者podB)上。<span style="color: rgba(153, 51, 102, 1)">请求的分配方法一般分配是采用轮询方法进行分配。另外，<span style="color: rgba(0, 0, 128, 1)">kubernetes还提供了一种在node节点上暴露一个端口，从而提供从外部访问service的方式。比如这里使用这样的一个manifest来创建service</span></span></p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_107537" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
<div class="line number3 index2 alt2">3</div>
<div class="line number4 index3 alt1">4</div>
<div class="line number5 index4 alt2">5</div>
<div class="line number6 index5 alt1">6</div>
<div class="line number7 index6 alt2">7</div>
<div class="line number8 index7 alt1">8</div>
<div class="line number9 index8 alt2">9</div>
<div class="line number10 index9 alt1">10</div>
<div class="line number11 index10 alt2">11</div>
<div class="line number12 index11 alt1">12</div>
<div class="line number13 index12 alt2">13</div>
<div class="line number14 index13 alt1">14</div>
<div class="line number15 index14 alt2">15</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">apiVersion: v1</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">kind: Service</code></div>
<div class="line number3 index2 alt2"><code class="bash plain">metadata:</code></div>
<div class="line number4 index3 alt1"><code class="bash spaces"> </code><code class="bash plain">labels:</code></div>
<div class="line number5 index4 alt2"><code class="bash spaces"> </code><code class="bash plain">name: mysql</code></div>
<div class="line number6 index5 alt1"><code class="bash spaces"> </code><code class="bash plain">role: service</code></div>
<div class="line number7 index6 alt2"><code class="bash spaces"> </code><code class="bash plain">name: mysql-service</code></div>
<div class="line number8 index7 alt1"><code class="bash plain">spec:</code></div>
<div class="line number9 index8 alt2"><code class="bash spaces"> </code><code class="bash plain">ports:</code></div>
<div class="line number10 index9 alt1"><code class="bash spaces"> </code><code class="bash plain">- port: 3306</code></div>
<div class="line number11 index10 alt2"><code class="bash spaces"> </code><code class="bash plain">targetPort: 3306</code></div>
<div class="line number12 index11 alt1"><code class="bash spaces"> </code><code class="bash plain">nodePort: 30964</code></div>
<div class="line number13 index12 alt2"><code class="bash spaces"> </code><code class="bash functions">type</code><code class="bash plain">: NodePort</code></div>
<div class="line number14 index13 alt1"><code class="bash spaces"> </code><code class="bash plain">selector:</code></div>
<div class="line number15 index14 alt2"><code class="bash spaces"> </code><code class="bash plain">mysql-service: </code><code class="bash string">"true"</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>上面配置的含义是在node上暴露出30964端口。当访问node上的30964端口时，其请求会转发到service对应的cluster IP的3306端口，并进一步转发到pod的3306端口。</p>
<p><span style="font-size: 15px"><strong><span style="color: rgba(255, 102, 0, 1)">Service, Endpoints与Pod的关系</span></strong></span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201906/907596-20190628121852780-440182082.png"></p>
<p><span style="color: rgba(0, 128, 0, 1)"><strong>Kube-proxy进程获取每个Service的Endpoints,实现Service的负载均衡功能</strong></span></p>
<p><span style="font-size: 16px"><strong><span style="color: rgba(255, 102, 0, 1)">Service的负载均衡转发规则</span></strong></span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201906/907596-20190628135357260-2132881445.png"></p>
<p><strong><span style="color: rgba(0, 128, 0, 1)">访问Service的请求，不论是Cluster
IP+TargetPort的方式；还是用Node节点IP+NodePort的方式，都被Node节点的Iptables规则重定向到Kube-proxy监听Service服务代理端口。kube-proxy接收到Service的访问请求后，根据负载策略，转发到后端的Pod。</span></strong></p>
<p><span style="font-size: 18pt; color: rgba(0, 0, 255, 1)"><strong>二. kubernetes服务发现</strong><br>Kubernetes提供了两种方式进行服务发现, 即<span style="color: rgba(255, 0, 255, 1)"><strong>环境变量</strong>和<strong><span style="color: rgba(255, 0, 255, 1)">DNS</span></strong>, 简单说明如下:</span></span></p>
<p>1) <strong><span style="color: rgba(255, 0, 255, 1)">环境变量</span></strong>： 当你创建一个Pod的时候，kubelet会在该Pod中注入集群内所有Service的相关环境变量。<span style="color: rgba(255, 0, 0, 1)"><strong><span>需要注意: </span></strong><span style="color: rgba(51, 51, 153, 1)">要想一个Pod中注入某个Service的环境变量，则必须Service要先比该Pod创建。这一点，几乎使得这种方式进行服务发现不可用。比如，一个ServiceName为redis-master的Service，对应的ClusterIP:Port为172.16.50.11:6379，则其对应的环境变量为:</span></span></p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_601165" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
<div class="line number3 index2 alt2">3</div>
<div class="line number4 index3 alt1">4</div>
<div class="line number5 index4 alt2">5</div>
<div class="line number6 index5 alt1">6</div>
<div class="line number7 index6 alt2">7</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">REDIS_MASTER_SERVICE_HOST=172.16.50.11</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">REDIS_MASTER_SERVICE_PORT=6379</code></div>
<div class="line number3 index2 alt2"><code class="bash plain">REDIS_MASTER_PORT=tcp:</code><code class="bash plain">//172</code><code class="bash plain">.16.50.11:6379</code></div>
<div class="line number4 index3 alt1"><code class="bash plain">REDIS_MASTER_PORT_6379_TCP=tcp:</code><code class="bash plain">//172</code><code class="bash plain">.16.50.11:6379</code></div>
<div class="line number5 index4 alt2"><code class="bash plain">REDIS_MASTER_PORT_6379_TCP_PROTO=tcp</code></div>
<div class="line number6 index5 alt1"><code class="bash plain">REDIS_MASTER_PORT_6379_TCP_PORT=6379</code></div>
<div class="line number7 index6 alt2"><code class="bash plain">REDIS_MASTER_PORT_6379_TCP_ADDR=172.16.50.11</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>2)<strong><span style="color: rgba(255, 0, 255, 1)"> DNS</span></strong>：这是k8s官方强烈推荐的方式!!! 可以通过cluster add-on方式轻松的创建KubeDNS来对集群内的Service进行服务发现。</p>
<p><span style="font-size: 18pt"><strong><span style="color: rgba(0, 0, 255, 1)">三. kubernetes发布(暴露)服务</span></strong><br>kubernetes原生的，一个Service的ServiceType决定了其发布服务的方式。<br><span><strong>-> </strong>ClusterIP：这是k8s默认的ServiceType。通过集群内的ClusterIP在内部发布服务。<br><span><strong>-> </strong>NodePort：这种方式是常用的，用来对集群外暴露Service，你可以通过访问集群内的每个NodeIP:NodePort的方式，访问到对应Service后端的Endpoint。<br><span><strong>-> </strong>LoadBalancer: 这也是用来对集群外暴露服务的，不同的是这需要Cloud Provider的支持，比如AWS等。<br><span><strong>-> </strong>ExternalName：这个也是在集群内发布服务用的，需要借助KubeDNS(version >= 1.7)的支持，就是用KubeDNS将该service和ExternalName做一个Map，KubeDNS返回一个CNAME记录。</span></span></span></span></span></p>
<p><span style="font-size: 18pt"><strong><span style="color: rgba(0, 0, 255, 1)">四. kube-proxy 工作原理 </span></strong><span style="color: rgba(255, 0, 0, 1)"><span style="color: rgba(0, 0, 0, 1)">(userspace, iptables, ipvs<span style="color: rgba(0, 0, 0, 1)">)<br>kube-proxy当前实现了三种代理模式：<strong><span style="font-size: 16px; color: rgba(255, 0, 0, 1)">userspace, </span></strong><span style="font-size: 16px"><strong><span style="color: rgba(255, 0, 0, 1)">iptables, ipvs</span></strong>。其中userspace
mode是v1.0及之前版本的默认模式，从v1.1版本中开始增加了iptables
mode，在v1.2版本中正式替代userspace模式成为默认模式。也就是说kubernetes在v1.2版本之前是默认模式,
v1.2版本之后默认模式是iptables。</span></span></span></span></span></p>
<p><span style="font-size: 16px"><strong><span style="color: rgba(255, 0, 0, 1)"><span style="color: rgba(0, 0, 0, 1)">1) <span style="font-size: 18px">userspace mode</span></span></span></strong>: <span style="color: rgba(0, 128, 0, 1)">userspace是在用户空间，通过kube-proxy来实现service的代理服务, 其原理如下:</span></span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201906/907596-20190628144448994-121687144.png"></p>
<p>可见，<span style="color: rgba(153, 51, 0, 1)">userspace这种mode最大的问题是，service的请求会先从用户空间进入内核iptables，然后再回到用户空间，由kube-proxy完成后端Endpoints的选择和代理工作，这样流量从用户空间进出内核带来的性能损耗是不可接受的。这也是k8s v1.0及之前版本中对kube-proxy质疑最大的一点，因此社区就开始研究iptables mode.</span></p>
<p><span style="color: rgba(153, 51, 102, 1)">userspace这种模式下，kube-proxy 持续监听 Service
以及 Endpoints 对象的变化；对每个
Service，它都为其在本地节点开放一个端口，作为其服务代理端口；发往该端口的请求会采用一定的策略转发给与该服务对应的后端 Pod
实体。kube-proxy 同时会在本地节点设置 iptables 规则，配置一个 Virtual IP，把发往 Virtual IP
的请求重定向到与该 Virtual IP 对应的服务代理端口上。其工作流程大体如下:</span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201903/907596-20190325170529464-807275929.png"></p>
<p><strong><span>由此分析: </span></strong>该模式请求在到达 iptables 进行处理时就会进入内核，而 kube-proxy 监听则是在用户态, 请求就形成了从用户态到内核态再返回到用户态的传递过程, 一定程度降低了服务性能。</p>
<p><span style="font-size: 16px"><strong>2) <span style="color: rgba(255, 0, 0, 1); font-size: 18px">iptables mode</span></strong>, <span style="color: rgba(0, 128, 0, 1)">该模式完全利用内核iptables来实现service的代理和LB, 这是K8s在v1.2及之后版本默认模式. 工作原理如下:</span></span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201906/907596-20190628150040159-782250464.png"></p>
<p>iptables mode因为使用iptable NAT来完成转发，也存在不可忽视的性能损耗。另外，<span style="color: rgba(153, 51, 0, 1)">如果集群中存在上万的Service/Endpoint，那么Node上的iptables rules将会非常庞大，性能还会再打折扣。这也导致目前大部分企业用k8s上生产时，都不会直接用kube-proxy作为服务代理，而是通过自己开发或者通过Ingress Controller来集成HAProxy, Nginx来代替kube-proxy。</span></p>
<p><span style="color: rgba(153, 51, 102, 1)">iptables 模式与 userspace
相同，kube-proxy 持续监听 Service 以及 Endpoints
对象的变化；但它并不在本地节点开启反向代理服务，而是把反向代理全部交给 iptables 来实现；即 iptables 直接将对 VIP
的请求转发给后端 Pod，通过 iptables 设置转发策略。其工作流程大体如下:</span></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201903/907596-20190325170538450-107344229.png"></p>
<p><strong><span>由此分析: </span></strong>该模式相比 userspace 模式，克服了请求在用户态-内核态反复传递的问题，性能上有所提升，但使用 iptables NAT 来完成转发，存在不可忽视的性能损耗，而且在大规模场景下，iptables 规则的条目会十分巨大，性能上还要再打折扣。</p>
<p>iptables的方式则是利用了linux的iptables的nat转发进行实现:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_630601" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
<div class="line number3 index2 alt2">3</div>
<div class="line number4 index3 alt1">4</div>
<div class="line number5 index4 alt2">5</div>
<div class="line number6 index5 alt1">6</div>
<div class="line number7 index6 alt2">7</div>
<div class="line number8 index7 alt1">8</div>
<div class="line number9 index8 alt2">9</div>
<div class="line number10 index9 alt1">10</div>
<div class="line number11 index10 alt2">11</div>
<div class="line number12 index11 alt1">12</div>
<div class="line number13 index12 alt2">13</div>
<div class="line number14 index13 alt1">14</div>
<div class="line number15 index14 alt2">15</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">apiVersion: v1</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">kind: Service</code></div>
<div class="line number3 index2 alt2"><code class="bash plain">metadata:</code></div>
<div class="line number4 index3 alt1"><code class="bash spaces"> </code><code class="bash plain">labels:</code></div>
<div class="line number5 index4 alt2"><code class="bash spaces"> </code><code class="bash plain">name: mysql</code></div>
<div class="line number6 index5 alt1"><code class="bash spaces"> </code><code class="bash plain">role: service</code></div>
<div class="line number7 index6 alt2"><code class="bash spaces"> </code><code class="bash plain">name: mysql-service</code></div>
<div class="line number8 index7 alt1"><code class="bash plain">spec:</code></div>
<div class="line number9 index8 alt2"><code class="bash spaces"> </code><code class="bash plain">ports:</code></div>
<div class="line number10 index9 alt1"><code class="bash spaces"> </code><code class="bash plain">- port: 3306</code></div>
<div class="line number11 index10 alt2"><code class="bash spaces"> </code><code class="bash plain">targetPort: 3306</code></div>
<div class="line number12 index11 alt1"><code class="bash spaces"> </code><code class="bash plain">nodePort: 30964</code></div>
<div class="line number13 index12 alt2"><code class="bash spaces"> </code><code class="bash functions">type</code><code class="bash plain">: NodePort</code></div>
<div class="line number14 index13 alt1"><code class="bash spaces"> </code><code class="bash plain">selector:</code></div>
<div class="line number15 index14 alt2"><code class="bash spaces"> </code><code class="bash plain">mysql-service: </code><code class="bash string">"true"</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>mysql-service对应的nodePort暴露出来的端口为30964，对应的cluster
IP(10.254.162.44)的端口为3306，进一步对应于后端的pod的端口为3306。 mysql-service后端代理了两个pod，ip分别是192.168.125.129和192.168.125.131,
这里先来看一下iptables:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_23984" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
<div class="line number3 index2 alt2">3</div>
<div class="line number4 index3 alt1">4</div>
<div class="line number5 index4 alt2">5</div>
<div class="line number6 index5 alt1">6</div>
<div class="line number7 index6 alt2">7</div>
<div class="line number8 index7 alt1">8</div>
<div class="line number9 index8 alt2">9</div>
<div class="line number10 index9 alt1">10</div>
<div class="line number11 index10 alt2">11</div>
<div class="line number12 index11 alt1">12</div>
<div class="line number13 index12 alt2">13</div>
<div class="line number14 index13 alt1">14</div>
<div class="line number15 index14 alt2">15</div>
<div class="line number16 index15 alt1">16</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">$iptables -S -t nat</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">...</code></div>
<div class="line number3 index2 alt2"><code class="bash plain">-A PREROUTING -m comment --comment </code><code class="bash string">"kubernetes service portals"</code> <code class="bash plain">-j KUBE-SERVICES</code></div>
<div class="line number4 index3 alt1"><code class="bash plain">-A OUTPUT -m comment --comment </code><code class="bash string">"kubernetes service portals"</code> <code class="bash plain">-j KUBE-SERVICES</code></div>
<div class="line number5 index4 alt2"><code class="bash plain">-A POSTROUTING -m comment --comment </code><code class="bash string">"kubernetes postrouting rules"</code> <code class="bash plain">-j KUBE-POSTROUTING</code></div>
<div class="line number6 index5 alt1"><code class="bash plain">-A KUBE-MARK-MASQ -j MARK --</code><code class="bash functions">set</code><code class="bash plain">-xmark 0x4000</code><code class="bash plain">/0x4000</code></div>
<div class="line number7 index6 alt2"><code class="bash plain">-A KUBE-NODEPORTS -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp --dport 30964 -j KUBE-MARK-MASQ</code></div>
<div class="line number8 index7 alt1"><code class="bash plain">-A KUBE-NODEPORTS -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp --dport 30964 -j KUBE-SVC-67RL4FN6JRUPOJYM</code></div>
<div class="line number9 index8 alt2"><code class="bash plain">-A KUBE-SEP-ID6YWIT3F6WNZ47P -s 192.168.125.129</code><code class="bash plain">/32</code> <code class="bash plain">-m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-MARK-MASQ</code></div>
<div class="line number10 index9 alt1"><code class="bash plain">-A KUBE-SEP-ID6YWIT3F6WNZ47P -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp -j DNAT --to-destination 192.168.125.129:3306</code></div>
<div class="line number11 index10 alt2"><code class="bash plain">-A KUBE-SEP-IN2YML2VIFH5RO2T -s 192.168.125.131</code><code class="bash plain">/32</code> <code class="bash plain">-m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-MARK-MASQ</code></div>
<div class="line number12 index11 alt1"><code class="bash plain">-A KUBE-SEP-IN2YML2VIFH5RO2T -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp -j DNAT --to-destination 192.168.125.131:3306</code></div>
<div class="line number13 index12 alt2"><code class="bash plain">-A KUBE-SERVICES -d 10.254.162.44</code><code class="bash plain">/32</code> <code class="bash plain">-p tcp -m comment --comment </code><code class="bash string">"default/mysql-service: cluster IP"</code> <code class="bash plain">-m tcp --dport 3306 -j KUBE-SVC-67RL4FN6JRUPOJYM</code></div>
<div class="line number14 index13 alt1"><code class="bash plain">-A KUBE-SERVICES -m comment --comment </code><code class="bash string">"kubernetes service nodeports; NOTE: this must be the last rule in this chain"</code> <code class="bash plain">-m addrtype --dst-</code><code class="bash functions">type</code> <code class="bash plain">LOCAL -j KUBE-NODEPORTS</code></div>
<div class="line number15 index14 alt2"><code class="bash plain">-A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ID6YWIT3F6WNZ47P</code></div>
<div class="line number16 index15 alt1"><code class="bash plain">-A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-SEP-IN2YML2VIFH5RO2T</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>首先如果是通过node的30964端口访问，则会进入到以下链:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_586996" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">-A KUBE-NODEPORTS -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp --dport 30964 -j KUBE-MARK-MASQ</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">-A KUBE-NODEPORTS -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp --dport 30964 -j KUBE-SVC-67RL4FN6JRUPOJYM</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>然后进一步跳转到KUBE-SVC-67RL4FN6JRUPOJYM的链:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_55130" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">-A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-ID6YWIT3F6WNZ47P</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">-A KUBE-SVC-67RL4FN6JRUPOJYM -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-SEP-IN2YML2VIFH5RO2T</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>这里利用了iptables的–probability的特性，使连接有50%的概率进入到KUBE-SEP-ID6YWIT3F6WNZ47P链，50%的概率进入到KUBE-SEP-IN2YML2VIFH5RO2T链。
KUBE-SEP-ID6YWIT3F6WNZ47P的链的具体作用就是将请求通过DNAT发送到192.168.125.129的3306端口:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_911610" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">-A KUBE-SEP-ID6YWIT3F6WNZ47P -s 192.168.125.129</code><code class="bash plain">/32</code> <code class="bash plain">-m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-MARK-MASQ</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">-A KUBE-SEP-ID6YWIT3F6WNZ47P -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp -j DNAT --to-destination 192.168.125.129:3306</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>同理KUBE-SEP-IN2YML2VIFH5RO2T的作用是通过DNAT发送到192.168.125.131的3306端口:</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_934064" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">-A KUBE-SEP-IN2YML2VIFH5RO2T -s 192.168.125.131</code><code class="bash plain">/32</code> <code class="bash plain">-m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-j KUBE-MARK-MASQ</code></div>
<div class="line number2 index1 alt1"><code class="bash plain">-A KUBE-SEP-IN2YML2VIFH5RO2T -p tcp -m comment --comment </code><code class="bash string">"default/mysql-service:"</code> <code class="bash plain">-m tcp -j DNAT --to-destination 192.168.125.131:3306</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>分析完nodePort的工作方式，接下里说一下clusterIP的访问方式。 对于直接访问cluster IP(10.254.162.44)的3306端口会直接跳转到KUBE-SVC-67RL4FN6JRUPOJYM</p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_8053" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">-A KUBE-SERVICES -d 10.254.162.44</code><code class="bash plain">/32</code> <code class="bash plain">-p tcp -m comment --comment </code><code class="bash string">"default/mysql-service: cluster IP"</code> <code class="bash plain">-m tcp --dport 3306 -j KUBE-SVC-67RL4FN6JRUPOJYM</code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p>接下来的跳转方式同NodePort方式。</p>
<p><span style="color: rgba(0, 0, 0, 1)"><span style="color: rgba(255, 0, 0, 1)"><strong><span style="color: rgba(0, 0, 0, 1)">3)<span style="font-size: 18px"> <span style="font-size: 18px">ipvs mode</span></span></span></strong>. <span style="color: rgba(0, 0, 255, 1)">在kubernetes 1.8以上的版本中，对于kube-proxy组件增加了除iptables模式和用户模式之外还支持ipvs模式。<span style="color: rgba(0, 0, 0, 1)">kube-proxy
ipvs 是基于 NAT 实现的，通过ipvs的NAT模式，对访问k8s service的请求进行虚IP到POD IP的转发。当创建一个
service 后，kubernetes 会在每个节点上创建一个网卡，同时帮你将 Service IP(VIP) 绑定上，此时相当于每个
Node 都是一个 ds，而其他任何 Node 上的 Pod，甚至是宿主机服务(比如 kube-apiserver 的 6443)都可能成为
rs；</span></span></span></span></p>
<p>与iptables、userspace 模式一样，kube-proxy 依然监听Service以及Endpoints对象的变化,
不过它并不创建反向代理, 也不创建大量的 iptables 规则, 而是通过netlink 创建ipvs规则，并使用k8s
Service与Endpoints信息，对所在节点的ipvs规则进行定期同步; netlink 与 iptables
底层都是基于 netfilter 钩子，但是 netlink 由于采用了 hash table 而且直接工作在内核态，在性能上比
iptables 更优。其工作流程大体如下:</p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201903/907596-20190325170554554-1168234966.png"></p>
<p><strong><span>由此分析：</span></strong>ipvs 是目前 kube-proxy 所支持的最新代理模式，相比使用 iptables，使用 ipvs 具有更高的性能。</p>
<p><span>
</span></p>
<p><span style="color: rgba(255, 102, 0, 1); font-size: 18px"><strong>Endpoint访问外部服务</strong><br>k8s访问集群外独立的服务最好的方式是采用Endpoint方式，以mysql服务为例:</span></p>
<div class="cnblogs_Highlighter sh-gutter">
<div>
<div id="highlighter_338560" class="syntaxhighlighterbash">
<table border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td class="gutter">
<div class="line number1 index0 alt2">1</div>
<div class="line number2 index1 alt1">2</div>
<div class="line number3 index2 alt2">3</div>
<div class="line number4 index3 alt1">4</div>
<div class="line number5 index4 alt2">5</div>
<div class="line number6 index5 alt1">6</div>
<div class="line number7 index6 alt2">7</div>
<div class="line number8 index7 alt1">8</div>
<div class="line number9 index8 alt2">9</div>
<div class="line number10 index9 alt1">10</div>
<div class="line number11 index10 alt2">11</div>
<div class="line number12 index11 alt1">12</div>
<div class="line number13 index12 alt2">13</div>
<div class="line number14 index13 alt1">14</div>
<div class="line number15 index14 alt2">15</div>
<div class="line number16 index15 alt1">16</div>
<div class="line number17 index16 alt2">17</div>
<div class="line number18 index17 alt1">18</div>
<div class="line number19 index18 alt2">19</div>
<div class="line number20 index19 alt1">20</div>
<div class="line number21 index20 alt2">21</div>
<div class="line number22 index21 alt1">22</div>
<div class="line number23 index22 alt2">23</div>
<div class="line number24 index23 alt1">24</div>
<div class="line number25 index24 alt2">25</div>
<div class="line number26 index25 alt1">26</div>
<div class="line number27 index26 alt2">27</div>
<div class="line number28 index27 alt1">28</div>
<div class="line number29 index28 alt2">29</div>
<div class="line number30 index29 alt1">30</div>
<div class="line number31 index30 alt2">31</div>
<div class="line number32 index31 alt1">32</div>
<div class="line number33 index32 alt2">33</div>
<div class="line number34 index33 alt1">34</div>
<div class="line number35 index34 alt2">35</div>
<div class="line number36 index35 alt1">36</div>
<div class="line number37 index36 alt2">37</div>
<div class="line number38 index37 alt1">38</div>
<div class="line number39 index38 alt2">39</div>
<div class="line number40 index39 alt1">40</div>
<div class="line number41 index40 alt2">41</div>
<div class="line number42 index41 alt1">42</div>
<div class="line number43 index42 alt2">43</div>
</td>
<td class="code">
<div class="container">
<div class="line number1 index0 alt2"><code class="bash plain">1）创建mysql-service.yaml</code></div>
<div class="line number2 index1 alt1"><code class="bash plain"></code><code class="bash comments"># vim mysql-service.yaml</code></div>
<div class="line number3 index2 alt2"><code class="bash plain">apiVersion: v1</code></div>
<div class="line number4 index3 alt1"><code class="bash plain">kind: Service</code></div>
<div class="line number5 index4 alt2"><code class="bash plain">metadata:</code></div>
<div class="line number6 index5 alt1"><code class="bash spaces"> </code><code class="bash plain">name: mysql-kevin</code></div>
<div class="line number7 index6 alt2"><code class="bash plain">spec:</code></div>
<div class="line number8 index7 alt1"><code class="bash spaces"> </code><code class="bash plain">ports:</code></div>
<div class="line number9 index8 alt2"><code class="bash spaces"> </code><code class="bash plain">- port: 3306</code></div>
<div class="line number10 index9 alt1"> </div>
<div class="line number11 index10 alt2"><code class="bash plain">2) 创建mysql-endpoints.yaml</code></div>
<div class="line number12 index11 alt1"><code class="bash plain"></code><code class="bash comments"># vim mysql-endpoints.yaml</code></div>
<div class="line number13 index12 alt2"><code class="bash plain">kind: Endpoints</code></div>
<div class="line number14 index13 alt1"><code class="bash plain">apiVersion: v1</code></div>
<div class="line number15 index14 alt2"><code class="bash plain">metadata:</code></div>
<div class="line number16 index15 alt1"><code class="bash spaces"> </code><code class="bash plain">name: mysql-kevin</code></div>
<div class="line number17 index16 alt2"><code class="bash spaces"> </code><code class="bash plain">namespace: default</code></div>
<div class="line number18 index17 alt1"><code class="bash plain">subsets:</code></div>
<div class="line number19 index18 alt2"><code class="bash spaces"> </code><code class="bash plain">- addresses:</code></div>
<div class="line number20 index19 alt1"><code class="bash spaces"> </code><code class="bash plain">- ip: 172.16.60.55</code></div>
<div class="line number21 index20 alt2"><code class="bash spaces"> </code><code class="bash plain">ports:</code></div>
<div class="line number22 index21 alt1"><code class="bash spaces"> </code><code class="bash plain">- port: 3306</code></div>
<div class="line number23 index22 alt2"> </div>
<div class="line number24 index23 alt1"><code class="bash plain">3) 测试连接数据库</code></div>
<div class="line number25 index24 alt2"><code class="bash plain"></code><code class="bash comments"># kubectl exec -it mysql-client-h7jk8 bash</code></div>
<div class="line number26 index25 alt1"><code class="bash functions">bash</code><code class="bash plain">-4.1</code><code class="bash comments"># mysql -hmysql-kevin -u user -p</code></div>
<div class="line number27 index26 alt2"><code class="bash plain">Enter password:</code></div>
<div class="line number28 index27 alt1"><code class="bash plain">.........</code></div>
<div class="line number29 index28 alt2"><code class="bash plain">mysql> </code></div>
<div class="line number30 index29 alt1"> </div>
<div class="line number31 index30 alt2"><code class="bash plain">4) 查看这个service</code></div>
<div class="line number32 index31 alt1"><code class="bash plain"></code><code class="bash comments"># kubectl describe svc mysql-kevin</code></div>
<div class="line number33 index32 alt2"><code class="bash plain">Name: mysql-kevin</code></div>
<div class="line number34 index33 alt1"><code class="bash plain">Namespace: default</code></div>
<div class="line number35 index34 alt2"><code class="bash plain">Labels: <none></code></div>
<div class="line number36 index35 alt1"><code class="bash plain">Annotations: <none></code></div>
<div class="line number37 index36 alt2"><code class="bash plain">Selector: <none></code></div>
<div class="line number38 index37 alt1"><code class="bash plain">Type: ClusterIP</code></div>
<div class="line number39 index38 alt2"><code class="bash plain">IP: 10.254.125.157</code></div>
<div class="line number40 index39 alt1"><code class="bash plain">Port: <</code><code class="bash functions">unset</code><code class="bash plain">> 3306</code><code class="bash plain">/TCP</code></div>
<div class="line number41 index40 alt2"><code class="bash plain">Endpoints: 172.16.60.55:3306</code></div>
<div class="line number42 index41 alt1"><code class="bash plain">Session Affinity: None</code></div>
<div class="line number43 index42 alt2"><code class="bash plain">Events: <none></code></div>
</div>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p><span style="font-size: 16px"><strong><span style="color: rgba(255, 102, 0, 1)">下面简单说kube-proxy是如何实现一个请求经过层层转发最后落到某个pod上的整个过程，这个请求可能来自pod也可能来自外部。</span></strong><br>-> kube-proxy为集群提供service功能，相同功能的pods对外抽象为service，service可以实现反向代理和服务发现。可以分为iptables模式和userspace模式。具体有iptables实现<br>-> 在反向代理方面，kube-proxy默认使用rr算法实现客户端流量分发到后端的pod</span></p>
<p><span style="font-size: 16px"><strong><span style="color: rgba(255, 102, 0, 1)">k8s的service和endpoine是如何关联和相互影响的？</span></strong><br>-> <span style="color: rgba(255, 0, 255, 1)">api-server创建service对象，与service绑定的pod地址：称之为endpoints<br>-> 服务发现方面：kube-proxy监控service后端endpoint的动态变化，并且维护service和endpoint的映射关系</span></span></p>
<p><span style="font-size: 16px"><strong><span style="color: rgba(255, 102, 0, 1)">一个经典pod的完整生命周期</span></strong><br>-> Pending <br>-> Running <br>-> Succeeded <br>-> Failed</span></p>
<p><strong><span style="color: rgba(51, 102, 255, 1)">关系流程图如下:</span></strong></p>
<p><img src="https://img2018.cnblogs.com/blog/907596/201906/907596-20190628115235914-1787259683.png"></p>
<p><span>
</span></p>
<p><span style="font-size: 16px; color: rgba(153, 51, 102, 1)"><strong>K8S Endpoint一会消失一会出现的问题</strong><br>在使用K8s集群时遇到的问题：发现某个service的后端endpoint一会显示有后端，一会显示没有。显示没有后端，意味着后端的address被判定为notready。</span></p>
<p><span style="color: rgba(0, 0, 255, 1)"><strong>经过排查确定原因：</strong><br>kubelet在准备上报信息时，需要收集容器、镜像等的信息。虽然kubelet默认是10秒上报一次，但是实际的上报周期约为20~50秒。而kube-controller-manager判断node上报心跳超时的时间为40秒。所以会有一定概率超时。一旦超时，kube-controller会将该node上的所有pod的conditions中type是Ready的字典中的status置为False。</span></p>
<p><strong><span style="color: rgba(0, 0, 255, 1)">解决办法：</span></strong><br>较为简单的方案是在kube-controller上配置这个超时时间<strong><span style="color: rgba(255, 0, 0, 1)">node-monitor-grace-period</span></strong>长一些。建议配置为<strong><span style="color: rgba(255, 0, 0, 1)">60 ~ 120s</span></strong>。</p>
<div id="MySignature" style="display: block">***************当你发现自己的才华撑不起野心时，就请安静下来学习吧***************</div>
<div style="display: block">参考地址：https://www.cnblogs.com/kevingrace/p/6655153.html</div><br><br>
来源：https://www.cnblogs.com/fuyuteng/p/11598768.html
頁:
[1]