Kubernetes 企业级集群部署方式

光头老王 發表於 2019-8-26 07:15:00

<h1>一、Kubernetes介绍与特性</h1>
<h2>1.1、kubernetes是什么</h2>
官方网站：http://www.kubernetes.io
• Kubernetes是Google在2014年开源的一个容器集群管理系统，Kubernetes简称K8S。 • K8S用于容器化应用程序的部署，扩展和管理。 • K8S提供了容器编排，资源调度，弹性伸缩，部署管理，服务发现等一系列功能。 • Kubernetes目标是让部署容器化应用简单高效。
<h2>1.2、kubernetes是什么</h2>
一个容器平台 一个微服务平台 便捷式云平台
<h2>1.3、kubernetes特性</h2>
- 自我修复 在节点故障时重新启动失败的容器，替换和重新部署，保证预期的副本数量；杀死健康检查失败的容器，并且在未准备好之前不会处理客户端请求，确保线上服务不中断。 - 弹性伸缩 使用命令、UI或者基于CPU使用情况自动快速扩容和缩容应用程序实例，保证应用业务高峰并发时的高可用性；业务低峰时回收资源，以最小成本运行服务。 - 自动部署和回滚 K8S采用滚动更新策略更新应用，一次更新一个Pod，而不是同时删除所有Pod，如果更新过程中出现问题，将回滚更改，确保升级不受影响业务。 - 服务发现和负载均衡 K8S为多个容器提供一个统一访问入口（内部IP地址和一个DNS名称），并且负载均衡关联的所有容器，使得用户无需考虑容器IP问题。 - 机密和配置管理 管理机密数据和应用程序配置，而不需要把敏感数据暴露在镜像里，提高敏感数据安全性。并可以将一些常用的配置存储在K8S中，方便应用程序使用。 - 存储编排 挂载外部存储系统，无论是来自本地存储，公有云（如AWS），还是网络存储（如NFS、GlusterFS、Ceph）都作为集群资源的一部分使用，极大提高存储使用灵活性。 - 批处理 提供一次性任务，定时任务；满足批量数据处理和分析的场景。
<h1>二、kubernetes组织架构介绍</h1>
<h2>2.1、整体架构组件详解</h2>
1、如图，有三个节点一个master节点和两个node节点。 2、Master有三个组件: 　　　　- API server：K8S提供的一个统一的入口，提供RESTful API访问方式接口服务。 　　　　　　- Auth：认证授权，判断是否有权限访问 　　　　　　- Etcd：存储的数据库、存储认证信息等，K8S状态，节点信息等 　　　　- scheduler：集群的调度，将集群分配到哪个节点内 　　　　- controller manager：控制器，来控制来做哪些任务，管理 pod service 控制器等 　　- Kubectl：管理工具，直接管理API Server，期间会有认证授权。 3、Node有两个组件： 　　　　- kubelet：接收K8S下发的任务，管理容器创建，生命周期管理等，将一个pod转换成一组容器。 　　　　- kube-proxy：Pod网络代理，四层负载均衡，对外访问 　　　　　　-  用户 -> 防火墙 -> kube-proxy -> 业务 　　　　Pod：K8S最小单元 　　　　　　- Container：运行容器的环境，运行容器引擎 　　　　　　　　- Docker
<img src="https://img2018.cnblogs.com/blog/1183448/201908/1183448-20190826070856495-56128924.png" alt="" width="836" height="449">
<h2>2.2、集群管理流程及核心概念</h2>
1、管理集群流程
<img src="https://img2018.cnblogs.com/blog/1183448/201908/1183448-20190826071009558-581778358.png" alt="">
2、Kubernetes核心概念
 <img src="https://img2018.cnblogs.com/blog/1183448/201908/1183448-20190826071112265-371701075.png" alt="">
Pod 　　• 最小部署单元 　　• 一组容器的集合 　　• 一个Pod中的容器共享网络命名空间 　　• Pod是短暂的 Controllers 　　• ReplicaSet ：确保预期的Pod副本数量 　　• Deployment ：无状态应用部署 　　• StatefulSet ：有状态应用部署 　　• DaemonSet ：确保所有Node运行同一个Pod 　　• Job ：一次性任务 　　• Cronjob ：定时任务 　　注：更高级层次对象，部署和管理Pod
Service 　　• 防止Pod失联 　　• 定义一组Pod的访问策略
Label ：标签，附加到某个资源上，用于关联对象、查询和筛选
Namespaces ：命名空间，将对象逻辑上隔离
Annotations ：注释
<h1> 三、Kubernetes 部署</h1>
<ul>
<li># K8S 相关服务包</li>
<li>百度云下载：https://pan.baidu.com/s/1d1zqoil3pfeThC-v45bWkg</li>
<li>密码：0ssx</li>
</ul>
<h2>3.1 服务版本及架构说明</h2>
服务版本
<ul>
<li>centos：7.4</li>
<li>etcd-v3.3.10</li>
<li>flannel-v0.10.0</li>
<li>kubernetes-1.12.1</li>
<li>nginx-1.16.1</li>
<li>keepalived-1.3.5</li>
<li>docker-19.03.1</li>

</ul>
单Master架构
<ul>
<li>k8s Master：172.16.105.220</li>
<li>k8s Node：172.16.105.230、172.16.105.213</li>
<li>etcd：172.16.105.220、172.16.105.230、172.16.105.213</li>

</ul>
双Master+Nginx+Keepalived
<ul>
<li>k8s Master1：192.168.1.108</li>
<li>k8s Master2：192.168.1.109</li>
<li>k8s Node3：192.168.1.110</li>
<li>k8s Node4：192.168.1.111</li>
<li>etc：192.168.1.108、192.168.1.109、192.168.1.110、192.168.1.111</li>
<li>Nginx+keepalived1:192.168.1.112</li>
<li>Nginx+keepalived2:192.168.1.113</li>
<li>vip:192.168.1.100</li>

</ul>
 
<h2>3.2、部署kubernetes准备</h2>
1、关闭防火墙
<div class="cnblogs_code">
<pre>systemctl stop firewalld.service</pre>
</div>
2、关闭SELINUX
<div class="cnblogs_code">
<pre>setenforce 0</pre>
</div>
3、修改主机名
<div class="cnblogs_code">
<pre>vim /etc/hostname
hostname ****</pre>
</div>
4、同步时间
<div class="cnblogs_code">
<pre>ntpdate time.windows.com</pre>
</div>
5、环境变量
注：下面配置所有用到的k8s最好部署环境变量
<h2>3.3、Etcd 数据库集群部署</h2>
<h3>1、部署 Etcd 自签证书 </h3>
1、创建k8s及证书目录
<div class="cnblogs_code">
<pre>mkdir ~/k8s && cd ~/k8s
mkdir k8s-cert
mkdir etcd-cert
cd etcd-cert</pre>
</div>
2、安装cfssl生成证书工具
<div class="cnblogs_code">
<pre># 通过选项生成证书
curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl
# 通过json生成证书
curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson
# 查看证书信息
curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo
# 添加执行权限
chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson /usr/local/bin/cfssl-certinfo</pre>
</div>
3、执行命令生成证书使用的json文件1
<div class="cnblogs_code"><img id="code_img_closed_471f1397-a8eb-4798-904a-883f88d66fc9" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_471f1397-a8eb-4798-904a-883f88d66fc9" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_471f1397-a8eb-4798-904a-883f88d66fc9" class="cnblogs_code_hide">
<pre>vim ca-config.json
{
"signing": {
"default": {
 "expiry": "87600h"
},
"profiles": {
 "www": {
 "expiry": "87600h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
}
}
}</pre>
</div>
vim ca-config.json</div>
4、执行命令生成证书使用的json文件2
<div class="cnblogs_code"><img id="code_img_closed_8a1a0be1-22ba-4c50-8de3-baa7055e116c" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_8a1a0be1-22ba-4c50-8de3-baa7055e116c" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_8a1a0be1-22ba-4c50-8de3-baa7055e116c" class="cnblogs_code_hide">
<pre>{
"CN": "etcd CA",
"key": {
 "algo": "rsa",
 "size": 2048
},
"names": [
 {
 "C": "CN",
 "L": "Beijing",
 "ST": "Beijing"
 }
]
}</pre>
</div>
vim ca-csr.json</div>
5、执行命令通过json文件生成CA根证书、会在当前目录生成ca.pem和ca-key.pem
<div class="cnblogs_code">
<pre>cfssl gencert -initca ca-csr.json | cfssljson -bare ca -</pre>
</div>
6、执行命令生成Etcd域名证书、首先创建json文件后生成
<div class="cnblogs_code"><img id="code_img_closed_4961da7b-b1f9-44a2-82c0-6606df5c3b56" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_4961da7b-b1f9-44a2-82c0-6606df5c3b56" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_4961da7b-b1f9-44a2-82c0-6606df5c3b56" class="cnblogs_code_hide">
<pre>{
"CN": "etcd",
"hosts": [
"172.16.105.220",
"172.16.105.230",
"172.16.105.213"
],
"key": {
 "algo": "rsa",
 "size": 2048
},
"names": [
 {
 "C": "CN",
 "L": "BeiJing",
 "ST": "BeiJing"
 }
]
}</pre>
</div>
vim server-csr.json</div>
注：hosts下面跟etcd部署服务的IP。
7、执行命令办法Etcd域名证书、当前目录下生成 server.pem 与 server-key.pem
<div class="cnblogs_code">
<pre>cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server</pre>
</div>
8、查看创建的证书
<div class="cnblogs_code">
<pre>ls *pem
ca-key.pemca.pemserver-key.pemserver.pem</pre>
</div>
<h3>2、部署 Etcd 数据库集群</h3>
<ul>
<li>使用etcd版本：etcd-v3.3.10-linux-amd64.tar.gz</li>
<li>二进制包下载地址：https://github.com/coreos/etcd/releases/tag/v3.2.12</li>
</ul>
1、下载本地后进行解压、进入到解压目录
<div class="cnblogs_code">
<pre>tar zxvf etcd-v3.3.10-linux-amd64.tar.gz
cd etcd-v3.3.10-linux-amd64</pre>
</div>
2、为了方便管理etcd创建几个目录、并移动文件
<div class="cnblogs_code">
<pre>mkdir /opt/etcd/{cfg,bin,ssl} -p
mv etcd etcdctl /opt/etcd/bin/</pre>
</div>
3、创建编写etcd配置文件
<div class="cnblogs_code"><img id="code_img_closed_dcfecfdb-9f1a-45c1-9ded-331550133c5f" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_dcfecfdb-9f1a-45c1-9ded-331550133c5f" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_dcfecfdb-9f1a-45c1-9ded-331550133c5f" class="cnblogs_code_hide">
<pre>#
ETCD_NAME="etcd01"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.105.220:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.105.220:2379"

#
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.105.220:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.105.220:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.105.220:2380,etcd02=https://172.16.105.230:2380,etcd03=https://172.16.105.213:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"</pre>
</div>
vim /opt/etcd/cfg/etcd </div>
<div class="cnblogs_code"><img id="code_img_closed_4fe44c99-aa6f-4d19-ac56-cf5867bf564a" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_4fe44c99-aa6f-4d19-ac56-cf5867bf564a" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_4fe44c99-aa6f-4d19-ac56-cf5867bf564a" class="cnblogs_code_hide">
<pre>· ETCD_NAME 节点名称
· ETCD_DATA_DIR 数据目录
· ETCD_LISTEN_PEER_URLS 集群通信监听地址
· ETCD_LISTEN_CLIENT_URLS 客户端访问监听地址
· ETCD_INITIAL_ADVERTISE_PEER_URLS 集群通告地址
· ETCD_ADVERTISE_CLIENT_URLS 客户端通告地址
· ETCD_INITIAL_CLUSTER 集群节点地址
· ETCD_INITIAL_CLUSTER_TOKEN 集群Token
· ETCD_INITIAL_CLUSTER_STATE 加入集群的当前状态，new是新集群，existing表示加入已有集群</pre>
</div>
参数含义</div>
4、创建systemd 管理 etcd
<div class="cnblogs_code"><img id="code_img_closed_8e1da762-98c3-4ded-8b90-41b08b82bc3f" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_8e1da762-98c3-4ded-8b90-41b08b82bc3f" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_8e1da762-98c3-4ded-8b90-41b08b82bc3f" class="cnblogs_code_hide">
<pre>
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd
ExecStart=/opt/etcd/bin/etcd \
--name=${ETCD_NAME} \
--data-dir=${ETCD_DATA_DIR} \
--listen-peer-urls=${ETCD_LISTEN_PEER_URLS} \
--listen-client-urls=${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379 \
--advertise-client-urls=${ETCD_ADVERTISE_CLIENT_URLS} \
--initial-advertise-peer-urls=${ETCD_INITIAL_ADVERTISE_PEER_URLS} \
--initial-cluster=${ETCD_INITIAL_CLUSTER} \
--initial-cluster-token=${ETCD_INITIAL_CLUSTER_TOKEN} \
--initial-cluster-state=new \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/etcd.service</div>
5、将证书文件copy到指定目录
<div class="cnblogs_code">
<pre>cp /root/k8s/etcd-cert/{ca,ca-key,server-key,server}.pem /opt/etcd/ssl/</pre>
</div>
6、启动 etcd、并设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service</pre>
</div>
7、开启后etcd可能会等待其他两个节点等待，需要讲其他两个节点etcd开启
<div class="cnblogs_code">
<pre># 1、将目录etcd配置目录 copy 到两个节点内
scp -r /opt/etcd/ root@172.16.105.230:/opt/
scp -r /opt/etcd/ root@172.16.105.213:/opt/
# 2、将启动服务配置文件 copy 到两个节点内
scp -r /usr/lib/systemd/system/etcd.service root@172.16.105.230:/usr/lib/systemd/system/
scp -r /usr/lib/systemd/system/etcd.service root@172.16.105.213:/usr/lib/systemd/system/</pre>
</div>
8、修改两个节点 etcd /opt/etcd/cfg/etcd 配置文件
<div class="cnblogs_code"><img id="code_img_closed_dd6b6bef-c9c3-4bc5-b244-1f909e784042" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_dd6b6bef-c9c3-4bc5-b244-1f909e784042" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_dd6b6bef-c9c3-4bc5-b244-1f909e784042" class="cnblogs_code_hide">
<pre>#
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.105.230:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.105.230:2379"

#
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.105.230:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.105.230:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.105.220:2380,etcd02=https://172.16.105.230:2380,etcd03=https://172.16.105.213:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"</pre>
</div>
节点：172.16.105.230 配置文件</div>
<div class="cnblogs_code"><img id="code_img_closed_7ca22308-a2b8-4563-b064-319e7063005c" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_7ca22308-a2b8-4563-b064-319e7063005c" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_7ca22308-a2b8-4563-b064-319e7063005c" class="cnblogs_code_hide">
<pre>#
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://172.16.105.213:2380"
ETCD_LISTEN_CLIENT_URLS="https://172.16.105.213:2379"

#
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.16.105.213:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://172.16.105.213:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://172.16.105.220:2380,etcd02=https://172.16.105.230:2380,etcd03=https://172.16.105.213:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"</pre>
</div>
节点：172.16.105.213 配置文件</div>
9、两个节点启动服务、并设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable etcd.service
systemctl start etcd.service</pre>
</div>
10、查看主etcd日志
<div class="cnblogs_code"><img id="code_img_closed_8d2e5de7-4ab6-4505-bfd3-4784404b6a5d" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_8d2e5de7-4ab6-4505-bfd3-4784404b6a5d" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_8d2e5de7-4ab6-4505-bfd3-4784404b6a5d" class="cnblogs_code_hide">
<pre>Aug6 11:13:54 izbp14x4an2p4z7awyek7mz etcd: updating the cluster version from 3.0 to 3.3
Aug6 11:13:54 izbp14x4an2p4z7awyek7mz etcd: updated the cluster version from 3.0 to 3.3
Aug6 11:13:54 izbp14x4an2p4z7awyek7mz etcd: enabled capabilities for version 3.3</pre>
</div>
tail /var/log/messages -f</div>
11、查看端口启动
<div class="cnblogs_code"><img id="code_img_closed_688a5c0c-d463-4d0e-a13f-e936b7d41497" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_688a5c0c-d463-4d0e-a13f-e936b7d41497" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_688a5c0c-d463-4d0e-a13f-e936b7d41497" class="cnblogs_code_hide">
<pre>tcp 0 0 172.16.105.220:2379 0.0.0.0:* LISTEN 13021/etcd
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 13021/etcd
tcp 0 0 172.16.105.220:2380 0.0.0.0:* LISTEN 13021/etcd </pre>
</div>
netstat -lnpt</div>
12、查看进程使用
<div class="cnblogs_code"><img id="code_img_closed_85157f13-2ed1-48bc-8e43-c4ac3822f9d0" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_85157f13-2ed1-48bc-8e43-c4ac3822f9d0" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_85157f13-2ed1-48bc-8e43-c4ac3822f9d0" class="cnblogs_code_hide">
<pre>root 130211.11.4 10541908 28052 ? Ssl11:13 0:02 /opt/etcd/bin/etcd --name=etcd01 --data-dir=/var/lib/etcd/default.etcd --listen-peer-urls=https://172.16.105.220:2380 --listen-client-urls=https://172.16.105.220:2379,http://127.0.0.1:2379 --advertise-client-urls=https://172.16.105.220:2379 --initial-advertise-peer-urls=https://172.16.105.220:2380 --initial-cluster=etcd01=https://172.16.105.220:2380,etcd02=https://172.16.105.230:2380,etcd03=https://172.16.105.213:2380 --initial-cluster-token=etcd-cluster --initial-cluster-state=new --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --peer-cert-file=/opt/etcd/ssl/server.pem --peer-key-file=/opt/etcd/ssl/server-key.pem --trusted-ca-file=/opt/etcd/ssl/ca.pem --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem</pre>
</div>
ps -aux | grep etcd</div>
13、通过工具验证etcd
<div class="cnblogs_code">
<pre># 添加证书文件绝对路径与etcd集群节点地址
/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379" cluster-health</pre>
</div>
<div class="cnblogs_code"><img id="code_img_closed_deb8d394-74d3-44b8-a46b-1cf35677e960" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_deb8d394-74d3-44b8-a46b-1cf35677e960" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_deb8d394-74d3-44b8-a46b-1cf35677e960" class="cnblogs_code_hide">
<pre>member 1d5fcc16a8c9361e is healthy: got healthy result from https://172.16.105.220:2379
member 7b28469233594fbd is healthy: got healthy result from https://172.16.105.230:2379
member b2e216e703023e21 is healthy: got healthy result from https://172.16.105.213:2379
cluster is healthy</pre>
</div>
输出如下表示没问题：</div>
其他：
<div class="cnblogs_code"><img id="code_img_closed_612147ac-27ea-4ff2-bfd0-2376bfa23016" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_612147ac-27ea-4ff2-bfd0-2376bfa23016" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_612147ac-27ea-4ff2-bfd0-2376bfa23016" class="cnblogs_code_hide">
<pre># 删除每个节点data文件重新启动
rm -rf /var/lib/etcd/default.etcd</pre>
</div>
报错：etcd: request cluster ID mismatch</div>
<h2>3.4、Node 部署 Docker 容器应用 </h2>
1、安装依赖包
<div class="cnblogs_code">
<pre>yum install -y yum-utils device-mapper-persistent-data lvm2</pre>
</div>
2、配置官方源
<div class="cnblogs_code">
<pre>yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo</pre>
</div>
3、安装docker最新版
<div class="cnblogs_code">
<pre>yum -y install docker-ce</pre>
</div>
4、配置docker仓库加速器
<div class="cnblogs_code">
<pre>官网:https://www.daocloud.io/mirror
加速命令：curl -sSL https://get.daocloud.io/daotools/set_mirror.sh | sh -s http://f1361db2.m.daocloud.io</pre>
</div>
5、重启docker
<div class="cnblogs_code">
<pre>systemctl restart docker</pre>
</div>
6、查看docker版本：docker version
<div class="cnblogs_code">
<pre>Version: 19.03.1</pre>
</div>
<h2>3.5、Node 部署 Flannel 网络模型</h2>
<ul>
<li>二进制包：https://github.com/coreos/flannel/releases</li>
</ul>
1、写入分配的子网到etcd、提供flanneld使用。
<div class="cnblogs_code">
<pre>/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379" set /coreos.com/network/config '{ "Network": "172.17.0.0/16", "Backend": {"Type": "vxlan"}}'</pre>
</div>
2、查看创建网络信息
<div class="cnblogs_code">
<pre>/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379" get /coreos.com/network/config</pre>
</div>
3、下载完flannel包后进行解压
<div class="cnblogs_code">
<pre>tar -xvzf flannel-v0.10.0-linux-amd64.tar.gz</pre>
</div>
4、创建目录将文件存放到指定目录下
<div class="cnblogs_code">
<pre>mkdir -p /opt/kubernetes/{bin,cfg,ssl}
mv flanneld mk-docker-opts.sh /opt/kubernetes/bin/</pre>
</div>
5、创建flanneld配置文件
<div class="cnblogs_code"><img id="code_img_closed_4f51d6af-1415-4df7-810f-12cfa9e4d161" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_4f51d6af-1415-4df7-810f-12cfa9e4d161" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_4f51d6af-1415-4df7-810f-12cfa9e4d161" class="cnblogs_code_hide">
<pre>FLANNEL_OPTIONS="--etcd-endpoints=https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem"</pre>
</div>
vim /opt/kubernetes/cfg/flanneld</div>
6、创建systemd管理flannel
<div class="cnblogs_code"><img id="code_img_closed_76209447-e29d-47e6-932b-49f851690c9e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_76209447-e29d-47e6-932b-49f851690c9e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_76209447-e29d-47e6-932b-49f851690c9e" class="cnblogs_code_hide">
<pre>
Description=Flanneld overlay address etcd agent
After=network-online.target network.target
Before=docker.service

Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld --ip-masq $FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/subnet.env
Restart=on-failure

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/flanneld.service</div>
7、配置Docker启动指定网段
<div class="cnblogs_code"><img id="code_img_closed_567aa932-da1c-4fee-a258-3c872683deb6" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_567aa932-da1c-4fee-a258-3c872683deb6" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_567aa932-da1c-4fee-a258-3c872683deb6" class="cnblogs_code_hide">
<pre>
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target

Type=notify
EnvironmentFile=/run/flannel/subnet.env
ExecStart=/usr/bin/dockerd $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/docker.service</div>
8、启动flannel与docker、设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable flanneld
systemctl start flanneld
systemctl restart docker</pre>
</div>
9、确认 docker 与 flannel 再同网段
<div class="cnblogs_code"><img id="code_img_closed_dc70bd08-0eed-43b4-8b8c-78ca2a06145e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_dc70bd08-0eed-43b4-8b8c-78ca2a06145e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_dc70bd08-0eed-43b4-8b8c-78ca2a06145e" class="cnblogs_code_hide">
<pre>docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 172.17.26.1 netmask 255.255.255.0 broadcast 172.17.26.255
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 172.17.26.0 netmask 255.255.255.255 broadcast 0.0.0.0</pre>
</div>
ifconfig</div>
10、查看路由信息
<div class="cnblogs_code">
<pre># 1、查看生成的文件
/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379" ls /coreos.com/network/subnets/</pre>
</div>
<div class="cnblogs_code"><img id="code_img_closed_5902f31c-d811-4c15-9e11-d9eec0551c0a" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_5902f31c-d811-4c15-9e11-d9eec0551c0a" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_5902f31c-d811-4c15-9e11-d9eec0551c0a" class="cnblogs_code_hide">
<pre>/coreos.com/network/subnets/172.17.59.0-24
/coreos.com/network/subnets/172.17.23.0-24
/coreos.com/network/subnets/172.17.26.0-24</pre>
</div>
输出：</div>
<div class="cnblogs_code">
<pre># 2、查看指定路由文件
/opt/etcd/bin/etcdctl --ca-file=/opt/etcd/ssl/ca.pem --cert-file=/opt/etcd/ssl/server.pem --key-file=/opt/etcd/ssl/server-key.pem --endpoints="https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379" get /coreos.com/network/subnets/172.17.59.0-24</pre>
</div>
<div class="cnblogs_code"><img id="code_img_closed_c3684e8d-4ba5-4b07-88d9-7e4024f2d049" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_c3684e8d-4ba5-4b07-88d9-7e4024f2d049" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_c3684e8d-4ba5-4b07-88d9-7e4024f2d049" class="cnblogs_code_hide">
<pre># 对应关系
{"PublicIP":"172.16.105.220","BackendType":"vxlan","BackendData":{"VtepMAC":"ae:6b:20:4a:bd:ed"}}</pre>
</div>
输出：</div>
<h2>3.6、部署 kubernetes 单Master集群</h2>
<img src="https://img2018.cnblogs.com/blog/1183448/201908/1183448-20190826141130751-831708556.png" alt="">
 
<ul>
<li>下载二进制包：https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.12.md</li>
<li>下载这个包（kubernetes-server-linux-amd64.tar.gz）就够了，包含了所需的所有组件。</li>
</ul>
1、生成证书 1.1、执行命令生成证书使用的json文件1
<div class="cnblogs_code"><img id="code_img_closed_17a771ea-fc1c-4ee6-9354-5d61dce39fd1" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_17a771ea-fc1c-4ee6-9354-5d61dce39fd1" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_17a771ea-fc1c-4ee6-9354-5d61dce39fd1" class="cnblogs_code_hide">
<pre>{
"signing": {
"default": {
 "expiry": "87600h"
},
"profiles": {
 "kubernetes": {
 "expiry": "87600h",
 "usages": [
 "signing",
 "key encipherment",
 "server auth",
 "client auth"
 ]
 }
}
}
}</pre>
</div>
vim ca-config.json</div>
1.2、执行命令生成证书使用的json文件2
<div class="cnblogs_code"><img id="code_img_closed_7480d040-701d-4123-9509-77da6f2ae007" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_7480d040-701d-4123-9509-77da6f2ae007" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_7480d040-701d-4123-9509-77da6f2ae007" class="cnblogs_code_hide">
<pre>{
"CN": "kubernetes",
"key": {
 "algo": "rsa",
 "size": 2048
},
"names": [
 {
 "C": "CN",
 "L": "Beijing",
 "ST": "Beijing",
 "O": "k8s",
 "OU": "System"
 }
]
}</pre>
</div>
vim ca-csr.json</div>
1.3、执行命令生成CA证书
<div class="cnblogs_code">
<pre>cfssl gencert -initca ca-csr.json | cfssljson -bare ca -</pre>
</div>
1.4、执行命令生成证书使用的json文件、注：添加所有使用到k8s的节点IP。
<div class="cnblogs_code"><img id="code_img_closed_6db7340e-8203-4e9c-97fc-553973e63db5" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_6db7340e-8203-4e9c-97fc-553973e63db5" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_6db7340e-8203-4e9c-97fc-553973e63db5" class="cnblogs_code_hide">
<pre>{
"CN": "kubernetes",
"hosts": [
 "10.0.0.1",
 "127.0.0.1",
 "172.16.105.220",
 "172.16.105.210",
 "多选添加IP，Node节点不用添加",
 "kubernetes",
 "kubernetes.default",
 "kubernetes.default.svc",
 "kubernetes.default.svc.cluster",
 "kubernetes.default.svc.cluster.local"
],
"key": {
 "algo": "rsa",
 "size": 2048
},
"names": [
 {
 "C": "CN",
 "L": "BeiJing",
 "ST": "BeiJing",
 "O": "k8s",
 "OU": "System"
 }
]
}</pre>
</div>
vim server-csr.json</div>
1.5、执行命令生成 apiserver 证书
<div class="cnblogs_code">
<pre>cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server</pre>
</div>
1.6、执行命令生成证书使用的json文件生成 kube-proxy 证书
<div class="cnblogs_code"><img id="code_img_closed_4e345e84-0211-4ccc-b19b-c8052089ea05" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_4e345e84-0211-4ccc-b19b-c8052089ea05" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_4e345e84-0211-4ccc-b19b-c8052089ea05" class="cnblogs_code_hide">
<pre>{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
 "C": "CN",
 "L": "BeiJing",
 "ST": "BeiJing",
 "O": "k8s",
 "OU": "System"
}
]
}</pre>
</div>
vim kube-proxy-csr.json</div>
1.7、执行命令生成 kube-proxy 证书
<div class="cnblogs_code">
<pre>cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy</pre>
</div>
1.8、查看所有生成证书
<div class="cnblogs_code"><img id="code_img_closed_aab37bf2-4648-4df7-a0f6-03da42c630da" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_aab37bf2-4648-4df7-a0f6-03da42c630da" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_aab37bf2-4648-4df7-a0f6-03da42c630da" class="cnblogs_code_hide">
<pre>ca-key.pemca.pemkube-proxy-key.pemkube-proxy.pemserver-key.pemserver.pem</pre>
</div>
 ls *pem</div>
<h3>2、部署Master apiserver 组件</h3>
1、下载到k8s目录解压、进入目录
<div class="cnblogs_code">
<pre>tar -xzvf kubernetes-server-linux-amd64.tar.gz
cd kubernetes/server/bin/</pre>
</div>
2、创建目录
<div class="cnblogs_code">
<pre>mkdir /opt/kubernetes/{bin,cfg,ssl,logs} -p</pre>
</div>
3、将二进制文件导入到相应目录下
<div class="cnblogs_code">
<pre>cp kube-apiserver kube-scheduler kube-controller-manager kubectl /opt/kubernetes/bin</pre>
</div>
4、将生成的证书文件存入到指定文件
<div class="cnblogs_code">
<pre>cp ca.pem ca-key.pem server.pem server-key.pem /opt/kubernetes/ssl/</pre>
</div>
5、创建 token 文件
<div class="cnblogs_code"><img id="code_img_closed_77629aa6-4a3d-42a7-ae33-7ff1f2ddf8fb" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_77629aa6-4a3d-42a7-ae33-7ff1f2ddf8fb" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_77629aa6-4a3d-42a7-ae33-7ff1f2ddf8fb" class="cnblogs_code_hide">
<pre>674c457d4dcf2eefe4920d7dbb6b0ddc,kubelet-bootstrap,10001,"system:kubelet-bootstrap"</pre>
</div>
vim /opt/kubernetes/cfg/token.csv</div>
<div class="cnblogs_code"><img id="code_img_closed_148be488-9090-48e3-b946-42679cba38f4" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_148be488-9090-48e3-b946-42679cba38f4" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_148be488-9090-48e3-b946-42679cba38f4" class="cnblogs_code_hide">
<pre>第一列：随机字符串，自己可生成
第二列：用户名
第三列：UID
第四列：用户组</pre>
</div>
说明</div>
6、创建 apiserver 配置文件、确保配置好生成证书，确保连接etcd
<div class="cnblogs_code"><img id="code_img_closed_3178deec-5af3-417e-babc-cb297f3fbf7f" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_3178deec-5af3-417e-babc-cb297f3fbf7f" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_3178deec-5af3-417e-babc-cb297f3fbf7f" class="cnblogs_code_hide">
<pre>KUBE_APISERVER_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs \
--v=4 \
--etcd-servers=https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379 \
--bind-address=172.16.105.220 \
--secure-port=6443 \
--advertise-address=172.16.105.220 \
--allow-privileged=true \
--service-cluster-ip-range=10.0.0.0/24 \
--service-node-port-range=30000-50000 \
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota,NodeRestriction \
--authorization-mode=RBAC,Node \
--enable-bootstrap-token-auth \
--token-auth-file=/opt/kubernetes/cfg/token.csv \
--tls-cert-file=/opt/kubernetes/ssl/server.pem\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \
--client-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \
--etcd-cafile=/opt/etcd/ssl/ca.pem \
--etcd-certfile=/opt/etcd/ssl/server.pem \
--etcd-keyfile=/opt/etcd/ssl/server-key.pem"</pre>
</div>
vim /opt/kubernetes/cfg/kube-apiserver</div>
<div class="cnblogs_code"><img id="code_img_closed_9a7a994b-140e-4d6c-952b-16d29bb17c88" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_9a7a994b-140e-4d6c-952b-16d29bb17c88" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_9a7a994b-140e-4d6c-952b-16d29bb17c88" class="cnblogs_code_hide">
<pre>参数说明：
· --logtostderr 启用日志
· ---v 日志等级
· --etcd-servers etcd集群地址
· --bind-address 监听地址
· --secure-port https安全端口
· --advertise-address 集群通告地址
· --allow-privileged 启用授权
· --service-cluster-ip-range Service虚拟IP地址段
· --enable-admission-plugins 准入控制模块
· --authorization-mode 认证授权，启用RBAC授权和节点自管理
· --enable-bootstrap-token-auth 启用TLS bootstrap功能，后面会讲到
· --token-auth-file token文件
· --service-node-port-range Service Node类型默认分配端口范围

日志：
# true 日志默认放到/var/log/messages
--logtostderr=true
# false 日志可以指定放到一个目录
--logtostderr=false
--log-dir=/opt/kubernetes/logs</pre>
</div>
参数说明：</div>
7、创建 systemd 管理 apiserver
<div class="cnblogs_code"><img id="code_img_closed_ca87b1c8-49a8-4f75-be70-17d76068acf6" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_ca87b1c8-49a8-4f75-be70-17d76068acf6" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_ca87b1c8-49a8-4f75-be70-17d76068acf6" class="cnblogs_code_hide">
<pre>
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes

EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/kube-apiserver.service</div>
8、启动、并设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver</pre>
</div>
9、查看端口
<div class="cnblogs_code"><img id="code_img_closed_cc7767ed-3bc5-45c0-9b26-963176993d15" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_cc7767ed-3bc5-45c0-9b26-963176993d15" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_cc7767ed-3bc5-45c0-9b26-963176993d15" class="cnblogs_code_hide">
<pre>tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN 5431/kube-apiserver </pre>
</div>
netstat -lnpt | grep 8080</div>
<div class="cnblogs_code"><img id="code_img_closed_85b15e2c-1ea4-40ed-b5fc-26a564bf5b75" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_85b15e2c-1ea4-40ed-b5fc-26a564bf5b75" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_85b15e2c-1ea4-40ed-b5fc-26a564bf5b75" class="cnblogs_code_hide">
<pre>tcp 0 0 172.16.105.220:6443 0.0.0.0:* LISTEN 5431/kube-apiserver </pre>
</div>
netstat -lnpt | grep 6443</div>
3、部署 Master scheduler 组件 1、创建 schduler 配置文件
<div class="cnblogs_code"><img id="code_img_closed_d3279e33-76b2-416f-a3ef-6d19c6db32eb" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_d3279e33-76b2-416f-a3ef-6d19c6db32eb" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_d3279e33-76b2-416f-a3ef-6d19c6db32eb" class="cnblogs_code_hide">
<pre>KUBE_SCHEDULER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect"</pre>
</div>
vim /opt/kubernetes/cfg/kube-scheduler</div>
<div class="cnblogs_code"><img id="code_img_closed_3ebcb5d0-7e4f-4f72-bd1f-4a62cc1c606f" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_3ebcb5d0-7e4f-4f72-bd1f-4a62cc1c606f" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_3ebcb5d0-7e4f-4f72-bd1f-4a62cc1c606f" class="cnblogs_code_hide">
<pre>参数说明：
· --master 连接本地apiserver
· --leader-elect 当该组件启动多个时，自动选举（HA）</pre>
</div>
参数说明：</div>
2、systemd管理schduler组件
<div class="cnblogs_code"><img id="code_img_closed_d7401393-c39a-4af2-8137-15ae6835afb9" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_d7401393-c39a-4af2-8137-15ae6835afb9" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_d7401393-c39a-4af2-8137-15ae6835afb9" class="cnblogs_code_hide">
<pre>
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes

EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/kube-scheduler.service</div>
3、启动并设置开机自启
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler</pre>
</div>
4、查看进程
<div class="cnblogs_code"><img id="code_img_closed_19e625b3-409e-4bfb-8250-190efdd9a041" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_19e625b3-409e-4bfb-8250-190efdd9a041" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_19e625b3-409e-4bfb-8250-190efdd9a041" class="cnblogs_code_hide">
<pre>root 8393 0.5 1.1 45360 21356 ? Ssl 11:23 0:00 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect</pre>
</div>
ps -aux | grep kube-scheduler</div>
4、部署 Master controller-manager 组件 1、创建 controller-manager 配置文件
<div class="cnblogs_code"><img id="code_img_closed_00a63fde-aa46-4305-82c0-a206fb3ad8ba" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_00a63fde-aa46-4305-82c0-a206fb3ad8ba" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_00a63fde-aa46-4305-82c0-a206fb3ad8ba" class="cnblogs_code_hide">
<pre>KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=true \
--v=4 \
--master=127.0.0.1:8080 \
--leader-elect=true \
--address=127.0.0.1 \
--service-cluster-ip-range=10.0.0.0/24 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \
--experimental-cluster-signing-duration=87600h0m0s"</pre>
</div>
vim /opt/kubernetes/cfg/kube-controller-manager</div>
2、systemd管理controller-manager组件
<div class="cnblogs_code"><img id="code_img_closed_ac59d030-7fc0-4183-b36c-2a62c2da537f" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_ac59d030-7fc0-4183-b36c-2a62c2da537f" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_ac59d030-7fc0-4183-b36c-2a62c2da537f" class="cnblogs_code_hide">
<pre>
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/kube-controller-manager.service</div>
3、启动并添加开机自启
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager</pre>
</div>
4、查看进程
<div class="cnblogs_code"><img id="code_img_closed_3e4e26c5-2810-4d9f-88e7-41ed7cda3e09" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_3e4e26c5-2810-4d9f-88e7-41ed7cda3e09" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_3e4e26c5-2810-4d9f-88e7-41ed7cda3e09" class="cnblogs_code_hide">
<pre>root 8966 0.4 1.1 45360 20900 ? Ssl 11:27 0:00 /opt/kubernetes/bin/kube-scheduler --logtostderr=true --v=4 --master=127.0.0.1:8080 --leader-elect</pre>
</div>
ps -aux | grep controller-manager</div>
5、通过 kubectl 检查所有组件状态
<div class="cnblogs_code"><img id="code_img_closed_6995bc7a-f8dd-42aa-ae6e-87df95db3e15" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_6995bc7a-f8dd-42aa-ae6e-87df95db3e15" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_6995bc7a-f8dd-42aa-ae6e-87df95db3e15" class="cnblogs_code_hide">
<pre>NAME STATUS MESSAGE ERROR
controller-manager Healthy ok
scheduler Healthy ok
etcd-2 Healthy {"health":"true"}
etcd-0 Healthy {"health":"true"}
etcd-1 Healthy {"health":"true"}</pre>
</div>
/opt/kubernetes/bin/kubectl get cs</div>
<h3>5、部署 kubecongig 文件</h3>
master 节点配置
1、将kubelet-bootstrap用户绑定到系统集群角色。生成的token文件中定义的角色。
<div class="cnblogs_code">
<pre># 主要为kuelet办法证书的最小全权限
/opt/kubernetes/bin/kubectl create clusterrolebinding kubelet-bootstrap \
--clusterrole=system:node-bootstrapper \
--user=kubelet-bootstrap</pre>
</div>
2、创建kubeconfig文件、在生成kubernetes证书的目录下执行以下命令生成kubeconfig文件：
<div class="cnblogs_code"><img id="code_img_closed_f4afe1b6-9f13-4abf-a563-477717aabb5e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_f4afe1b6-9f13-4abf-a563-477717aabb5e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_f4afe1b6-9f13-4abf-a563-477717aabb5e" class="cnblogs_code_hide">
<pre># 创建kubelet bootstrapping kubeconfig 
BOOTSTRAP_TOKEN=674c457d4dcf2eefe4920d7dbb6b0ddc
KUBE_APISERVER="https://172.16.105.220:6443"

# 设置集群参数
kubectl config set-cluster kubernetes \
--certificate-authority=/root/k8s/k8s-cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=bootstrap.kubeconfig

# 设置客户端认证参数
kubectl config set-credentials kubelet-bootstrap \
--token=${BOOTSTRAP_TOKEN} \
--kubeconfig=bootstrap.kubeconfig

# 设置上下文参数
kubectl config set-context default \
--cluster=kubernetes \
--user=kubelet-bootstrap \
--kubeconfig=bootstrap.kubeconfig

# 设置默认上下文
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig

#----------------------

# 创建kube-proxy kubeconfig文件

kubectl config set-cluster kubernetes \
--certificate-authority=/root/k8s/k8s-cert/ca.pem \
--embed-certs=true \
--server=${KUBE_APISERVER} \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-credentials kube-proxy \
--client-certificate=/root/k8s/k8s-cert/kube-proxy.pem \
--client-key=/root/k8s/k8s-cert/kube-proxy-key.pem \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig

kubectl config set-context default \
--cluster=kubernetes \
--user=kube-proxy \
--kubeconfig=kube-proxy.kubeconfig

kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig</pre>
</div>
vim kubeconfig.sh</div>
3、执行脚本
<div class="cnblogs_code">
<pre>bash kubeconfig.sh</pre>
</div>
4、将生成的kube-proxy.kubeconfig与bootstrap.kubeconfig copy 到 Node 机器内。
<div class="cnblogs_code">
<pre>scp bootstrap.kubeconfig kube-proxy.kubeconfig root@172.16.105.230:/opt/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig root@172.16.105.213:/opt/kubernetes/cfg/</pre>
</div>
<h3>6、部署Node kubelet 组件</h3>
1、Node节点创建目录
<div class="cnblogs_code">
<pre>mkdir -p /opt/kubernetes/{cfg,bin,logs,ssl}</pre>
</div>
2、copy下列文件到指定目录下
<ul>
<li>使用：/kubernetes/server/bin/kubelet</li>
<li>使用：/kubernetes/server/bin/kube-proxy</li>
<li>将上面两个文件copy到Node端/opt/kubernetes/bin/目录下</li>
</ul>
3、创建 kubelet 配置文件
<div class="cnblogs_code"><img id="code_img_closed_5746f300-8e13-4ecb-9db4-f9969586ccab" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_5746f300-8e13-4ecb-9db4-f9969586ccab" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_5746f300-8e13-4ecb-9db4-f9969586ccab" class="cnblogs_code_hide">
<pre>KUBELET_OPTS="--logtostderr=false \
--log-dir=/opt/kubernetes/logs/ \
--v=4 \
--hostname-override=172.16.105.213 \
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \
--config=/opt/kubernetes/cfg/kubelet.config \
--cert-dir=/opt/kubernetes/ssl \
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"</pre>
</div>
vim /opt/kubernetes/cfg/kubelet</div>
<div class="cnblogs_code"><img id="code_img_closed_ec97223f-2485-416d-b01d-338d83e0934b" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_ec97223f-2485-416d-b01d-338d83e0934b" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_ec97223f-2485-416d-b01d-338d83e0934b" class="cnblogs_code_hide">
<pre>参数说明：
· --hostname-override 在集群中显示的主机名
· --kubeconfig 指定kubeconfig文件位置，会自动生成
· --bootstrap-kubeconfig 指定刚才生成的bootstrap.kubeconfig文件
· --cert-dir 颁发证书存放位置
· --pod-infra-container-image 管理Pod网络的镜像</pre>
</div>
参数说明：</div>
2、创建 kubelet.config 配置文件
<div class="cnblogs_code"><img id="code_img_closed_20e41d22-54ec-47a6-b9ad-2e97132d2a95" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_20e41d22-54ec-47a6-b9ad-2e97132d2a95" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_20e41d22-54ec-47a6-b9ad-2e97132d2a95" class="cnblogs_code_hide">
<pre>kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 172.16.105.213
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS: ["10.0.0.2"]
clusterDomain: cluster.local.
failSwapOn: false
authentication:
anonymous:
enabled: true</pre>
</div>
vim /opt/kubernetes/cfg/kubelet.config</div>
3、systemd 管理 kubelet 组件
<div class="cnblogs_code"><img id="code_img_closed_a9c4ca01-ea8e-4037-a767-65a30b00a256" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_a9c4ca01-ea8e-4037-a767-65a30b00a256" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_a9c4ca01-ea8e-4037-a767-65a30b00a256" class="cnblogs_code_hide">
<pre>
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service

EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet $KUBELET_OPTS
Restart=on-failure
KillMode=process

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/kubelet.service</div>
4、启动并设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable kubelet.service
systemctl start kubelet.service</pre>
</div>
5、查看进程
<div class="cnblogs_code"><img id="code_img_closed_745ef0d6-b5c7-4654-8943-2d6ec3d1e7f4" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_745ef0d6-b5c7-4654-8943-2d6ec3d1e7f4" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_745ef0d6-b5c7-4654-8943-2d6ec3d1e7f4" class="cnblogs_code_hide">
<pre>root 246070.81.7 626848 69140 ? Ssl16:03 0:05 /opt/kubernetes/bin/kubelet --logtostderr=false --log-dir=/opt/kubernetes/logs/ --v=4 --hostname-override=172.16.105.213 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0</pre>
</div>
ps -aux | grep kubelet</div>
6、Master 端审批Node 加入集群：
<ul>
<li>启动后还没加入到集群中，需要手动允许该节点才可以。</li>
<li>在Master节点查看请求签名的Node：</li>
</ul>
7、查看请求加入集群的Node
<div class="cnblogs_code"><img id="code_img_closed_7b763052-b50d-40f0-a801-85c9ebcc3a7e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_7b763052-b50d-40f0-a801-85c9ebcc3a7e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_7b763052-b50d-40f0-a801-85c9ebcc3a7e" class="cnblogs_code_hide">
<pre>NAME AGE REQUESTOR CONDITION
node-csr-7ZHhg19mVh1w2gfJOh55eaBsRisA_wT8EHZQfqCLPLE 21s kubelet-bootstrap Pending
node-csr-weeFsR6VVUNIHyohOgaGvy2Hr6M9qSUIkoGjQ_mUyOo 28s kubelet-bootstrap Pending</pre>
</div>
kubectl get csr</div>
8、同意请求让Node节点加入
<div class="cnblogs_code">
<pre>kubectl certificate approve node-csr-7ZHhg19mVh1w2gfJOh55eaBsRisA_wT8EHZQfqCLPLE
kubectl certificate approve node-csr-weeFsR6VVUNIHyohOgaGvy2Hr6M9qSUIkoGjQ_mUyOo</pre>
</div>
9、查看加入节点
<div class="cnblogs_code"><img id="code_img_closed_c4767766-611d-4177-a323-f1878518f6a3" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_c4767766-611d-4177-a323-f1878518f6a3" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_c4767766-611d-4177-a323-f1878518f6a3" class="cnblogs_code_hide">
<pre>NAME STATUS ROLES AGE VERSION
172.16.105.213 Ready <none> 42s v1.12.1
172.16.105.230 Ready <none> 57s v1.12.1</pre>
</div>
kubectl get node</div>
<h3>7、部署Node kube-proxy组件</h3>
1、创建 kube-proxy 配置文件
<div class="cnblogs_code"><img id="code_img_closed_a0382842-9fa9-45f5-b4b5-3f0288ef0d41" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_a0382842-9fa9-45f5-b4b5-3f0288ef0d41" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_a0382842-9fa9-45f5-b4b5-3f0288ef0d41" class="cnblogs_code_hide">
<pre>KUBE_PROXY_OPTS="--logtostderr=true \
--v=4 \
--hostname-override=172.16.105.213 \
--cluster-cidr=10.0.0.0/24 \
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"</pre>
</div>
vim /opt/kubernetes/cfg/kube-proxy</div>
2、systemd管理kube-proxy组件
<div class="cnblogs_code"><img id="code_img_closed_5cb2560b-029f-452d-a690-932e522a7592" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_5cb2560b-029f-452d-a690-932e522a7592" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_5cb2560b-029f-452d-a690-932e522a7592" class="cnblogs_code_hide">
<pre>
Description=Kubernetes Proxy
After=network.target

EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy $KUBE_PROXY_OPTS
Restart=on-failure

WantedBy=multi-user.target</pre>
</div>
vim /usr/lib/systemd/system/kube-proxy.service</div>
3、启动并设置开机自启动
<div class="cnblogs_code">
<pre>systemctl daemon-reload
systemctl enable kube-proxy
systemctl start kube-proxy</pre>
</div>
4、查看进程
<div class="cnblogs_code"><img id="code_img_closed_aebb19a2-12ff-4f3e-bcc8-c22be2b5bda7" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_aebb19a2-12ff-4f3e-bcc8-c22be2b5bda7" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_aebb19a2-12ff-4f3e-bcc8-c22be2b5bda7" class="cnblogs_code_hide">
<pre>root 271660.30.541588 21332 ? Ssl16:16 0:00 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=172.16.105.213 --cluster-cidr=10.0.0.0/24 --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig</pre>
</div>
ps -aux | grep kube-proxy</div>
<h3>8、其他设置</h3>
1、解决：将匿名用户绑定到系统用户
<div class="cnblogs_code">
<pre>kubectl create clusterrolebinding system:anonymous --clusterrole=cluster-admin --user=system:anonymous</pre>
</div>
<h2> 3.7、部署 kubernetes 多Master集群</h2>
<img src="https://img2018.cnblogs.com/blog/1183448/201908/1183448-20190826141327930-2131307766.png" alt="" width="965" height="325">
<h3>1、Master2配置部署</h3>
<ul>
<li>注：Master节点2配置与单Master相同下面我这里只直接略过相同配置。</li>
<li>注：直接复制配置文件可能会导致etcd链接问题</li>
<li>注：最好以master为etcd端。</li>
</ul>
1、修改Master02配置文件中的IP，更改为Master02IP
<div class="cnblogs_code"><img id="code_img_closed_58a0d5cd-f24e-42d0-8811-fb5940508b13" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_58a0d5cd-f24e-42d0-8811-fb5940508b13" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_58a0d5cd-f24e-42d0-8811-fb5940508b13" class="cnblogs_code_hide">
<pre>--bind-address=172.16.105.212
--advertise-address=172.16.105.212</pre>
</div>
vim kube-apiserver</div>
2、启动Master02 k8s
<div class="cnblogs_code">
<pre>systemctl start kube-apiserver
systemctl start kube-scheduler
systemctl start kube-controller-manager</pre>
</div>
3、查看集群状态
<div class="cnblogs_code"><img id="code_img_closed_c3867393-fcce-4c78-9f7c-48a4da2a7586" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_c3867393-fcce-4c78-9f7c-48a4da2a7586" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_c3867393-fcce-4c78-9f7c-48a4da2a7586" class="cnblogs_code_hide">
<pre>NAME STATUS MESSAGE ERROR
etcd-1 Healthy {"health":"true"}
etcd-2 Healthy {"health":"true"}
controller-manager Healthy ok
scheduler Healthy ok
etcd-0 Healthy {"health":"true"} </pre>
</div>
kubectl get cs</div>
5、查看etcd连接状态
<div class="cnblogs_code"><img id="code_img_closed_33cee399-0bf9-4cfc-ba51-4fa82d6b0fa6" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_33cee399-0bf9-4cfc-ba51-4fa82d6b0fa6" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_33cee399-0bf9-4cfc-ba51-4fa82d6b0fa6" class="cnblogs_code_hide">
<pre>NAME STATUS ROLES AGE VERSION
172.16.105.213 Ready <none> 41h v1.12.1
172.16.105.230 Ready <none> 41h v1.12.1</pre>
</div>
kubectl get node</div>
2、部署 Nginx 负载均衡
<ul>
<li>注：保证系统时间统一证书正常使用</li>
<li>nginx官网：http://www.nginx.org</li>
<li>documentation --> Installing nginx --> packages</li>
</ul>
1、复制nginx官方源写入到/etc/yum.repos.d/nginx.repo、修该centos版本
<div class="cnblogs_code"><img id="code_img_closed_fdf3b9d1-ad3e-4414-ba46-81de364ad8fd" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_fdf3b9d1-ad3e-4414-ba46-81de364ad8fd" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_fdf3b9d1-ad3e-4414-ba46-81de364ad8fd" class="cnblogs_code_hide">
<pre>
name=nginx stable repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key

name=nginx mainline repo
baseurl=http://nginx.org/packages/mainline/centos/7/$basearch/
gpgcheck=1
enabled=0
gpgkey=https://nginx.org/keys/nginx_signing.key</pre>
</div>
vim /etc/yum.repos.d/nginx.repo</div>
2、从新加载yum
<div class="cnblogs_code">
<pre>yum clean all
yum makecache</pre>
</div>
3、安装 nginx
<div class="cnblogs_code">
<pre>yum install nginx -y</pre>
</div>
4、修该配置文件，events同级添加
<div class="cnblogs_code"><img id="code_img_closed_71653438-ca79-43d8-b232-dc1a7d113462" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_71653438-ca79-43d8-b232-dc1a7d113462" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_71653438-ca79-43d8-b232-dc1a7d113462" class="cnblogs_code_hide">
<pre>events {
worker_connections1024;
}

stream {
log_format main "$remote_addr $upstream_addr - $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
 server 172.16.105.220:6443;
 server 172.16.105.210:6443;
}
server {
 listen 172.16.105.231:6443;
 proxy_pass k8s-apiserver;
}
}</pre>
</div>
vim /etc/nginx/nginx.conf</div>
<div class="cnblogs_code"><img id="code_img_closed_8dfc90c9-aa67-497b-93c4-aa38815cb9b6" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_8dfc90c9-aa67-497b-93c4-aa38815cb9b6" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_8dfc90c9-aa67-497b-93c4-aa38815cb9b6" class="cnblogs_code_hide">
<pre>参数说明：
# 创建四层负载均衡
stream {
# 记录日志
log_format main "$remote_addr $upstream_addr $time_local $status"
# 日志存放路径
access_log /var/log/nginx/k8s-access.log main;
# 创建调度集群 k8s-apiserver 为服务名称
upstream k8s-apiserver {
 server 172.16.105.220:6443;
 server 172.16.105.210:6443;
}
# 创建监听服务
 server {
 # 本地监听访问开启的使用IP与端口
 listen 172.16.105.231:6443;
 # 调度的服务名称，由于是4层则不是用http
 proxy_pass k8s-apiserver;
}
}</pre>
</div>
参数说明：</div>
5、启动nginx并生效配置文件
<div class="cnblogs_code">
<pre>systemctl start nginx</pre>
</div>
6、查看监听端口
<div class="cnblogs_code"><img id="code_img_closed_7233326f-a3df-4c92-9446-75aa63867c19" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_7233326f-a3df-4c92-9446-75aa63867c19" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_7233326f-a3df-4c92-9446-75aa63867c19" class="cnblogs_code_hide">
<pre>tcp 0 0 172.16.105.231:6443 0.0.0.0:* LISTEN 19067/nginx: master</pre>
</div>
netstat -lnpt | grep 6443</div>
8、修改每个Node 节点中配置文件。将引用的连接端，改为该负载均衡的机器内。
<div class="cnblogs_code">
<pre>vim bootstrap.kubeconfig
server: https://172.16.105.231:6443

vim kubelet.kubeconfig
server: https://172.16.105.231:6443

vim kube-proxy.kubeconfig
server: https://172.16.105.231:6443</pre>
</div>
9、重启 kubelet Node 客户端
<div class="cnblogs_code">
<pre>systemctl restart kubelet
systemctl restart kube-proxy</pre>
</div>
10、查看Node 启动进程
<div class="cnblogs_code"><img id="code_img_closed_57d432a6-fb8a-441b-aec8-b4a9f2fd2140" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_57d432a6-fb8a-441b-aec8-b4a9f2fd2140" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_57d432a6-fb8a-441b-aec8-b4a9f2fd2140" class="cnblogs_code_hide">
<pre>root 23226 0.0 0.4 300552 16460 ? Ssl Aug08 0:25 /opt/kubernetes/bin/flanneld --ip-masq --etcd-endpoints=https://172.16.105.220:2379,https://172.16.105.230:2379,https://172.16.105.213:2379 -etcd-cafile=/opt/etcd/ssl/ca.pem -etcd-certfile=/opt/etcd/ssl/server.pem -etcd-keyfile=/opt/etcd/ssl/server-key.pem
root 26986 1.5 1.5 632676 60740 ? Ssl 11:30 0:01 /opt/kubernetes/bin/kubelet --logtostderr=false --log-dir=/opt/kubernetes/logs/ --v=4 --hostname-override=172.16.105.213 --kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig --bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig --config=/opt/kubernetes/cfg/kubelet.config --cert-dir=/opt/kubernetes/ssl --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0
root 27584 0.7 0.5 41588 19896 ? Ssl 11:32 0:00 /opt/kubernetes/bin/kube-proxy --logtostderr=true --v=4 --hostname-override=172.16.105.213 --cluster-cidr=10.0.0.0/24 --kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig</pre>
</div>
ps -aux | grep kube</div>
11、重启Master kube-apiserver
<div class="cnblogs_code">
<pre>systemctl restart kube-apiserver</pre>
</div>
12、查看Nginx日志
<div class="cnblogs_code"><img id="code_img_closed_8137b3bf-6d2d-4c8a-b6a6-9c6187d12529" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_8137b3bf-6d2d-4c8a-b6a6-9c6187d12529" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_8137b3bf-6d2d-4c8a-b6a6-9c6187d12529" class="cnblogs_code_hide">
<pre>172.16.105.213 172.16.105.220:6443 09/Aug/2019:13:34:59 +0800 200
172.16.105.230 172.16.105.220:6443 09/Aug/2019:13:34:59 +0800 200
172.16.105.213 172.16.105.220:6443 09/Aug/2019:13:34:59 +0800 200
172.16.105.230 172.16.105.220:6443 09/Aug/2019:13:34:59 +0800 200
172.16.105.230 172.16.105.220:6443 09/Aug/2019:13:35:00 +0800 200</pre>
</div>
tail -f /var/log/nginx/k8s-access.log</div>
<h3>3、部署 Nginx2+keepalived 高可用</h3>
<ul>
<li>注：VIP 要设置为证书授权过得ip否则会无法通过外网访问</li>
<li>注：安装Nginx2与单Nginx的安装步骤相同，这里我不再重复部署，只讲解重点。</li>
</ul>
1、Nginx1与Nginx2安装keepalive高可用
<div class="cnblogs_code">
<pre>yum -y install keepalived</pre>
</div>
2、修改Nginx1 Master 主配置文件
<div class="cnblogs_code"><img id="code_img_closed_a2d82d9e-90b5-478e-88cd-7e15057f0c1e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_a2d82d9e-90b5-478e-88cd-7e15057f0c1e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_a2d82d9e-90b5-478e-88cd-7e15057f0c1e" class="cnblogs_code_hide">
<pre>! Configuration File for keepalived
global_defs {
# 接收邮件地址
 notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
 notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

# 通过vrrp协议检查本机nginx服务是否正常
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
state MASTER
interface ens32
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 100 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒

# 密码认证
 authentication {
 auth_type PASS
 auth_pass 1111
}
# VIP
 virtual_ipaddress {
 192.168.1.100/24
}

# 使用检查脚本
 track_script {
 check_nginx
}
}</pre>
</div>
vim /etc/keepalived/keepalived.conf</div>
3、修改Nginx2 Slave 主配置文件
<div class="cnblogs_code"><img id="code_img_closed_b95a25a6-015f-4c0f-a230-ae2c4a1a1761" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_b95a25a6-015f-4c0f-a230-ae2c4a1a1761" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_b95a25a6-015f-4c0f-a230-ae2c4a1a1761" class="cnblogs_code_hide">
<pre>! Configuration File for keepalived
global_defs {
# 接收邮件地址
 notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
# 邮件发送地址
 notification_email_from Alexandre.Cassen@firewall.loc
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id NGINX_MASTER
}

# 通过vrrp协议检查本机nginx服务是否正常
vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
state BACKUP
interface ens32
virtual_router_id 51 # VRRP 路由 ID实例，每个实例是唯一的
priority 90 # 优先级，备服务器设置 90
advert_int 1 # 指定VRRP 心跳包通告间隔时间，默认1秒

# 密码认证
 authentication {
 auth_type PASS
 auth_pass 1111
}
# VIP
 virtual_ipaddress {
 192.168.1.100/24
}

# 使用检查脚本
 track_script {
 check_nginx
}
}</pre>
</div>
vim /etc/keepalived/keepalived.conf</div>
 4、Ngin1与Nginx2创建检查脚本
<div class="cnblogs_code"><img id="code_img_closed_41b36e76-9743-4773-8b7d-4f4eac13a67b" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_41b36e76-9743-4773-8b7d-4f4eac13a67b" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_41b36e76-9743-4773-8b7d-4f4eac13a67b" class="cnblogs_code_hide">
<pre># 检查nginx进程数
count=$(ps -ef |grep nginx |egrep -cv "grep|$$")

if [ "$count" -eq 0 ];then
systemctl stop keepalived
fi</pre>
</div>
vim /etc/keepalived/check_nginx.sh</div>
5、给脚本添加权限
<div class="cnblogs_code">
<pre>chmod +x /etc/keepalived/check_nginx.sh</pre>
</div>
6、Ngin1与Nginx2启动keepalived
<div class="cnblogs_code">
<pre>systemctl start keepalived</pre>
</div>
7、查看进程
<div class="cnblogs_code"><img id="code_img_closed_b4fea762-95e8-43c0-b0cf-dc8fd525f6bd" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_b4fea762-95e8-43c0-b0cf-dc8fd525f6bd" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_b4fea762-95e8-43c0-b0cf-dc8fd525f6bd" class="cnblogs_code_hide">
<pre>root 1969 0.0 0.1 118608 1396 ? Ss 09:41 0:00 /usr/sbin/keepalived -D
root 1970 0.0 0.2 120732 2832 ? S 09:41 0:00 /usr/sbin/keepalived -D
root 1971 0.0 0.2 120732 2380 ? S 09:41 0:00 /usr/sbin/keepalived -D</pre>
</div>
ps aux | grep keepalived</div>
8、Master 查看虚拟IP
<div class="cnblogs_code"><img id="code_img_closed_cd127e73-00af-4c88-81de-329932e3627e" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_cd127e73-00af-4c88-81de-329932e3627e" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_cd127e73-00af-4c88-81de-329932e3627e" class="cnblogs_code_hide">
<pre>ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3d:1c:d0 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.115/24 brd 192.168.1.255 scope global dynamic ens32
 valid_lft 5015sec preferred_lft 5015sec
inet 192.168.1.100/24 scope global secondary ens32
 valid_lft forever preferred_lft forever
inet6 fe80::4db8:8591:9f94:8837/64 scope link
 valid_lft forever preferred_lft forever</pre>
</div>
ip addr</div>
9、Slave 6、查看虚拟IP（没有就正常）
<div class="cnblogs_code"><img id="code_img_closed_ae010fed-694e-474a-a281-25b3cdc28948" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_ae010fed-694e-474a-a281-25b3cdc28948" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_ae010fed-694e-474a-a281-25b3cdc28948" class="cnblogs_code_hide">
<pre>ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:09:b3:c4 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.112/24 brd 192.168.1.255 scope global dynamic ens32
 valid_lft 7200sec preferred_lft 7200sec
inet6 fe80::1dbe:11ff:f093:ef49/64 scope link
 valid_lft forever preferred_lft forever</pre>
</div>
ip addr</div>
10、测试
<div class="cnblogs_code"><img id="code_img_closed_da2a38d8-84fa-4416-8939-e9d2bb143dad" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_da2a38d8-84fa-4416-8939-e9d2bb143dad" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_da2a38d8-84fa-4416-8939-e9d2bb143dad" class="cnblogs_code_hide">
<pre>测试IP飘逸
1、关闭Master Nginx1
pkill nginx
2、查看Slave Nginx2 虚拟IP是否飘逸
ip addr
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:09:b3:c4 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.112/24 brd 192.168.1.255 scope global dynamic ens32
 valid_lft 4387sec preferred_lft 4387sec
inet 192.168.1.100/24 scope global secondary ens32
 valid_lft forever preferred_lft forever
inet6 fe80::1dbe:11ff:f093:ef49/64 scope link
 valid_lft forever preferred_lft forever
3、启动Master Nginx1 keepalived 测试ip飘回
systemctl start nginx
systemctl start keepalived
4、查看Nginx1 vip
ip addr
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3d:1c:d0 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.115/24 brd 192.168.1.255 scope global dynamic ens32
 valid_lft 7010sec preferred_lft 7010sec
inet 192.168.1.100/24 scope global secondary ens32
 valid_lft forever preferred_lft forever
inet6 fe80::4db8:8591:9f94:8837/64 scope link
 valid_lft forever preferred_lft forever</pre>
</div>
测试IP飘逸</div>
 11、修改Nginx1 与 Nginx2 代理监听
<div class="cnblogs_code"><img id="code_img_closed_3ef754b0-6584-492d-b2bb-29d0334b1b49" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_3ef754b0-6584-492d-b2bb-29d0334b1b49" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_3ef754b0-6584-492d-b2bb-29d0334b1b49" class="cnblogs_code_hide">
<pre>stream {
log_format main "$remote_addr $upstream_addr - $time_local $status";
access_log /var/log/nginx/k8s-access.log main;
upstream k8s-apiserver {
 server 192.168.1.108:6443;
 server 192.168.1.109:6443;
}
server {
 listen 0.0.0.0:6443;
 proxy_pass k8s-apiserver;
}
}</pre>
</div>
vim /etc/nginx/nginx.conf</div>
12、重启nginx
<div class="cnblogs_code">
<pre>systemctl restart nginx</pre>
</div>
 13、接入K8S  修改所有Node配置文件IP为 VIP
1、修改配置文件
<div class="cnblogs_code">
<pre>vim bootstrap.kubeconfig
server: https://192.168.1.100:6443
vim kube-proxy.kubeconfig
server: https://192.168.1.100:6443</pre>
</div>
2、重启Node
<div class="cnblogs_code">
<pre>systemctl restart kubelet
systemctl restart kube-proxy</pre>
</div>
3、查看Master nginx1 日志
<div class="cnblogs_code"><img id="code_img_closed_a02fa8c1-7e41-46c4-825e-8f84f93b679c" class="code_img_closed" src="https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt=""><img id="code_img_opened_a02fa8c1-7e41-46c4-825e-8f84f93b679c" class="code_img_opened" style="display: none" src="https://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="">
<div id="cnblogs_code_open_a02fa8c1-7e41-46c4-825e-8f84f93b679c" class="cnblogs_code_hide">
<pre>192.168.1.111 192.168.1.108:6443 - 22/Aug/2019:11:02:36 +0800 200
192.168.1.111 192.168.1.109:6443 - 22/Aug/2019:11:02:36 +0800 200
192.168.1.110 192.168.1.108:6443 - 22/Aug/2019:11:02:36 +0800 200
192.168.1.110 192.168.1.109:6443 - 22/Aug/2019:11:02:36 +0800 200
192.168.1.111 192.168.1.108:6443 - 22/Aug/2019:11:02:37 +0800 200</pre>
</div>
tail /var/log/nginx/k8s-access.log -f</div>
  
来源：https://www.cnblogs.com/xiangsikai/p/11410261.html

頁: [1]

琼殿技术论坛's Archiver

Kubernetes 企业级集群部署方式