静心寒 發表於 2022-8-11 14:51:00

kubernetes使用二进制安装部署k8s的v1.23版本安装方法及详细步骤

<h1 id="二进制安装k8s环境">二进制安装k8s环境</h1>
<p>@</p><div class="toc"><div class="toc-container-header">目录</div><ul><li>二进制安装k8s环境<ul><li>前置环境准备及网段规划</li></ul></li><li>一、k8s基础环境准备<ul><li>1.1 配置静态IP</li><li>1.2 配置主机名</li><li>1.3 配置 hosts 文件</li><li>1.4 免密登录</li><li>1.5 关闭 firewalld 防火墙</li><li>1.6 关闭 selinux</li><li>1.7 关闭交换分区 swap</li><li>1.8 修改内核参数</li><li>1.9 配置阿里云 repo 源</li><li>1.10 配置时间同步</li><li>1.11 安装 iptables</li><li>1.12 安装环境基础软件包</li><li>1.13 安装 docker环境</li><li>1.14 配置 docker 镜像加速器</li></ul></li><li>二、k8s环境部署<ul><li>2.1 搭建 etcd 集群<ul><li>2.1.1 配置 etcd 工作目录</li><li>2.1.2 安装签发证书工具 cfssl</li><li>2.1.3 配置 ca 证书</li><li>2.1.4 生成 etcd 证书</li></ul></li><li>2.1.5 部署 etcd 集群</li><li>2.2 部署 apiserver 组件<ul><li>TLS bootstrapping 具体引导过程</li></ul></li><li>2.3 部署 kubectl 组件<ul><li>2.3.1 kubectl组件介绍</li><li>2.3.2 创建 csr 请求文件</li><li>2.3.2 生成客户端的证书</li><li>2.3.3 配置安全上下文</li></ul></li></ul></li><li>查看集群组件状态</li><li>配置 kubectl 子命令补全，依次执行下面命令即可<ul><li>2.4 部署 kube-controller-manager 组件<ul><li>2.4.1 创建 kube-controller-manager csr 请求文件</li><li>2.4.2 生成 kube-controller-manager证书</li><li>2.4.3 创建 kube-controller-manager 的 kubeconfig</li><li>2.4.4 创建kube-controller-manager配置文件</li><li>2.4.5 创建kube-controller-manager启动文件</li><li>2.4.6 启动 kube-controller-manager服务</li></ul></li><li>2.5部署 kube-scheduler 组件<ul><li>2.5.1 创建kube-scheduler的csr 请求</li><li>2.5.2 生成kube-scheduler证书</li><li>2.5.3 创建 kube-scheduler 的 kubeconfig文件</li><li>2.5.4 创建配置文件 kube-scheduler的配置文件</li><li>2.5.5 创建kube-scheduler的服务启动文件</li><li>2.5.6 拷贝文件到master2节点并启动服务</li></ul></li><li>2.6. 部署 kubelet 组件<ul><li>2.6.1 创建 kubelet-bootstrap.kubeconfig</li><li>2.6.2 创建配置文件 kubelet.json</li><li>2.6.3 创建kubelet服务启动文件</li><li>2.6.4 拷贝kubelet的证书及配置文件到node节点</li><li>2.6.5 在node节点上启动 kubelet 服务</li><li>2.6.6 在master节点审批node节点的请求</li></ul></li><li>2.7 部署 kube-proxy 组件<ul><li>2.7.1 创建kube-proxy的 csr 请求</li><li>2.7.2 生成证书</li><li>2.7.3 创建 kubeconfig 文件</li><li>2.7.4 创建 kube-proxy 配置文件</li><li>2.7.5 创建kube-proxy服务启动文件</li><li>2.7.6 拷贝kube-proxy文件到node节点上</li><li>2.7.7 启动kube-proxy服务</li></ul></li><li>2.8 部署 calico 组件</li></ul></li><li>解压离线镜像压缩包</li><li>把 calico.yaml 文件上传到 k8s-master1 上的的/data/work 目录<ul><li>2.9 部署 coredns 组件</li><li>2.10 查看集群状态</li></ul></li><li>三、集群组件功能验证测试<ul><li>3.1 对系统用户 kubernetes 做授权</li><li>3.2 测试 k8s 集群部署 tomcat 服务</li><li>3.3 验证 cordns 是否正常</li></ul></li><li>四、 实现k8s apiserver 高可用</li></ul></div><p></p>
<h2 id="前置环境准备及网段规划">前置环境准备及网段规划</h2>
<p><em><strong>*k8s 环境规划：*</strong></em></p>
<p><em><strong>*Pod*</strong></em> <em><strong>*网段：*</strong></em>        <em><strong>*10.0.0.0/16*</strong></em></p>
<p><em><strong>*Service 网段：10.255.0.0/16*</strong></em></p>
<p><em><strong>*实验环境规划：*</strong></em></p>
<p><em><strong>*操作系统：centos7.6*</strong></em></p>
<p>配置： 4Gib 内存/4vCPU/100G 硬盘开启虚拟机的虚拟化：</p>
<table>
<thead>
<tr>
<th><strong>集群角色</strong></th>
<th><strong>Ip</strong></th>
<th><strong>主机名</strong></th>
<th><strong>安装的组件</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>控制节点</strong></td>
<td><strong>192.168.7.10</strong></td>
<td><strong>k8s-master1</strong></td>
<td><strong>apiserver、controller-manager、****scheduler、etcd、docker、keepalived、 nginx</strong></td>
</tr>
<tr>
<td><strong>控制节点</strong></td>
<td><strong>192.168.7.11</strong></td>
<td><strong>k8s-maste2</strong></td>
<td><strong>apiserver、controller-manager、****scheduler、etcd、docker、keepalived、 nginx</strong></td>
</tr>
<tr>
<td><strong>工作节点</strong></td>
<td><strong>192.168.1.13</strong></td>
<td><strong>k8s-node1</strong></td>
<td><strong>kubelet、kube-proxy、docker、****calico、coredns</strong></td>
</tr>
<tr>
<td><strong>Vip</strong></td>
<td><strong>192.168.7.99</strong></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
<h1 id="一k8s基础环境准备">一、k8s基础环境准备</h1>
<h2 id="11-配置静态ip">1.1 配置静态IP</h2>
<p>虚拟机或者物理机配置成静态 ip 地址，这样机器重新启动后 ip 地址也不会发生改变。</p>
<p>以 <strong>k8s-master1</strong> 主机修改静态 IP 为例</p>
<pre><code class="language-bash">修改/etc/sysconfig/network-scripts/ifcfg-ens33 文件，变成如下：
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
IPADDR=192.168.7.10
NETMASK=255.255.255.0
GATEWAY=192.168.7.2
DNS1=192.168.7.2
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=eth0
DEVICE=eth0
ONBOOT=yes
</code></pre>
<h2 id="12-配置主机名">1.2 配置主机名</h2>
<p>#配置主机名：</p>
<pre><code class="language-bash">在 192.168.7.10 上执行如下：
hostnamectl set-hostname k8s-master1 &amp;&amp; bash
在 192.168.7.11 上执行如下：
hostnamectl set-hostname k8s-master2 &amp;&amp; bash
在 192.168.7.13 上执行如下：
hostnamectl set-hostname k8s-node1 &amp;&amp; bash
</code></pre>
<h2 id="13-配置-hosts-文件">1.3 配置 hosts 文件</h2>
<p>#修改 k8s-master1、k8s-master2、k8s-node1 机器的/etc/hosts 文件，增加如下四行：</p>
<pre><code class="language-bash">192.168.7.10 k8s-master1 master1
192.168.7.11 k8s-master2 master2
192.168.7.13 k8s-node1 node1
</code></pre>
<h2 id="14-免密登录">1.4 免密登录</h2>
<p>配置主机之间无密码登录每台机器都按照如下操作</p>
<p>#生成 ssh 密钥对</p>
<p>ssh-keygen -t rsa       #一路回车，不输入密码把本地的 ssh 公钥文件安装到远程主机对应的账户</p>
<pre><code class="language-bash">ssh-copy-id -i .ssh/id_rsa.pub k8s-master1
ssh-copy-id -i .ssh/id_rsa.pub k8s-master2
ssh-copy-id -i .ssh/id_rsa.pub k8s-node1
</code></pre>
<h2 id="15-关闭-firewalld-防火墙">1.5 关闭 firewalld 防火墙</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">systemctl stop firewalld&amp;&amp; systemctl disable firewalld &amp;&amp; systemctl status firewalld
</code></pre>
<h2 id="16-关闭-selinux">1.6 关闭 selinux</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
#修改 selinux 配置文件之后，重启机器，selinux 配置才能永久生效重启之后登录机器验证是否修改成功：
getenforce
#显示 Disabled 说明 selinux 已经关闭
</code></pre>
<h2 id="17-关闭交换分区-swap">1.7 关闭交换分区 swap</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">临时关闭 swapoff -a
永久关闭：注释 swap 挂载，给 swap 这行开头加一下注释
vim /etc/fstab
#dev/mapper/centos-swap swap        swap        defaults        0 0
</code></pre>
<h2 id="18-修改内核参数">1.8 修改内核参数</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash"># 加载 br_netfilter 模块
modprobe br_netfilter
# 验证模块是否加载成功：
lsmod |grep br_netfilter
# 修改内核参数
cat &gt; /etc/sysctl.d/k8s.conf &lt;&lt;EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#使刚才修改的内核参数生效
sysctl -p /etc/sysctl.d/k8s.conf
</code></pre>
<p>为什么要执行 modprobe br_netfilter？</p>
<p>如不执行上面步骤则在修改/etc/sysctl.d/k8s.conf 文件后再执行 sysctl -p /etc/sysctl.d/k8s.conf 会出现如下报错：</p>
<p>sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory</p>
<p>sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory</p>
<p>所以解决方法就是提前加载相应模块</p>
<pre><code class="language-bash">modprobe br_netfilter
</code></pre>
<p>net.ipv4.ip_forward 是数据包转发：</p>
<p>出于安全考虑，Linux 系统默认是禁止数据包转发的。所谓转发即当主机拥有多于一块的网卡时，其中一块收到数据包，根据数据包的目的 ip 地址将数据包发往本机另一块网卡，该网卡根据路由表继续发送数据包。这通常是路由器所要实现的功能。</p>
<p>要让 Linux 系统具有路由转发功能，需要配置一个 Linux 的内核参数 net.ipv4.ip_forward。这个参数指定了 Linux 系统当前对路由转发功能的支持情况；其值为 0 时表示禁止进行 IP 转发；如果是 1,则说明 IP 转发功能已经打开。</p>
<h2 id="19-配置阿里云-repo-源">1.9 配置阿里云 repo 源</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">安装 rzsz scp命令
yum install lrzsz openssh-clients yum-utils -y
#配置国内阿里云 docker 的 repo 源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
</code></pre>
<h2 id="110-配置时间同步">1.10 配置时间同步</h2>
<pre><code class="language-bash">#安装 ntpdate 或chrony服务都可以
yum install ntpdate -y
#跟网络源做同步
ntpdate cn.pool.ntp.org
#把时间同步做成计划任务
crontab -e
* */1 * * * /usr/sbin/ntpdate        cn.pool.ntp.org
#重启 crond 服务
service crond restart
</code></pre>
<h2 id="111-安装-iptables">1.11 安装 iptables</h2>
<p>如果用 firewalld 不习惯，可以安装 iptables ，在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">#安装 iptables
yum install iptables-services -y
#禁用 iptables
service iptables stop        &amp;&amp; systemctl disable iptables
#清空防火墙规则
iptables -F
</code></pre>
<h2 id="112-安装环境基础软件包">1.12 安装环境基础软件包</h2>
<p>yum安装</p>
<pre><code class="language-bash">yum install -y device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socatipvsadm conntrack ntpdate telnet rsync
</code></pre>
<h2 id="113-安装-docker环境">1.13 安装 docker环境</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">yum install docker-ce-y
systemctl start docker &amp;&amp; systemctl enable docker.service &amp;&amp; systemctl status docker
</code></pre>
<h2 id="114-配置-docker-镜像加速器">1.14 配置 docker 镜像加速器</h2>
<p>在 k8s-master1、k8s-master2、k8s-node1 上操作：</p>
<pre><code class="language-bash">tee /etc/docker/daemon.json &lt;&lt; 'EOF'
{
"registry-mirrors":["https://rsbud4vc.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com","http://qtid6917.mirror.aliyuncs.com", "https://rncxm540.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF
systemctl daemon-reload &amp;&amp; systemctl restart docker &amp;&amp; systemctl status docker
Active: active (running) since Tue 2022-08-09 23:55:58 CST; 4ms ago
</code></pre>
<p>#修改 docker 文件驱动为 systemd，默认为 cgroupfs，kubelet 默认使用 systemd，两者必须一致才可以。</p>
<h1 id="二k8s环境部署">二、k8s环境部署</h1>
<h2 id="21-搭建-etcd-集群">2.1 搭建 etcd 集群</h2>
<h3 id="211-配置-etcd-工作目录">2.1.1 配置 etcd 工作目录</h3>
<p>#创建配置文件和证书文件存放目录(master1和2同时操作)</p>
<pre><code class="language-bash"># mkdir -p /etc/etcd/ssl
</code></pre>
<h3 id="212-安装签发证书工具-cfssl">2.1.2 安装签发证书工具 cfssl</h3>
<p>在master1操作</p>
<pre><code class="language-bash"> # mkdir /data/work -p
# cd /data/work/
#cfssl-certinfo_linux-amd64cfssljson_linux-amd64cfssl_linux-amd64 上传到/data/work/目录下
# ls
cfssl-certinfo_linux-amd64cfssljson_linux-amd64cfssl_linux-amd64
#把文件变成可执行权限
# chmod +x *
# mv cfssl_linux-amd64 /usr/local/bin/cfssl
# mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
# mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
</code></pre>
<p><img src="https://img-blog.csdnimg.cn/9506b968187642f99f49044f5893af98.png" alt="在这里插入图片描述" loading="lazy"></p>
<h3 id="213-配置-ca-证书">2.1.3 配置 ca 证书</h3>
<p>#生成 ca 证书请求文件</p>
<pre><code class="language-bash"># vim ca-csr.json
# cat ca-csr.json
{
"CN": "kubernetes",
"key": {
      "algo": "rsa",
      "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
],
"ca": {
          "expiry": "87600h"
}
}
# cfssl gencert -initca ca-csr.json| cfssljson -bare ca
2022/08/10 13:37:04 generating a new CA key and certificate from CSR
2022/08/10 13:37:04 generate received request
2022/08/10 13:37:04 received CSR
2022/08/10 13:37:04 generating key: rsa-2048
2022/08/10 13:37:04 encoded CSR
2022/08/10 13:37:04 signed certificate with serial number 277004694657827998333236437935739196848568561695
</code></pre>
<blockquote>
<p>注：</p>
<p>CN：Common Name（公用名称），kube-apiserver从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；对于SSL证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端证书则为证书申请者的姓名。</p>
<p>O：Organization（单位名称），kube-apiserver从证书中提取该字段作为请求用户所属的组</p>
<p>(Group)；对于SSL证书，一般为网站域名；而对于代码签名证书则为申请单位名称；而对于客户端单位证书则为证书申请者所在单位名称。</p>
<p>L 字段：所在城市</p>
<p>S 字段：所在省份</p>
<p>C 字段：只能是国家字母缩写，如中国：CN</p>
</blockquote>
<p>#生成 ca 证书文件</p>
<pre><code class="language-bash"># cat ca-config.json
{
"signing": {
      "default": {
          "expiry": "87600h"
      },
      "profiles": {
          "kubernetes": {
            "usages": [
                  "signing",
                  "key encipherment",
                  "server auth",
                  "client auth"
            ],
            "expiry": "87600h"
          }
      }
}
}
</code></pre>
<h3 id="214-生成-etcd-证书">2.1.4 生成 etcd 证书</h3>
<p>#配置 etcd 证书请求，hosts 的 ip 变成自己 etcd 所在节点的 ip</p>
<pre><code class="language-bash"># cat etcd-csr.json
{
"CN": "etcd",
"hosts": [
    "127.0.0.1",
    "192.168.7.10",
    "192.168.7.11",
    "192.168.7.13",
    "192.168.7.99"
],
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [{
    "C": "CN",
    "ST": "Hubei",
    "L": "Wuhan",
    "O": "k8s",
    "OU": "system"
}]
}
</code></pre>
<p>#上述文件 hosts 字段中 IP 为所有 etcd 节点的集群内部通信 IP，可以预留几个，做扩容用。</p>
<pre><code class="language-bash"># cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson-bare etcd
2022/08/10 13:49:29 generate received request
2022/08/10 13:49:29 received CSR
2022/08/10 13:49:29 generating key: rsa-2048
2022/08/10 13:49:29 encoded CSR
2022/08/10 13:49:29 signed certificate with serial number 580147585393953277974184358784294478670114176750
2022/08/10 13:49:29 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
# 查看一下
# ls etcd*.pem
etcd-key.pemetcd.pem
</code></pre>
<h2 id="215-部署-etcd-集群">2.1.5 部署 etcd 集群</h2>
<p>把 etcd-v3.4.13-linux-amd64.tar.gz 上传到/data/work 目录下</p>
<pre><code class="language-bash"># pwd
/data/work
# tar -xf etcd-v3.4.13-linux-amd64.tar.gz
# cp -ar etcd-v3.4.13-linux-amd64/etcd* /usr/local/bin/
# chmod +x /usr/local/bin/etcd*
# 同样拷贝到master2下
# scp -r etcd-v3.4.13-linux-amd64/etcd* k8s-master2:/usr/local/bin/
# chmod +x /usr/local/bin/etcd*
</code></pre>
<p>#创建配置文件</p>
<pre><code class="language-bash"> k8s-master1
# cat etcd.conf
#
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.7.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.7.10:2379,http://127.0.0.1:2379"
#
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.10:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.7.10:2380,etcd2=https://192.168.7.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<p>#注：</p>
<blockquote>
<p>ETCD_NAME：节点名称，集群中唯一</p>
<p>ETCD_DATA_DIR：数据目录</p>
<p>ETCD_LISTEN_PEER_URLS：集群通信监听地址</p>
<p>ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址</p>
<p>ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址</p>
<p>ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址</p>
<p>ETCD_INITIAL_CLUSTER：集群节点地址</p>
<p>ETCD_INITIAL_CLUSTER_TOKEN：集群 Token</p>
<p>ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new 是新集群，existing 表示加入已有集群</p>
</blockquote>
<p>#创建启动服务文件</p>
<p># vim etcd.service</p>
<pre><code class="language-bash"># cat etcd.service

Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target


Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd \
--cert-file=/etc/etcd/ssl/etcd.pem \
--key-file=/etc/etcd/ssl/etcd-key.pem \
--trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-cert-file=/etc/etcd/ssl/etcd.pem \
--peer-key-file=/etc/etcd/ssl/etcd-key.pem \
--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \
--peer-client-cert-auth \
--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536


WantedBy=multi-user.target
</code></pre>
<p>把etcd的证书都拷贝到一开始创建证书目录下，并同步拷贝到master2</p>
<pre><code class="language-bash"># cp ca*.pem /etc/etcd/ssl/
# cp etcd*.pem /etc/etcd/ssl/
# cp etcd.conf /etc/etcd/
# cp etcd.service /usr/lib/systemd/system/
# for i in k8s-master2 ;do rsync -vaz etcd.conf $i:/etc/etcd/;done
# for i in k8s-master2 ;do rsync -vaz etcd*.pem ca*.pem $i:/etc/etcd/ssl/;done
# for i in k8s-master2;do rsync -vaz etcd.service $i:/usr/lib/systemd/system/;done
</code></pre>
<p>#启动 etcd 集群</p>
<p># mkdir -p /var/lib/etcd/default.etcd</p>
<p># mkdir -p /var/lib/etcd/default.etcd</p>
<p>修改master2的etcd配置文件</p>
<p># vim /etc/etcd/etcd.conf</p>
<pre><code class="language-bash"># cat /etc/etcd/etcd.conf
#
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.7.11:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.7.11:2379,http://127.0.0.1:2379"
#
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.7.11:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.7.11:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.7.10:2380,etcd2=https://192.168.7.11:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
</code></pre>
<p>启动etc服务</p>
<pre><code class="language-bash"># systemctl daemon-reload &amp;&amp; systemctl enable etcd.service &amp;&amp; systemctl start etcd.service
# systemctl daemon-reload &amp;&amp; systemctl enable etcd.service &amp;&amp; systemctl start etcd.service
</code></pre>
<blockquote>
<p>注意：</p>
<p>启动 etcd 的时候，先启动 k8s-master1 的 etcd 服务，会一直卡住在启动的状态，然后接着再启动k8s-master2 的 etcd，这样 k8s-master1 这个节点 etcd 才会正常起来</p>
</blockquote>
<p># systemctl status etcd</p>
<p># systemctl status etcd</p>
<p>启动正常<br>
<img src="https://img-blog.csdnimg.cn/4de4edea557543878416606ee8dcc2a7.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>#查看 etcd 集群</p>
<pre><code class="language-bash">#
ETCDCTL_API=3 &amp;&amp; /usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://192.168.7.10:2379,https://192.168.7.11:2379 endpoint health
+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK   | ERROR |
+---------------------------+--------+-------------+-------+
| https://192.168.7.10:2379 |   true |8.808558ms |       |
| https://192.168.7.11:2379 |   true | 10.902347ms |       |
+---------------------------+--------+-------------+-------+
</code></pre>
<p>2.2 安装kubernetes组件</p>
<p>下载安装包</p>
<p>二进制包所在的 github 地址如下：</p>
<p>https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/</p>
<p>#把 kubernetes1.23-server-linux-amd64.tar.gz 上传到 k8s-master1 上的/data/work 目录下:</p>
<pre><code class="language-bash"> # tar zxvf kubernetes1.23-server-linux-amd64.tar.gz
# cd kubernetes/server/bin/
# cp kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
# rsync -vaz kube-apiserver kube-controller-manager kube- scheduler kubectl k8s-master2:/usr/local/bin/
# scp kubelet kube-proxy k8s-node1:/usr/local/bin/
# cd /data/work/
# mkdir -p /etc/kubernetes/ssl
# mkdir /var/log/kubernetes
</code></pre>
<h2 id="22-部署-apiserver-组件">2.2 部署 apiserver 组件</h2>
<p><mark>启动 TLS Bootstrapping 机制</mark></p>
<blockquote>
<p>Master apiserver 启用 TLS 认证后，每个节点的 kubelet 组件都要使用由 apiserver使用的CA 签发的有效证书才能与 apiserver 通讯，当 Node 节点很多时，这种客户端证书颁发需要大量工作，同样也会增加集群扩展复杂度。</p>
<p>为了简化流程，Kubernetes 引入了 TLS bootstraping 机制来自动颁发客户端证书，kubelet 会以一个低权限用户自动向 apiserver 申请证书，kubelet 的证书由 apiserver 动态签署。</p>
<p>Bootstrap 是很多系统中都存在的程序，比如 Linux 的 bootstrap，bootstrap 一般都是作为预先配置在开启或者系统启动的时候加载，这可以用来生成一个指定环境。Kubernetes的kubelet 在启动时同样可以加载一个这样的配置文件，这个文件的内容类似如下形式：</p>
<p>apiVersion: v1 clusters: null contexts:</p>
<p>- context:</p>
<p>cluster: kubernetes</p>
<p>user: kubelet-bootstrap</p>
<p>name: default</p>
<p>current-context: default</p>
<p>kind: Config preferences: {}</p>
<p>users:</p>
<p>-name: kubelet-bootstrap user: {}</p>
</blockquote>
<h3 id="tls-bootstrapping-具体引导过程">TLS bootstrapping 具体引导过程</h3>
<ol>
<li>TLS 作用</li>
</ol>
<blockquote>
<p>TLS的作用就是对通讯加密，防止中间人窃听；同时如果证书不信任的话根本就无法与 apiserver建立连接，更不用提有没有权限向 apiserver 请求指定内容。</p>
</blockquote>
<p><strong>2.</strong> RBAC 作用</p>
<blockquote>
<p>当 TLS 解决了通讯问题后，那么权限问题就应由 RBAC解决(可以使用其他权限模型，如ABAC)；RBAC 中规定了一个用户或者用户组(subject)具有请求哪些 api 的权限；在配合 TLS 加密的时候，实际上 apiserver读取客户端证书的 CN 字段作为用户名，读取O 字段作为用户组.</p>
<p>以上说明：第一，想要与 apiserver 通讯就必须采用由 apiserver CA 签发的证书，这样才能形成信任关系，建立 TLS 连接；第二，可以通过证书的CN、O字段来提供 RBAC 所需的用户与用户组。</p>
</blockquote>
<p><mark>kubelet 首次启动流程</mark></p>
<blockquote>
<p>TLS bootstrapping 功能是让 kubelet 组件去 apiserver   申请证书，然后用于连接apiserver；那么第一次启动时没有证书如何连接apiserver ?</p>
<p>在 apiserver 配置中指定了一个 token.csv 文件，该文件中是一个预设的用户配置；同时该用户的Token 和 由 apiserver 的 CA 签发的用户被写入了 kubelet 所使用的 bootstrap.kubeconfig 配置文件中；这样在首次请求时，kubelet 使用 bootstrap.kubeconfig 中被 apiserver CA 签发证书时信任的用户来与apiserver建立TLS通讯，使用 bootstrap.kubeconfig 中的用户 Token 来向apiserver声明自己的RBAC 授权身份.</p>
<p>token.csv 格式:</p>
<p>3940fd7fbb391d1b4d861ad17a1f0613,kubelet-bootstrap,10001,"system:kubelet- bootstrap"</p>
<p>首次启动时，可能与遇到 kubelet 报 401 无权访问 apiserver 的错误；这是因为在默认情况下，kubelet 通过 bootstrap.kubeconfig 中的预设用户 Token 声明了自己的身份，然后创建 CSR请求；但是不要忘记这个用户在我们不处理的情况下他没任何权限的，包括创建 CSR 请求；所以需要创建一个ClusterRoleBinding，将预设用户 kubelet-bootstrap 与内置的 ClusterRole system:node-bootstrapper 绑定到一起，使其能够发起 CSR 请求。稍后安装 kubelet 的时候演示。</p>
</blockquote>
<p>#创建 token.csv 文件</p>
<pre><code class="language-bash"># cd /data/work/
# cat &gt; token.csv &lt;&lt; EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

# 格式：token，用户名，UID，用户组
</code></pre>
<p>#创建 csr 请求文件，替换为自己机器的 IP</p>
<p># vim kube-apiserver-csr.json</p>
<blockquote>
<p>#注： 如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表。 由于该证书后续被 kubernetes master 集群使用，需要将 master 节点的 IP 都填上，同时还需要填写service 网络的首个 IP。(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个 IP，如 10.255.0.1)</p>
</blockquote>
<pre><code class="language-bash"># cat kube-apiserver-csr.json
{
"CN": "kubernetes",
"hosts": [
    "127.0.0.1",
    "192.168.7.10",
    "192.168.7.11",
    "192.168.7.12",
    "192.168.7.13",
    "192.168.7.99",
    "10.255.0.1",
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local"
],
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
]
}
</code></pre>
<p>#生成证书</p>
<pre><code class="language-bash">#
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver
2022/08/10 16:34:32 generate received request
2022/08/10 16:34:32 received CSR
2022/08/10 16:34:32 generating key: rsa-2048
2022/08/10 16:34:32 encoded CSR
2022/08/10 16:34:32 signed certificate with serial number 477144676195233738114152251729193994277731824828
2022/08/10 16:34:32 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
</code></pre>
<p>创建apiserver的配置文件</p>
<pre><code class="language-bash">#创建 api-server 的配置文件，替换成自己的 ip
# vim kube-apiserver.conf
# cat kube-apiserver.conf
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
--anonymous-auth=false \
--bind-address=192.168.7.10 \
--secure-port=6443 \
--advertise-address=192.168.7.10 \
--insecure-port=0 \
--authorization-mode=Node,RBAC \
--runtime-config=api/all=true \
--enable-bootstrap-token-auth \
--service-cluster-ip-range=10.255.0.0/16 \
--token-auth-file=/etc/kubernetes/token.csv \
--service-node-port-range=30000-50000 \
--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem\
--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \
--client-ca-file=/etc/kubernetes/ssl/ca.pem \
--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \
--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \
--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \
--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem\
--service-account-issuer=https://kubernetes.default.svc.cluster.local \
--etcd-cafile=/etc/etcd/ssl/ca.pem \
--etcd-certfile=/etc/etcd/ssl/etcd.pem \
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \
--etcd-servers=https://192.168.7.10:2379,https://192.168.7.11:2379 \
--enable-swagger-ui=true \
--allow-privileged=true \
--apiserver-count=3 \
--audit-log-maxage=30 \
--audit-log-maxbackup=3 \
--audit-log-maxsize=100 \
--audit-log-path=/var/log/kube-apiserver-audit.log \
--event-ttl=1h \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=4"


# 注解
--logtostderr：启用日志
--v：日志等级
--log-dir：日志目录
--etcd-servers：etcd 集群地址
--bind-address：监听地址
--secure-port：https 安全端口
--advertise-address：集群通告地址
--allow-privileged：启用授权
--service-cluster-ip-range：Service 虚拟 IP 地址段
--enable-admission-plugins：准入控制模块
--authorization-mode：认证授权，启用 RBAC 授权和节点自管理
--enable-bootstrap-token-auth：启用 TLS bootstrap 机制
--token-auth-file：bootstrap token 文件
--service-node-port-range：Service nodeport 类型默认分配端口范围
--kubelet-client-xxx：apiserver 访问 kubelet 客户端证书
--tls-xxx-file：apiserver https 证书
--etcd-xxxfile：连接 Etcd 集群证书
–-audit-log-xxx：审计日志
</code></pre>
<p>#创建服务启动文件</p>
<pre><code class="language-bash"># vim kube-apiserver.service

Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service


EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver $KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536


WantedBy=multi-user.target

# 拷贝证书文件到相应的目录，同时也拷贝到master2节点
# cp ca*.pem /etc/kubernetes/ssl
cp kube-apiserver*.pem /etc/kubernetes/ssl/
cp token.csv /etc/kubernetes/
cp kube-apiserver.conf /etc/kubernetes/
cp kube-apiserver.service /usr/lib/systemd/system/

# rsync -vaz token.csv k8s-master2:/etc/kubernetes/
# rsync -vaz kube-apiserver*.pem k8s-master2:/etc/kubernetes/ssl/
# rsync -vaz ca*.pem k8s-master2:/etc/kubernetes/ssl/
rsync -vaz kube-apiserver.conf k8s-master2:/etc/kubernetes/
rsync -vaz kube-apiserver.service k8s-master2:/usr/lib/systemd/system/
注意！！！！k8s-master2 配置文件 kube-apiserver.conf 的 IP 地址修改为实际的本机 IP

启动kube-apiserver
# systemctl daemon-reload &amp;&amp; systemctl enable kube-apiserver &amp;&amp; systemctl start kube-apiserver
# systemctl daemon-reload &amp;&amp; systemctl enable kube-apiserver &amp;&amp; systemctl start kube-apiserver

# systemctl status kube-apiserver
Active: active (running) since Wed 2022-08-10 16:58:26 CST; 15s ago
#
curl --insecure https://192.168.7.10:6443/
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "Unauthorized",
"reason": "Unauthorized",
"code": 401
}
上面看到 401，这个是正常的的状态，还没认证
</code></pre>
<h2 id="23-部署-kubectl-组件">2.3 部署 kubectl 组件</h2>
<h3 id="231-kubectl组件介绍">2.3.1 kubectl组件介绍</h3>
<blockquote>
<p>Kubectl 是客户端工具，操作k8s 资源的，如增删改查等。</p>
<p>Kubectl 操作资源的时候，怎么知道连接到哪个集群，需要一个文件/etc/kubernetes/admin.conf，kubectl 会根据这个文件的配置，去访问 k8s 资源。</p>
<p>/etc/kubernetes/admin.conf 文件记录了访问的 k8s 集群，和用到的证书。可以设置一个环境变量 KUBECONFIG</p>
<pre><code class="language-bash"># export KUBECONFIG=/etc/kubernetes/admin.conf
</code></pre>
<p>这样在操作 kubectl，就会自动加载 KUBECONFIG 来操作要管理哪个集群的 k8s 资源了也可以按照下面方法，这个是在 kubeadm 初始化 k8s 的时候会提示我们要用的一个方法</p>
<pre><code class="language-bash"># cp /etc/kubernetes/admin.conf /root/.kube/config
</code></pre>
<p>这样我们在执行 kubectl，就会加载/root/.kube/config 文件，去操作 k8s 资源了</p>
<p>如果设置了 KUBECONFIG，那就会先找到 KUBECONFIG 去操作 k8s，如果没有 KUBECONFIG变量，那就会使用/root/.kube/config 文件决定管理哪个 k8s 集群的资源注意：admin.conf 还没创建，下面步骤创建</p>
</blockquote>
<h3 id="232-创建-csr-请求文件">2.3.2 创建 csr 请求文件</h3>
<pre><code class="language-bash"># vim admin-csr.json
{
"CN": "admin",
"hosts": [],
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "system:masters",            
      "OU": "system"
    }
]
}
</code></pre>
<blockquote>
<p>说明： 后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权； kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Role cluster-admin绑定，该 Role 授予了调用 kube-apiserver 的所有 API 的权限； O 指定该证书的 Group 为 system:masters，kubelet使用该证书访问 kube-apiserver时，由于证书被CA 签名，所以认证通过，同时由于证书用户组为经过预授权的system:masters，所以被授予访问所有API 的权限；</p>
<p>注： 这个 admin 证书，是将来生成管理员用的 kube config 配置文件用的，现在我们一般建议使用 RBAC 来对 kubernetes 进行角色权限控制， kubernetes 将证书中的 CN 字段 作为 User，</p>
<p>O 字段作为Group； "O": "system:masters",必须是 system:masters，否则后面 kubectl create</p>
<p>clusterrolebinding 报错。</p>
<p>证书 O 配置为 system:masters 在集群内部 cluster-admin 的 clusterrolebinding 将system:masters 组和cluster-admin clusterrole 绑定在一起</p>
</blockquote>
<h3 id="232-生成客户端的证书">2.3.2 生成客户端的证书</h3>
<pre><code class="language-bash">#cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin
2022/08/10 17:14:04 generate received request
2022/08/10 17:14:04 received CSR
2022/08/10 17:14:04 generating key: rsa-2048
2022/08/10 17:14:04 encoded CSR
2022/08/10 17:14:04 signed certificate with serial number 116808315728556635428311377758058019811028156802
2022/08/10 17:14:04 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

# cp admin*.pem /etc/kubernetes/ssl/
</code></pre>
<h3 id="233-配置安全上下文">2.3.3 配置安全上下文</h3>
<p>#创建 kubeconfig 配置文件，比较重要</p>
<p>kubeconfig 为kubectl的配置文件，包含访问apiserver的所有信息，如apiserver地址、 CA 证书和自身使用的证书（这里如果报错找不到 kubeconfig 路径，请手动复制到相应路径下，没有则忽略）</p>
<p>1.设置集群参数</p>
<pre><code class="language-bash"> k8s-master1 #
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.7.10:6443 --kubeconfig=kube.config
Cluster "kubernetes" set
</code></pre>
<p>#查看 kube.config 内容</p>
<pre><code class="language-bash">cat kube.config
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR0akNDQXA2Z0F3SUJBZ0lVTUlWUjhrM0haSHhmb0tPZmNrSE9hU05nY0I4d0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RGpBTUJnTlZCQWdUQlVoMVltVnBNUTR3REFZRFZRUUhFd1ZYZFdoaApiakVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WnplWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SGhjTk1qSXdPREV3TURVek1qQXdXaGNOTXpJd09EQTNNRFV6TWpBd1dqQmhNUXN3Q1FZRFZRUUcKRXdKRFRqRU9NQXdHQTFVRUNCTUZTSFZpWldreERqQU1CZ05WQkFjVEJWZDFhR0Z1TVF3d0NnWURWUVFLRXdOcgpPSE14RHpBTkJnTlZCQXNUQm5ONWMzUmxiVEVUTUJFR0ExVUVBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKCktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU1PSlFwMDBtZ1AzZkUvMzQwN0ZqTUhWYWVGd2FGQWwKeXJTU1lqT1h0emlkdC9DSTRiVTU4TERQbEFqM1FrYUJQOEZnc3podnJxY0s0aGEydUNDTG93dWFzYkJVR2RsNgp0ejRyY2w3TWFuNzFEWDVXaWVKOWxQVWxlbWRKblFwc3NlR3puRmZtU1NSMHFqTVNtbXlMSzRvci81OW1NVkxvCk1sL0JNd2pMSzk3Q1NQdHhDL2g5ODBydDhWbENsNWxGWnBYbmJJWW1FbkhaakxGa2xnRzVLTktvMm54MGhUNk4KTzd4M0FtYTBzb0VjUVgxWXBtYy82ZHBFcXRnZXE1Ry9GajJkanZud2o4R09aSEJhVmNLNFBWWG9pYi81U3JmTgppamR1S2dBeis3Q1MxRGFYcytocndEMjNBRm1FelJNZVF3aG0yK0ZWVjFkRXBVRDlkMFg0UWE4Q0F3RUFBYU5tCk1HUXdEZ1lEVlIwUEFRSC9CQVFEQWdFR01CSUdBMVVkRXdFQi93UUlNQVlCQWY4Q0FRSXdIUVlEVlIwT0JCWUUKRkNIN3dmRUVQWG1EY2RxT25wcUdad2JjSGptVU1COEdBMVVkSXdRWU1CYUFGQ0g3d2ZFRVBYbURjZHFPbnBxRwpad2JjSGptVU1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2ZLWUhCZ2JwSkZMMkxKT2FOeGNCWDEwYlMvbkJECnI5VTJSRGdtRkNSVXQ0LzRMT25BaU9QTUpvZ3UxQndWZ0lLc2UrS1NSVFVnMjc4S1lBc002QzBkeW9mM2dVN0sKdWpMYWV4NDZQNXJvTXJINEY3N2xtL2FqaUdSS1lNRTQvNjV0clF3VDJtenBQYVhUZFZGb3grRUVsbTZnNU5wSApKYWk1L0QzSzNYRUdDL1lMYmdaUFpUYVRCSDN6bUlkV2hNQXo1N1U0U1FVak0vRTdIN09rcDRiTEQzM2pVRitSCndDL1dTWTVvenM2SkVqZ1V3SGpjR2o4MFpTdnZLcU1OTXplUjdMQmVVUXRQY20wU1ppQlp1VE1HL0l4aFFLN0sKeU1DVzhieXlUM3EzNFUrN1pYNGE4cDZRd04xVURSb3gxZ3lyajJaYUo2aHFWeTR0c1VLMTl2RTgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://192.168.7.10:6443
name: kubernetes
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null
</code></pre>
<p>2.设置客户端认证参数</p>
<pre><code class="language-bash">kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=kube.config
User "admin" set
</code></pre>
<p>3.设置上下文参数</p>
<pre><code class="language-bash">#
kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=kube.config
Context "kubernetes" created
</code></pre>
<p>4.设置当前上下文</p>
<pre><code class="language-bash"> k8s-master1
# kubectl config use-context kubernetes --kubeconfig=kube.config
Switched to context "kubernetes".
k8s-master1
#mkdir ~/.kube -p
k8s-master1
# cp kube.config ~/.kube/config
k8s-master1
# cp kube.config /etc/kubernetes/admin.conf
</code></pre>
<p>5.授权 kubernetes 证书访问kubelet api 权限</p>
<pre><code class="language-bash"># kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes
clusterrolebinding.rbac.authorization.k8s.io/kube-apiserver:kubelet-apis created
</code></pre>
<h1 id="查看集群组件状态">查看集群组件状态</h1>
<p># kubectl cluster-info</p>
<pre><code class="language-bash">#kubectl cluster-info
Kubernetes control plane is running at https://192.168.7.10:6443
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
执行下面命令查看集群信息
k8s-master1
# kubectl get componentstatuses
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME               STATUS      MESSAGE            ERROR
controller-manager   Unhealthy   Get "https://127.0.0.1:10257/healthz": dial tcp 127.0.0.1:10257: connect: connection refused   
scheduler            Unhealthy   Get "https://127.0.0.1:10259/healthz": dial tcp 127.0.0.1:10259: connect: connection refused   
etcd-0               Healthy   {"health":"true"}                                                                        
etcd-1               Healthy   {"health":"true"}
</code></pre>
<pre><code class="language-bash"> k8s-master1
#kubectl get all --all-namespaces
NAMESPACE   NAME               TYPE      CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default   service/kubernetes   ClusterIP   10.255.0.1   &lt;none&gt;      443/TCP   36m
</code></pre>
<p><img src="https://img-blog.csdnimg.cn/688b7f72f30b4801b44c3bc0aa765152.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>同步 kubectl 文件到其他节点</p>
<pre><code class="language-bash"># mkdir /root/.kube/
# rsync -vaz /root/.kube/config k8s-master2:/root/.kube/
</code></pre>
<h1 id="配置-kubectl-子命令补全依次执行下面命令即可">配置 kubectl 子命令补全，依次执行下面命令即可</h1>
<pre><code class="language-bash"># yum install -y bash-completion
source /usr/share/bash-completion/bash_completion
source &lt;(kubectl completion bash)
kubectl completion bash &gt; ~/.kube/completion.bash.inc
source '/root/.kube/completion.bash.inc'
source $HOME/.bash_profile
</code></pre>
<p>Kubectl 官方备忘单：（自动补全）</p>
<p>https://kubernetes.io/zh/docs/reference/kubectl/cheatsheet/</p>
<h2 id="24-部署-kube-controller-manager-组件">2.4 部署 kube-controller-manager 组件</h2>
<h3 id="241-创建-kube-controller-manager-csr-请求文件">2.4.1 创建 kube-controller-manager csr 请求文件</h3>
<pre><code class="language-bash"> k8s-master1
# cat kube-controller-manager-csr.json
{
    "CN": "system:kube-controller-manager",
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "hosts": [
      "127.0.0.1",
      "192.168.7.10",
      "192.168.7.11",
      "192.168.7.13",
      "192.168.7.99"
    ],
    "names": [
      {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "system:kube-controller-manager",
      "OU": "system"
      }
    ]
}

# 注： hosts 列表包含所有 kube-controller-manager 节点 IP；
CN 为 system:kube- controller-manager
O 为 system:kube-controller-manager，
kubernetes 内置的 ClusterRoleBindings system:kube-controller-manager 赋予 kube-controller-manager 工作所需的权限
</code></pre>
<h3 id="242-生成-kube-controller-manager证书">2.4.2 生成 kube-controller-manager证书</h3>
<pre><code class="language-bash">#
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

2022/08/10 17:46:58 generate received request
2022/08/10 17:46:58 received CSR
2022/08/10 17:46:58 generating key: rsa-2048
2022/08/10 17:46:58 encoded CSR
2022/08/10 17:46:58 signed certificate with serial number 402122087182562792164197882344665077108635062015
2022/08/10 17:46:58 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
</code></pre>
<h3 id="243-创建-kube-controller-manager-的-kubeconfig">2.4.3 创建 kube-controller-manager 的 kubeconfig</h3>
<p>1.设置集群参数</p>
<pre><code class="language-bash">#
kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.7.10:6443 --kubeconfig=kube-controller-manager.kubeconfig
Cluster "kubernetes" set
</code></pre>
<p>2.设置客户端认证参数</p>
<pre><code class="language-bash"># kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=kube-controller-manager.kubeconfig
User "system:kube-controller-manager" set.
</code></pre>
<p>3.设置上下文参数</p>
<pre><code class="language-bash">#
kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
Context "system:kube-controller-manager" created.
</code></pre>
<p>4.设置当前上下文</p>
<pre><code class="language-bash"># kubectl config use-context system:kube-controller-manager --kubeconfig=kube-controller-manager.kubeconfig
Switched to context "system:kube-controller-manager"
</code></pre>
<h3 id="244-创建kube-controller-manager配置文件">2.4.4 创建kube-controller-manager配置文件</h3>
<pre><code class="language-bash"># cat kube-controller-manager.conf
KUBE_CONTROLLER_MANAGER_OPTS="--port=0 \
--secure-port=10257 \
--bind-address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \
--service-cluster-ip-range=10.255.0.0/16 \
--cluster-name=kubernetes \
--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \
--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \
--allocate-node-cidrs=true \
--cluster-cidr=10.0.0.0/16 \
--experimental-cluster-signing-duration=87600h \
--root-ca-file=/etc/kubernetes/ssl/ca.pem \
--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \
--leader-elect=true \
--feature-gates=RotateKubeletServerCertificate=true \
--controllers=*,bootstrapsigner,tokencleaner \
--horizontal-pod-autoscaler-sync-period=10s \
--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \
--tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \
--use-service-account-credentials=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
</code></pre>
<h3 id="245-创建kube-controller-manager启动文件">2.4.5 创建kube-controller-manager启动文件</h3>
<pre><code class="language-bash"># catkube-controller-manager.service

Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes

EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager $KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5

WantedBy=multi-user.target
</code></pre>
<h3 id="246-启动-kube-controller-manager服务">2.4.6 启动 kube-controller-manager服务</h3>
<pre><code class="language-bash"># cp kube-controller-manager*.pem /etc/kubernetes/ssl/
# cp kube-controller-manager.kubeconfig /etc/kubernetes/
# cp kube-controller-manager.conf /etc/kubernetes/
# cp kube-controller-manager.service /usr/lib/systemd/system/
# rsync -vaz kube-controller-manager.pem k8s-master2:/etc/kubernetes/ssl/
# rsync -vaz kube-controller-manager.kubeconfig kube-controller-manager.conf k8s-master2:/etc/kubernetes/
# rsync -vaz kube-controller-manager.service k8s-master2:/usr/lib/systemd/system/
</code></pre>
<p>启动</p>
<pre><code class="language-bash"># systemctl daemon-reload&amp;&amp;systemctl enable kube-controller-manager &amp;&amp; systemctl start kube-controller-manager &amp;&amp; systemctl status kube-controller-manager
Active: active (running) since Wed 2022-08-10 18:18:05 CST; 51ms ago
</code></pre>
<h2 id="25--部署-kube-scheduler-组件">2.5部署 kube-scheduler 组件</h2>
<h3 id="251-创建kube-scheduler的csr-请求">2.5.1 创建kube-scheduler的csr 请求</h3>
<pre><code class="language-bash"># cat kube-scheduler-csr.json
{
    "CN": "system:kube-scheduler",
    "hosts": [
      "127.0.0.1",
      "192.168.7.10",
      "192.168.7.11",
      "192.168.7.13",
      "192.168.7.99"
    ],
    "key": {
      "algo": "rsa",
      "size": 2048
    },
    "names": [
      {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "system:kube-scheduler",
      "OU": "system"
      }
    ]
}

注： hosts 列表包含所有 kube-scheduler 节点 IP； CN 为 system:kube-scheduler、O 为 system:kube-scheduler，kubernetes 内置的 ClusterRoleBindings system:kube-scheduler 将赋予kube-scheduler 工作所需的权限。
</code></pre>
<h3 id="252-生成kube-scheduler证书">2.5.2 生成kube-scheduler证书</h3>
<pre><code class="language-bash"># cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler
2022/08/10 20:47:45 generate received request
2022/08/10 20:47:45 received CSR
2022/08/10 20:47:45 generating key: rsa-2048
2022/08/10 20:47:46 encoded CSR
2022/08/10 20:47:46 signed certificate with serial number 579944385128158770562264376350906952867199765265
2022/08/10 20:47:46 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").

</code></pre>
<h3 id="253-创建-kube-scheduler-的-kubeconfig文件">2.5.3 创建 kube-scheduler 的 kubeconfig文件</h3>
<p>1.设置集群参数</p>
<pre><code class="language-bash"> kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.7.10:6443 --kubeconfig=kube-scheduler.kubeconfig
Cluster "kubernetes" set
</code></pre>
<p>2.设置客户端认证参数</p>
<pre><code class="language-bash"># kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=kube-scheduler.kubeconfig
User "system:kube-scheduler" set.
</code></pre>
<p>3.设置上下文参数</p>
<pre><code class="language-bash"># kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
Context "system:kube-scheduler" created
</code></pre>
<p>4.设置当前上下文</p>
<pre><code class="language-bash"># kubectl config use-context system:kube-scheduler --kubeconfig=kube-scheduler.kubeconfig
Switched to context "system:kube-scheduler"
</code></pre>
<h3 id="254-创建配置文件-kube-scheduler的配置文件">2.5.4 创建配置文件 kube-scheduler的配置文件</h3>
<pre><code class="language-bash">cat kube-scheduler.conf
KUBE_SCHEDULER_OPTS="--address=127.0.0.1 \
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \
--leader-elect=true \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2"
</code></pre>
<h3 id="255-创建kube-scheduler的服务启动文件">2.5.5 创建kube-scheduler的服务启动文件</h3>
<pre><code class="language-bash"># cat kube-scheduler.service

Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes


EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler $KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5


WantedBy=multi-user.target
</code></pre>
<h3 id="256-拷贝文件到master2节点并启动服务">2.5.6 拷贝文件到master2节点并启动服务</h3>
<pre><code class="language-bash"># cp kube-scheduler*.pem /etc/kubernetes/ssl/
# cp kube-scheduler.kubeconfig /etc/kubernetes/
# cp kube-scheduler.conf /etc/kubernetes/
# cp kube-scheduler.service /usr/lib/systemd/system/
# rsync -vaz kube-scheduler*.pem k8s-master2:/etc/kubernetes/ssl/
# rsync -vaz kube-scheduler.kubeconfig kube-scheduler.conf k8s-master2:/etc/kubernetes/
# rsync -vaz kube-scheduler.service k8s-master2:/usr/lib/systemd/system/
# systemctl daemon-reload &amp;&amp;systemctl enable kube-scheduler &amp;&amp; systemctl start kube-scheduler &amp;&amp; systemctl status kube-scheduler
Active: active (running) since Wed 2022-08-10 21:10:28 CST; 52ms ago
# systemctl daemon-reload &amp;&amp; systemctl enable kube-scheduler &amp;&amp; systemctl start kube-scheduler &amp;&amp; systemctl status kube-scheduler
</code></pre>
<h2 id="26-部署-kubelet-组件">2.6. 部署 kubelet 组件</h2>
<p>kubelet： 每个 Node 节点上的 kubelet 定期就会调用 API Server 的 REST 接口报告自身状态， API Server 接收这些信息后，将节点状态信息更新到 etcd 中。kubelet 也通过 API Server 监听 Pod信息，从而对 Node 机器上的 POD 进行管理，如创建、删除、更新 Pod</p>
<p>先导入离线镜像压缩包</p>
<p>#把 pause-cordns.tar.gz 上传到 k8s-node1 节点，手动解压</p>
<pre><code class="language-bash"># docker load -i pause-cordns.tar.gz
</code></pre>
<p>以下操作在 k8s-master1 上操作</p>
<h3 id="261-创建-kubelet-bootstrapkubeconfig">2.6.1 创建 kubelet-bootstrap.kubeconfig</h3>
<pre><code class="language-bash"># cd /data/work/
# BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
# rm -r kubelet-bootstrap.kubeconfig
# kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.7.10:6443 --kubeconfig=kubelet-bootstrap.kubeconfig
Cluster "kubernetes" set.
# kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig
# kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig
# kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig
# kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap
</code></pre>
<h3 id="262-创建配置文件-kubeletjson">2.6.2 创建配置文件 kubelet.json</h3>
<p>"cgroupDriver": "systemd"要和 docker 的驱动一致。</p>
<p>address 替换为自己 k8s-node1 的 IP 地址。</p>
<pre><code class="language-bash"># cat kubelet.json
{
"kind": "KubeletConfiguration",
"apiVersion": "kubelet.config.k8s.io/v1beta1",
"authentication": {
    "x509": {
      "clientCAFile": "/etc/kubernetes/ssl/ca.pem"
    },
    "webhook": {
      "enabled": true,
      "cacheTTL": "2m0s"
    },
    "anonymous": {
      "enabled": false
    }
},
"authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
},
"address": "192.168.7.10",
"port": 10250,
"readOnlyPort": 10255,
"cgroupDriver": "systemd",
"hairpinMode": "promiscuous-bridge",
"serializeImagePulls": false,
"featureGates": {
    "RotateKubeletServerCertificate": true
},
"clusterDomain": "cluster.local.",
"clusterDNS": ["10.255.0.2"]
}
</code></pre>
<h3 id="263-创建kubelet服务启动文件">2.6.3 创建kubelet服务启动文件</h3>
<pre><code class="language-bash"># vim kubelet.service

Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service

WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \
--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \
--cert-dir=/etc/kubernetes/ssl \
--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \
--config=/etc/kubernetes/kubelet.json \
--network-plugin=cni \
--pod-infra-container-image=k8s.gcr.io/pause:3.2 \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5


WantedBy=multi-user.target

#注： –hostname-override：显示名称，集群中唯一

–network-plugin：启用 CNI
–kubeconfig：空路径，会自动生成，后面用于连接 apiserver
–bootstrap-kubeconfig：首次启动向 apiserver 申请证书
–config：配置参数文件
–cert-dir：kubelet 证书生成目录
–pod-infra-container-image：管理 Pod 网络容器的镜像
#注：kubelete.json 配置文件 address 改为各个节点的 ip 地址，在各个 work 节点上启动服务
</code></pre>
<h3 id="264-拷贝kubelet的证书及配置文件到node节点">2.6.4 拷贝kubelet的证书及配置文件到node节点</h3>
<pre><code class="language-bash"># mkdir /etc/kubernetes/ssl -p
# scp kubelet-bootstrap.kubeconfig kubelet.json k8s-node1:/etc/kubernetes/
# scp ca.pem k8s-node1:/etc/kubernetes/ssl/
# scpkubelet.service k8s-node1:/usr/lib/systemd/system/
</code></pre>
<h3 id="265-在node节点上启动-kubelet-服务">2.6.5 在node节点上启动 kubelet 服务</h3>
<pre><code class="language-bash"># mkdir /var/lib/kubelet
# mkdir /var/log/kubernetes
# systemctl daemon-reload &amp;&amp; systemctl enable kubelet &amp;&amp;systemctl start kubelet &amp;&amp; systemctl status kubelet
</code></pre>
<p><img src="https://img-blog.csdnimg.cn/d7be74adf75b42888a0b6729d0bf5a69.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>确认 kubelet 服务启动成功后，接着到 k8s-master1 节点上 Approve 一下 bootstrap 请求。</p>
<p>执行如下命令可以看到一个 worker 节点发送了一个CSR 请求：</p>
<pre><code class="language-bash"># kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR         REQUESTEDDURATION   CONDITION
node-csr-AxkX7huDjqOMnljZXyLju5SOhIjtcEkTEhM3AH-K_Vs   95s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   &lt;none&gt;            Pending
</code></pre>
<h3 id="266-在master节点审批node节点的请求">2.6.6 在master节点审批node节点的请求</h3>
<pre><code class="language-bash"># kubectl certificate approve node-csr-AxkX7huDjqOMnljZXyLju5SOhIjtcEkTEhM3AH-K_Vs
certificatesigningrequest.certificates.k8s.io/node-csr-AxkX7huDjqOMnljZXyLju5SOhIjtcEkTEhM3AH-K_Vs approved

# 再次查看申请
k8s-master1
#kubectl get csr
NAME                                                   AGE   SIGNERNAME                                    REQUESTOR         REQUESTEDDURATION   CONDITION
node-csr-AxkX7huDjqOMnljZXyLju5SOhIjtcEkTEhM3AH-K_Vs   5m51s   kubernetes.io/kube-apiserver-client-kubelet   kubelet-bootstrap   &lt;none&gt;            Approved,Issued
</code></pre>
<p><img src="https://img-blog.csdnimg.cn/3708acbe59264a3cb658f5df31f87956.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>在master上查看一下node节点是否已经正常加入进来了</p>
<pre><code class="language-bash"> k8s-master1
# kubectl get node
NAME      STATUS   ROLES    AGE   VERSION
k8s-node1   NotReady   &lt;none&gt;   8s    v1.23.1
</code></pre>
<p>#注意：STATUS 是 NotReady 表示还没有安装网络插件</p>
<h2 id="27-部署-kube-proxy-组件">2.7 部署 kube-proxy 组件</h2>
<h3 id="271-创建kube-proxy的-csr-请求">2.7.1 创建kube-proxy的 csr 请求</h3>
<p>#</p>
<pre><code class="language-bash"># cat kube-proxy-csr.json
{
"CN": "system:kube-proxy",
"key": {
    "algo": "rsa",
    "size": 2048
},
"names": [
    {
      "C": "CN",
      "ST": "Hubei",
      "L": "Wuhan",
      "O": "k8s",
      "OU": "system"
    }
]
}
</code></pre>
<h3 id="272-生成证书">2.7.2 生成证书</h3>
<pre><code class="language-bash"># cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy
2022/08/10 21:45:19 generate received request
2022/08/10 21:45:19 received CSR
2022/08/10 21:45:19 generating key: rsa-2048
2022/08/10 21:45:19 encoded CSR
2022/08/10 21:45:19 signed certificate with serial number 418084057922153637657451382122276535476972007518
2022/08/10 21:45:19 This certificate lacks a "hosts" field. This makes it unsuitable for
websites. For more information see the Baseline Requirements for the Issuance and Management
of Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);
specifically, section 10.2.3 ("Information Requirements").
</code></pre>
<h3 id="273-创建-kubeconfig-文件">2.7.3 创建 kubeconfig 文件</h3>
<pre><code class="language-bash"># kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://192.168.7.10:6443 --kubeconfig=kube-proxy.kubeconfig
# kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig
# kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
# kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig
</code></pre>
<h3 id="274-创建-kube-proxy-配置文件">2.7.4 创建 kube-proxy 配置文件</h3>
<pre><code class="language-bash"># vim kube-proxy.yaml
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 192.168.7.13
clientConnection:
kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 192.168.7.0/24
healthzBindAddress: 192.168.7.13:10256
kind: KubeProxyConfiguration
metricsBindAddress: 192.168.7.13:10249
mode: "ipvs"
</code></pre>
<h3 id="275-创建kube-proxy服务启动文件">2.7.5 创建kube-proxy服务启动文件</h3>
<pre><code class="language-bash">#
# cat kube-proxy.service

Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target


WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yaml \
--alsologtostderr=true \
--logtostderr=false \
--log-dir=/var/log/kubernetes \
--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536


WantedBy=multi-user.target
</code></pre>
<h3 id="276-拷贝kube-proxy文件到node节点上">2.7.6 拷贝kube-proxy文件到node节点上</h3>
<pre><code class="language-bash"># scp kube-proxy.kubeconfig kube-proxy.yaml k8s-node1:/etc/kubernetes/
#scpkube-proxy.service k8s-node1:/usr/lib/systemd/system/
</code></pre>
<h3 id="277-启动kube-proxy服务">2.7.7 启动kube-proxy服务</h3>
<pre><code class="language-bash"># mkdir -p /var/lib/kube-proxy
# systemctl daemon-reload &amp;&amp; systemctl enable kube-proxy &amp;&amp; systemctl start kube-proxy &amp;&amp; systemctl status kube-proxy
Active: active (running) since Wed 2022-08-10 21:53:44 CST; 36ms ago
</code></pre>
<h2 id="28-部署-calico-组件">2.8 部署 calico 组件</h2>
<h1 id="解压离线镜像压缩包">解压离线镜像压缩包</h1>
<p>#把 cni.tar.gz 和node.tar.gz 上传到 k8s-node1 节点，手动解压</p>
<pre><code class="language-bash"># docker load -i calico.tar.gz
</code></pre>
<h1 id="把-calicoyaml-文件上传到-k8s-master1-上的的datawork-目录">把 calico.yaml 文件上传到 k8s-master1 上的的/data/work 目录</h1>
<pre><code class="language-bash"># kubectl apply -f calico.yaml
# kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
k8s-node1   Ready    &lt;none&gt;   30m   v1.23.1

k8s-master1
# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-677cd97c8d-jmtsh   1/1   Running   0          40s
calico-node-vc8gz                        1/1   Running   0          40s
</code></pre>
<h2 id="29-部署-coredns-组件">2.9 部署 coredns 组件</h2>
<p>创建coredns资源</p>
<pre><code class="language-bash"># kubectl apply -f coredns.yaml
serviceaccount/coredns created
clusterrole.rbac.authorization.k8s.io/system:coredns created
clusterrolebinding.rbac.authorization.k8s.io/system:coredns created
configmap/coredns created
deployment.apps/coredns created
service/kube-dns created
# 再次查看pod
k8s-master1
# kubectl get pod -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-677cd97c8d-jmtsh   1/1   Running   0          3m44s
calico-node-vc8gz                        1/1   Running   0          3m44s
coredns-86d879486-9sgwk                  1/1   Running   0          43s
</code></pre>
<h2 id="210-查看集群状态">2.10 查看集群状态</h2>
<p>环境部署完成，再次查看集群节点状态</p>
<pre><code class="language-bash"># kubectl get nodes
NAME      STATUS   ROLES    AGE   VERSION
k8s-node1   Ready    &lt;none&gt;   41m   v1.23.1
</code></pre>
<h1 id="三集群组件功能验证测试">三、集群组件功能验证测试</h1>
<h2 id="31-对系统用户-kubernetes-做授权">3.1 对系统用户 kubernetes 做授权</h2>
<p>使用clusterrolebinding进行授权</p>
<pre><code class="language-bash"># kubectl create clusterrolebinding kubernetes-kubectl --clusterrole=cluster-admin --user=kubernetes
</code></pre>
<h2 id="32-测试-k8s-集群部署-tomcat-服务">3.2 测试 k8s 集群部署 tomcat 服务</h2>
<p>#把 tomcat.tar.gz 和 busybox-1-28.tar.gz 上传到 k8s-node1，手动解压</p>
<p>上传解压</p>
<pre><code class="language-bash"># docker load -i tomcat.tar.gz&amp;&amp; docker load -i busybox-1-28.tar.gz
</code></pre>
<p>创建tomcat资源</p>
<pre><code class="language-bash"># kubectl apply -f tomcat.yaml
#kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
demo-pod   2/2   Running   0          24s

k8s-master1
# kubectl apply -f tomcat-service.yaml
service/tomcat created
k8s-master1
#kubectl get svc
NAME         TYPE      CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.255.0.1      &lt;none&gt;      443/TCP          5h46m
tomcat       NodePort    10.255.185.15   &lt;none&gt;      8080:30080/TCP   12s
</code></pre>
<p>在浏览器访问 k8s-node1 节点的 ip:30080 即可请求到tomcat页面<br>
<img src="https://img-blog.csdnimg.cn/cd50b68845234f27a934252a8d5819e6.png" alt="在这里插入图片描述" loading="lazy"></p>
<h2 id="33-验证-cordns-是否正常">3.3 验证 cordns 是否正常</h2>
<p>临时运行一个busybox容器进行验证</p>
<pre><code class="language-bash"># kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh
/ # ping www.baidu.com
PING (http://www.baidu.com/) (39.156.66.18): 56 data bytes
64 bytes from 39.156.66.14: seq=2 ttl=127 time=6.409 ms
#通过上面可以看到能访问网络
/ # nslookup kubernetes.default.svc.cluster.local
Server:    10.255.0.2
Address 1: 10.255.0.2 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.255.0.1 kubernetes.default.svc.cluster.local

#注意：
busybox 要用指定的 1.28 版本，不能用最新版本，最新版本，nslookup 会解析不到 dns 和 ip
10.255.0.2 就是我们 coreDNS 的 clusterIP，说明 coreDNS 配置好了。解析内部 Service 的名称，是通过 coreDNS 去解析的。
</code></pre>
<h1 id="四-实现k8s-apiserver-高可用">四、 实现k8s apiserver 高可用</h1>
<p>安装 keepalived+nginx</p>
<p>1、安装 nginx 主备：</p>
<p>在 k8s-master1 和 k8s-master2 上做 nginx 主备安装</p>
<p># yum install nginx keepalived -y</p>
<p>#yum install nginx keepalived -y</p>
<p>2、修改 nginx 配置文件。master1和master2配置一样</p>
<p># cat /etc/nginx/nginx.conf ，添加下面12行内容</p>
<pre><code class="language-bash"># cat /etc/nginx/nginx.conf
# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}
--------------------------添加以下12行upstream内容--------------------------------
stream {
      log_format   main   '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';
      access_log /var/log/nginx/k8s-access.log main;
      upstream k8s-apiserver {
      server 192.168.7.10:6443;       
      server 192.168.7.11:6443;
      }
      server {
      listen 16443;
      proxy_pass k8s-apiserver;
      }
}
----------------------------------------------------------------------------------
http {
    log_formatmain'$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log/var/log/nginx/access.logmain;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 4096;

    include             /etc/nginx/mime.types;
    default_type      application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
      listen       80;
      listen       [::]:80;
      server_name_;
      root         /usr/share/nginx/html;

      # Load configuration files for the default server block.
      include /etc/nginx/default.d/*.conf;

      error_page 404 /404.html;
      location = /404.html {
      }

      error_page 500 502 503 504 /50x.html;
      location = /50x.html {
      }
    }

}
</code></pre>
<p>配置完成后检查一下语法，如遇到下面报错，怎么yum安装一下包即可（ 缺少stream的modules模块 ）</p>
<pre><code class="language-bash"># nginx -t
nginx: unknown directive "stream" in /etc/nginx/nginx.conf:17
nginx: configuration file /etc/nginx/nginx.conf test failed
# yum安装一下即可
yum -y install nginx-mod-stream(或者安装这个nginx-all-modules.noarch包也可以）
</code></pre>
<p>3、keepalive 配置主 keepalived</p>
<p># cat /etc/keepalived/keepalived.conf</p>
<pre><code class="language-bash"># cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
   acassen@firewall.loc
   failover@firewall.loc
   sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX_MASTER
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass 1111
    }
    virtual_ipaddress {
      192.168.7.99/24
    }
track_script {
   check_nginx
   }
}

#vrrp_script：指定检查 nginx 工作状态脚本（根据 nginx 状态判断是否故障转移）
#virtual_ipaddress：虚拟 IP（VIP）
</code></pre>
<p># cat /etc/keepalived/check_nginx.sh</p>
<pre><code class="language-bash">#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ "$count" -eq 0 ];then
   systemctl stop keepalived
fi

chmod +x/etc/keepalived/check_nginx.sh
</code></pre>
<p>备 keepalive</p>
<p># cat /etc/keepalived/keepalived.conf</p>
<pre><code class="language-bash"># cat keepalived.conf
! Configuration File for keepalived

global_defs {
   notification_email {
   acassen@firewall.loc
   failover@firewall.loc
   sysadmin@firewall.loc
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX_BACKUP
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass 1111
    }
    virtual_ipaddress {
      192.168.7.99/24
    }
track_script {
   check_nginx
   }
}

# 注：keepalived 根据脚本返回状态码（0 为工作正常，非 0 不正常）判断是否故障转移。
</code></pre>
<p>4、启动服务：</p>
<pre><code class="language-bash"># systemctl start nginx &amp;&amp; systemctl start keepalived &amp;&amp;systemctl enable nginx keepalived
# systemctl daemon-reload &amp;&amp; systemctl start nginx &amp;&amp; systemctl start keepalived &amp;&amp; systemctl enable nginx keepalived
</code></pre>
<p>一定要查看一下keepalived服务是否正常，是否有vip地址绑定成功</p>
<p>5、测试 vip 是否绑定成功</p>
<p>① 先查看是否在master1节点上绑定了<br>
<img src="https://img-blog.csdnimg.cn/a823e7f8f2674025a18f7a6495a16721.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>6、测试 keepalived：</p>
<p>停掉 k8s-master1 上的 nginx。vip 会漂移到 k8s-master2</p>
<p># service nginx stop<br>
到master2上查看<br>
<img src="https://img-blog.csdnimg.cn/6a034ec92fad4790a5741cd6869a4414.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>目前所有的 Worker Node 组件连接都还是 k8s-master1 Node，如果不改为连接 VIP 走负载均衡器，那么 Master 还是单点故障。</p>
<p>因此接下来就是要改所有 Worker Node（kubectl get node 命令查看到的节点）组件配置文件，由原来 192.168.7.10 修改为 192.168.7.99（VIP）。</p>
<p>在所有 Worker Node 执行：</p>
<pre><code class="language-bash"># sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet-bootstrap.kubeconfig
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet.json
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet.kubeconfig
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kube-proxy.yaml
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kube-proxy.kubeconfig
</code></pre>
<p>最后需要重启一下kube-proxy组件</p>
<pre><code class="language-bash"># systemctl restart kubelet kube-proxy
</code></pre>
<p>这样master的kube-apiserver高可用就安装好了</p>
<pre><code class="language-bbash">erver 192.168.200.1
   smtp_connect_timeout 30
   router_id NGINX_BACKUP
}

vrrp_script check_nginx {
script "/etc/keepalived/check_nginx.sh"
}

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 90
    advert_int 1
    authentication {
      auth_type PASS
      auth_pass 1111
    }
    virtual_ipaddress {
      192.168.7.99/24
    }
track_script {
   check_nginx
   }
}

# 注：keepalived 根据脚本返回状态码（0 为工作正常，非 0 不正常）判断是否故障转移。
</code></pre>
<p>4、启动服务：</p>
<pre><code class="language-bash"># systemctl start nginx &amp;&amp; systemctl start keepalived &amp;&amp;systemctl enable nginx keepalived
# systemctl daemon-reload &amp;&amp; systemctl start nginx &amp;&amp; systemctl start keepalived &amp;&amp; systemctl enable nginx keepalived
</code></pre>
<p>一定要查看一下keepalived服务是否正常，是否有vip地址绑定成功</p>
<p>5、测试 vip 是否绑定成功</p>
<p>① 先查看是否在master1节点上绑定了</p>
<p><img src="https://img-blog.csdnimg.cn/330987bd68654dd0a8afaf673066c997.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>6、测试 keepalived：</p>
<p>停掉 k8s-master1 上的 nginx。vip 会漂移到 k8s-master2</p>
<p># service nginx stop<br>
<img src="https://img-blog.csdnimg.cn/36488e48da58447890432d78f3ddabd7.png" alt="在这里插入图片描述" loading="lazy"></p>
<p>目前所有的 Worker Node 组件连接都还是 k8s-master1 Node，如果不改为连接 VIP 走负载均衡器，那么 Master 还是单点故障。</p>
<p>因此接下来就是要改所有 Worker Node（kubectl get node 命令查看到的节点）组件配置文件，由原来 192.168.7.10 修改为 192.168.7.99（VIP）。</p>
<p>在所有 Worker Node 执行：</p>
<pre><code class="language-bash"># sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet-bootstrap.kubeconfig
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet.json
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kubelet.kubeconfig
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kube-proxy.yaml
# sed -i 's#192.168.7.10:6443#192.168.7.99:16443#' /etc/kubernetes/kube-proxy.kubeconfig
</code></pre>
<p>最后需要重启一下kube-proxy组件</p>
<pre><code class="language-bash"># systemctl restart kubelet kube-proxy
</code></pre>
<p>这样master的kube-apiserver高可用就安装好了</p>


</div>
<div id="MySignature" role="contentinfo">
    我有梦，有远方，我会为了它们奔跑，奋斗，直到成为连我自己都佩服的人。<br><br>
来源：https://www.cnblogs.com/fengdejiyixx/p/16576021.html
頁: [1]
查看完整版本: kubernetes使用二进制安装部署k8s的v1.23版本安装方法及详细步骤