kubeadm安装kubernetes（v1.18.8）

我是一只善良的巨蟹座 發表於 2020-8-22 23:16:00

<h1>1. 前言</h1>
    kubernetes版本更新迭代非常快，上一篇写kubernetes搭建时，版本还是v1.15.0，现在已经更新到v1.18。看kubernetes在github的官方仓库，8月14日小版本已经到了v1.18.8。本篇文章基于kubeadm搭建kubernetes v1.18.8版。 本篇文章与上篇文章几点不同之处：     1. 操作系统采用CentOS。     2. master采用单节点，如果要搭建多master高可用集群可以参考上篇文章。     3. 解决无法访问kubernetes官方容器镜像仓库问题，上篇文章直接从kubernetes官方拉取镜像。     4. docker加速，解决从dockerhub拉镜像慢问题。     5. kube-proxy开启ivps，使用ipvs替代iptables转发流量。     6. 给出了一些常见的错误及排错思路。     话不多说，那就直奔主题，走起~~~
<h1>2. 环境准备</h1>
<table border="0" align="left">
<tbody>
<tr>
<td>机器名称</td>
<td>机器配置</td>
<td>机器系统</td>
<td>IP地址</td>
<td>角色</td>

</tr>
<tr>
<td>master1</td>
<td>2C4G</td>
<td>CentOS7.6</td>
<td>10.13.1.11</td>
<td>主节点</td>

</tr>
<tr>
<td>node1</td>
<td>2C4G</td>
<td>CentOS7.6</td>
<td>10.13.1.15</td>
<td>工作节点1</td>

</tr>
<tr>
<td>node2</td>
<td>2C4G</td>
<td>CentOS7.6</td>
<td>10.13.1.16</td>
<td>工作节点2</td>

</tr>

</tbody>

</table>
 
 
 
 
 
说明： 硬件配置要求：2C2G +； 操作系统要求：CentOS7 + 防火墙说明：如果使用的是云厂商的虚拟机，主节点安全组需放行tcp端口6443、2379-2380、10250-12025，工作节点安全组需放行tcp端口：10250、30000-32767
<h1>3. 实操过程</h1>
<h2>3.1 关闭防火墙和selinux</h2>
<div class="cnblogs_code">
<pre>root@master1 ~]# systemctl stop firewalld
# setenforce 0
# sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config</pre>
</div>
<h2>3.2 关闭交换分区</h2>
<div class="cnblogs_code">
<pre>root@master1 ~]# swapoff -a
永久关闭，修改/etc/fstab,注释掉swap一行</pre>
</div>
<h2>3.3 修改hosts文件</h2>
<div class="cnblogs_code">
<pre># cat >> /etc/hosts << EOF
10.13.1.11 master1
10.13.1.15 node1
10.13.1.16 node2</pre>
</div>
<h2>3.4 时间同步</h2>
<div class="cnblogs_code">
<pre># yum install chrony -y
# systemctl start chronyd
# systemctl enable chronyd
# chronyc sources</pre>
</div>
<h2>3.5 修改内核参数</h2>
让iptables能查看桥接流量
<div class="cnblogs_code">
<pre># cat > /etc/sysctl.d/k8s.conf << EOF
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
# sysctl --system</pre>
</div>
<h2>3.6 加载ipvs模块</h2>
<div class="cnblogs_code">
<pre># modprobe -- ip_vs
# modprobe -- ip_vs_rr
# modprobe -- ip_vs_wrr
# modprobe -- ip_vs_sh
# modprobe -- nf_conntrack_ipv4
# lsmod | grep ip_vs
# lsmod | grep nf_conntrack_ipv4
# yum install -y ipvsadm</pre>
</div>
<h2> 3.7 安装并配置docker</h2>
<h3> 3.7.1 修改docker的yum源为阿里源</h3>
<div class="cnblogs_code">
<pre># wget https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -O /etc/yum.repos.d/docker-ce.repo
或者
# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo</pre>
</div>
<h3>3.7.2 安装docker</h3>
<div class="cnblogs_code">
<pre># yum install -y docker-ce
# docker --version Docker version 19.03.12, build 48a66213fe
# systemctl enable docker
# systemctl start docker</pre>
</div>
<h3>3.7.3 配置docker加速并修改驱动</h3>
网上有很多大佬无私地提供了一些dockerhub加速地址，可以选择几个使用。
<div class="cnblogs_code">
<pre># cat /etc/docker/daemon.json
{
"exec-opts": ["native.cgroupdriver=systemd"],
"registry-mirrors": [
 "https://1nj0zren.mirror.aliyuncs.com",
 "https://kfwkfulq.mirror.aliyuncs.com",
 "https://2lqq34jg.mirror.aliyuncs.com",
 "https://pee6w651.mirror.aliyuncs.com",
 "http://hub-mirror.c.163.com",
 "https://docker.mirrors.ustc.edu.cn",
 "http://f1361db2.m.daocloud.io",
 "https://registry.docker-cn.com"
]
}
# systemctl restart docker
# docker info | grep "Cgroup Driver"
Cgroup Driver: systemd</pre>
</div>
    kubernetes官方建议docker驱动采用systemd，当然可以不修改，只是kubeadm init时会有warning（: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/）可以忽略，生产环境建议修改，因为更稳定。
<h2>3.8 安装kubernents组件</h2>
<h3>3.8.1 配置kubernentes的yum源为阿里源</h3>
因为国内无法访问kubernents的官方yum源，所以需要修改
<div class="cnblogs_code">
<pre># cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo

name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg</pre>
</div>
<h3>3.8.2 安装组件</h3>
<div class="cnblogs_code">
<pre># yum -y install kubelet kubeadm kubectl
# kubelet --version
Kubernetes v1.18.8
# systemctl start kubelet</pre>
</div>
此时kubelet处于不断重启状态，因为集群还没有初始化，kubelet等待kubeadm初始化完成后运行状态正常。
<h2>3.9 初始化集群</h2>
<h3>3.9.1 查看初始化需要的镜像</h3>
<div class="cnblogs_code">
<pre># kubeadm config images list
W0822 15:58:54.182176 25602 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups
k8s.gcr.io/kube-apiserver:v1.18.8
k8s.gcr.io/kube-controller-manager:v1.18.8
k8s.gcr.io/kube-scheduler:v1.18.8
k8s.gcr.io/kube-proxy:v1.18.8
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7</pre>
</div>
<h3>3.9.2 kubeadm init介绍</h3>
    初始化集群需使用kubeadm init命令，可以指定具体参数初始化，也可以指定配置文件初始化。 可选参数： --apiserver-advertise-address apiserver的监听地址，有多块网卡时需要指定 --apiserver-bind-port apiserver的监听端口，默认是6443 --cert-dir 通讯的ssl证书文件，默认/etc/kubernetes/pki --control-plane-endpoint 控制台平面的共享终端，可以是负载均衡的ip地址或者dns域名，高可用集群时需要添加 --image-repository 拉取镜像的镜像仓库，默认是k8s.gcr.io --kubernetes-version 指定kubernetes版本 --pod-network-cidr pod资源的网段，需与pod网络插件的值设置一致 --service-cidr service资源的网段 --service-dns-domain service全域名的后缀，默认是cluster.local
<h3>3.9.3 kubeadm指定具体参数初始化 </h3>
    因为以上镜像都需要从kubernetes官方镜像仓库拉取，国内无法访问，所以需要设置国内的阿里镜像仓库。 但是目前至发稿为止，阿里的kube-apiserver、kube-controller、proxy镜像只更新到v1.18.6所以无法拉取到v1.18.8版镜像。     如果你需要安装的kubernetes版本是v1.18.6及以下那么请继续往下走，如果是v1.18.7及以上请跳过这一步进入下一步。
<h4>3.9.3.1 初始化</h4>
<div class="cnblogs_code">
<pre># kubeadm init --kubernetes-version=v1.18.6--apiserver-advertise-address 0.0.0.0 --image-repository registry.aliyuncs.com/google_containers --pod-network-cidr 192.168.0.0/16 --service-cidr 10.10.0.0/16
--kubernetes-version 请指定为你需要安装的v1.18.6及以下的版本。</pre>
</div>
初始化成功后会出现如下信息
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822220223865-694583745.png" alt="" loading="lazy">
<h4> 3.9.3.2 开启ipvs</h4>
修改kube-proxy的configmap
<div class="cnblogs_code">
<pre># kubectl edit cm kube-proxy -n=kube-system
修改mode: ipvs</pre>
</div>
<h3>3.9.4 kubeadm指定配置文件初始化</h3>
<h4>3.9.4.1 下载kubernetes所需的全部镜像</h4>
dockerhub上面已经有大佬已经上传了最新的1.18.8镜像，这里我们直接下载下来即可。
<div class="cnblogs_code">
<pre># vim images.txt
kube-apiserver:v1.18.8 # node节点不需要
kube-controller-manager:v1.18.8 # node节点不需要
kube-scheduler:v1.18.8 # node节点不需要
kube-proxy:v1.18.8
pause:3.2
etcd:3.4.3-0 # node节点不需要
coredns:1.6.7 # node节点不需要
# vim images.sh
for image in `cat images.txt`
do
docker pull gotok8s/$image
docker tag gotok8s/$image k8s.gcr.io/$image
docker rmi gotok8s/$image
done
# sh images.sh</pre>
</div>
<h4>3.9.4.2 配置初始化文件</h4>
<div class="cnblogs_code">
<pre># kubeadm config print init-defaults > kubeadm.yaml
# vim kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta2
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 0.0.0.0# 修改为本机ip地址，多块网卡可以指定具体ip
bindPort: 6443
nodeRegistration:
criSocket: /var/run/dockershim.sock
name: master1
taints:
- effect: NoSchedule
key: node-role.kubernetes.io/master
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.18.8 # 修改为最新版本
networking:
dnsDomain: cluster.local
serviceSubnet: 10.10.0.0/16 # service网段
podSubnet: 192.168.0.0/16 # pod网段，需与网络插件网段一致
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs # 开启ipvs
# kubeadm init --config=kubeadm.yaml</pre>
</div>
可以看到久违的成功
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822220824129-1964299192.png" alt="" loading="lazy">
<h4>3.9.4.3 配置kubectl与kube-apiserver交互</h4>
<div class="cnblogs_code">
<pre># mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
至此可以使用kubectl查看集群节点状态了
# kubectl get nodes</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822220903585-1395405195.png" alt="" loading="lazy">
 可以看到版本是v1.18.8最新版，但是master状态是NotReady，因为此时还没有安装网络组件。
<h3> 3.9.5 安装网络组件 </h3>
<div class="cnblogs_code">
<pre># wget https://docs.projectcalico.org/v3.14/manifests/calico.yaml
# kubectl apply -f calico.yaml</pre>
</div>
此时再来看节点状态，已经正常
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221123077-1247754647.png" alt="" loading="lazy">
查组件状态
<div class="cnblogs_code">
<pre># kubectl get cs</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221215195-150811728.png" alt="" loading="lazy">
<h2>3.10 worker节点加入集群</h2>
<div class="cnblogs_code">
<pre># kubeadm join 10.13.1.11:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:c214cf4c42766dd3d4ab2842c11efbefd54aa445993708ccdbdb8f111658445e
同样的第二个worker节点加入集群
此次查看集群状态
# kubectl get nodes
# kubectl get pods -A</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200823104541968-1471699527.png" alt="" loading="lazy">
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200823214400500-2118045122.png" alt="" loading="lazy">
如果join的token之前没有记住，没关系，在master重新生成一下
<div class="cnblogs_code">
<pre># kubeadm token create --print-join-command</pre>
</div>
可以看到节点和个组件的pod状态均正常，至此集群搭建完毕！
<h1>4.  trouble shooting</h1>
    出现了问题不可怕，因为如果不非常仔细，按照文档敲下来很有可能会出错。可怕的是，出错了连去网上多搜索一下都懒得搜，甚至开始抱怨。kubernets已经很成熟了，网上的资料非常多，出现的问题很多人也遇到过，在网上基本都有大佬给出回答。
<h2>4.1 初始化集群长期卡住，最终报错</h2>
拉镜像失败
<div class="cnblogs_code">
<pre>: failed to pull image k8s.gcr.io/kube-apiserver:v1.18.8: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1</pre>
</div>
 
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221526164-1288480628.png" alt="" loading="lazy">
执行拉镜像脚本
<div class="cnblogs_code">
<pre># sh images.sh</pre>
</div>
4.2 网桥报错
<div class="cnblogs_code">
<pre>W0822 17:05:25.135752 3367 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups
Using Kubernetes version: v1.18.8
Running pre-flight checks
: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: Some fatal errors occurred:
: /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1
If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221722081-600123311.png" alt="" loading="lazy">
检查前面的内核参数是否修改 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1
<h2>4.3 kubelet-check健康检查报错</h2>
<div class="cnblogs_code">
<pre> The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp: lookup localhost on 198.18.254.40:53: no such host.
It seems like the kubelet isn't running or healthy.
The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp: lookup localhost on 198.18.254.40:53: no such host.
It seems like the kubelet isn't running or healthy.
The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp: lookup localhost on 198.18.254.40:53: no such host.
It seems like the kubelet isn't running or healthy.
The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp: lookup localhost on 198.18.254.40:53: no such host.
It seems like the kubelet isn't running or healthy.
The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp: lookup localhost on 198.18.254.40:53: no such host.</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221828178-851987712.png" alt="" loading="lazy">
检查hosts文件，本机地址解析是否被删除了 127.0.0.1 localhost
<h2>4.4 查看组件处于不健康状态</h2>
<div class="cnblogs_code">
<pre># kubectl get cs</pre>
</div>
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221913290-1107873216.png" alt="" loading="lazy">
发现controller-manager和scheduler状态是不健康 如果节点都处于Ready后，实际上该状态是不影响的。 因为kubeadm v1.18.6及以后的版本，是默认不开启controller-manager的10252和scheduler的10251端口的，一般10251和10252是监听在http上面，不需要证书认证，属于不安全的端口。 查看机器监听端口，发现默认这两个端口没有监听
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822221938422-1032104809.png" alt="" loading="lazy">
可以将/etc/kubernetes/manifests/kube-controller-manager.yaml、/etc/kubernetes/manifests/kube-scheduler.yaml中--port=0注释掉，再次kubect get cs组件状态，这时都是ok了。 默认不监听http端口，但是客户端查看组件状态有默认是检查http端口，不知道这算不算是kuberadm的一个小bug呢，哈哈～～～
<h2>4.5 worker节点加入集群后长期处于NotReady</h2>
查看node
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222020345-733421768.png" alt="" loading="lazy">
查看pod状态，kube-proxy一直处于创建状态，网络组件calico一直处于初始化
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222041160-868675984.png" alt="" loading="lazy">
kubect describe pod查看报错：FailedCreatePodSandBox
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222109111-1638202310.png" alt="" loading="lazy">
说明节点没有pull基础镜像pause，到相应节点上面去拉取pause镜像即可。
<h2>4.6 node节点运行一段时间后出现错误</h2>
查看pod状态
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222200706-192828345.png" alt="" loading="lazy">
 查看报错的pod，发现cgroup报错
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222305256-318670946.png" alt="" loading="lazy">
 到node节点查看kubelet也在报错
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222325142-1294627072.png" alt="" loading="lazy">
原因是docker的驱动改为了systemd，但是kubelet的驱动没有改 kubelet的驱动是在kubeadm join初始化的时候与docker保持一致的，docker修改了，kubelet也需要修改
<div class="cnblogs_code">
<pre># sed -i 's/--cgroup-driver=cgroupfs/--cgroup-driver=systemd/g' /var/lib/kubelet/kubeadm-flags.env
# systemctl restart kubelet</pre>
</div>
同样如果node节点处于NotReady，kubelet一直在重启，也需要检查kubelet的cgroup驱动是否和docker的一致。
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222349856-1832979882.png" alt="" loading="lazy">
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222403839-263509932.png" alt="" loading="lazy">
<h2>4.7 worker节点加入节点后，网络组件一直不ready，一直重启</h2>
该pod一直处于Running和CrashLoopBackOff交替状态
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222440167-721009150.png" alt="" loading="lazy">
查看pod日志，发现有大量的健康检查错误
<img src="https://img2020.cnblogs.com/blog/1450914/202008/1450914-20200822222501944-465930276.png" alt="" loading="lazy">
到node节点上查看hosts文件，本机地址解析是否被删除了 127.0.0.1 localhost
<h1>参考文档</h1>
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ https://kubernetes.io/zh/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
 
全剧终
 
 
<h2> </h2>
  
来源：https://www.cnblogs.com/technology178/p/13547342.html

頁: [1]

琼殿技术论坛's Archiver

kubeadm安装kubernetes（v1.18.8）