Kubernetes 部署集群1.28.2版本(无坑)

微豆豆 發表於 2024-4-21 23:36:00

<ul>
<li>初步搭建一个一个主节点和两个从节点Kubernetes 1.28.2 集群。先准备好机器</li>
</ul>
<table>
<thead>
<tr>
<th>host</th>
<th>hostname</th>
<th>os</th>
<th>role</th>
<th>hardware</th>
</tr>
</thead>
<tbody>
<tr>
<td>192.168.31.200</td>
<td>master01</td>
<td>centos7.9</td>
<td>control-plane</td>
<td>cpu:2c 内存: 3G 硬盘1:50G</td>
</tr>
<tr>
<td>192.168.31.201</td>
<td>node01</td>
<td>centos7.9</td>
<td>worker</td>
<td>cpu:2c 内存: 3G 硬盘1:50G 硬盘2:50G</td>
</tr>
<tr>
<td>192.168.31.202</td>
<td>node02</td>
<td>centos7.9</td>
<td>worker</td>
<td>cpu:2c 内存: 3G 硬盘1:50G 硬盘2:50G</td>
</tr>
</tbody>
</table>
<ul>
<li>预留了201、202节点，后续扩容集群做control-plane节点高可用。</li>
<li>所有work节点各分配一块50硬盘，后续做ceph存储用。</li>
</ul>
<h3 id="1-基础环境准备">1. 基础环境准备</h3>
<ul>
<li><strong>所有节点全部执行环境初始化，（后面如果要给集群新增节点也要做这个操作）</strong></li>
<li><strong>所有节点按照表格修改主机名</strong></li>
</ul>
<h4 id="1所有节点全部关闭防火墙">1、所有节点全部关闭防火墙</h4>
<pre><code class="language-bash">systemctl stop firewalld
systemctl disable firewalld
systemctl is-enabled firewalld
</code></pre>
<h4 id="2配置ntp-server同步时间">2、配置ntp server同步时间</h4>
<pre><code class="language-bash">ntpdate ntp1.aliyun.com
vi /etc/crontab
1 * * * * root /usr/sbin/ntpdate ntp1.aliyun.com && /sbin/hwclock -w
</code></pre>
<h4 id="3永久关闭selinux">3、永久关闭selinux</h4>
<pre><code class="language-bash">sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
setenforce 0
</code></pre>
<h4 id="4关闭swap分区">4、关闭swap分区</h4>
<pre><code class="language-bash">sed -i '/swap/s/^/#/g' /etc/fstab
swapoff -a
</code></pre>
<h4 id="5配置hosts">5、配置hosts</h4>
<pre><code class="language-bash"># cat >>/etc/hosts <<EOF
192.168.31.200 master01
192.168.31.201 node01
192.168.31.202 node02
EOF
</code></pre>
<h4 id="6内核升级">6、内核升级</h4>
<p><strong>简介：</strong> centos7 yum工具在线升级内核</p>
<h5 id="1查看当前内核版本">1、查看当前内核版本</h5>
<pre><code class="language-bash"># uname -a
Linux localhost.localdomain 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>
<h5 id="2导入elpepo仓库公共密钥">2、导入ELPepo仓库公共密钥</h5>
<pre><code class="language-bash">rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
</code></pre>
<h5 id="3安装elpepo的仓库yum源">3、安装ELPepo的仓库yum源</h5>
<pre><code class="language-bash">rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
</code></pre>
<h5 id="4选择lt版本安装">4、选择lt版本安装</h5>
<ul>
<li>lt长期维护版</li>
<li>ml最新稳定版</li>
</ul>
<pre><code class="language-bash">yum -y--enablerepo=elrepo-kernel install kernel-lt
</code></pre>
<h5 id="5设置内核默认启动">5、设置内核默认启动</h5>
<pre><code class="language-bash">sudo awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
grub2-set-default 0
</code></pre>
<p>6、重启</p>
<pre><code class="language-bash">reboot
</code></pre>
<p>7、查看内核版本</p>
<pre><code class="language-bash"># uname -a
Linux localhost.localdomain 4.4.244-1.el7.elrepo.x86_64 #1 SMP Tue Nov 17 18:57:10 EST 2020 x86_64 x86_64 x86_64 GNU/Linux
</code></pre>
<h3 id="2-配置kubernetes运行环境">2. 配置Kubernetes运行环境</h3>
<ul>
<li><strong>下面的操作所有节点全部执行，后面如果要给集群新增节点也要做这个操作</strong></li>
</ul>
<h4 id="1配置内核参数">1、配置内核参数</h4>
<pre><code class="language-bash">cat > /etc/sysctl.d/Kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF
# 使配置生效
sysctl --system
</code></pre>
<p>这些配置参数的含义是：</p>
<ol>
<li><code>net.bridge.bridge-nf-call-ip6tables = 1</code>：当通过桥接网络接收到IPv6数据包时，将调用<code>ip6tables</code>的规则进行处理。</li>
<li><code>net.bridge.bridge-nf-call-iptables = 1</code>：当通过桥接网络接收到IPv4数据包时，将调用<code>iptables</code>的规则进行处理。</li>
<li><code>net.ipv4.ip_forward = 1</code>：允许IPv4的数据包转发，即使数据包的目标不是本机。</li>
<li><code>vm.swappiness = 0</code>: vm.swappiness是操作系统控制物理内存交换出去的策略。它允许的值是一个百分比的值，最小为0，最大运行100，该值默认为60。vm.swappiness设置为0表示尽量少swap，100表示尽量将inactive的内存页交换出去。</li>
</ol>
<p>Kubernetes通过iptables实现服务发现和网络流量路由，pod通信。这一步很重要。没有设置的话会导致集群网络通信故障，如pod无法通信。核模块</p>
<pre><code class="language-bash">yum -y install conntrack ipvsadm ipset jq iptables curl sysstat libseccomp wget vim net-tools git
# 相关内核模块
cat > /etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF
# 启动服务
systemctl enable --now systemd-modules-load
</code></pre>
<ul>
<li><code>ip_vs</code>，<code>ip_vs_rr</code>，<code>ip_vs_wrr</code>，<code>ip_vs_sh</code> 是IPVS相关的内核模块。它们提供了不同的负载均衡算法（round-robin，加权轮询，最短任务优先）。</li>
<li><code>nf_conntrack</code> 和 <code>nf_conntrack_ipv4</code> 是用于网络连接跟踪的内核模块，这在防火墙和NAT中非常重要。</li>
<li>linux kernel 4.19版本已经将nf_conntrack_ipv4 更新为 nf_conntrack</li>
</ul>
<h4 id="3重启系统">3、重启系统</h4>
<pre><code class="language-bash"># reboot
# 检查是否加载成功
lsmod |egrep "ip_vs|nf_conntrack_ipv4"
nf_conntrack_ipv4    1505326
nf_defrag_ipv4       127291 nf_conntrack_ipv4
ip_vs_sh             126880
ip_vs_wrr          126970
ip_vs_rr             126000
ip_vs             1454586 ip_vs_rr,ip_vs_sh,ip_vs_wrr
nf_conntrack       13926410 ip_vs,nf_nat,nf_nat_ipv4,nf_nat_ipv6,xt_conntrack,nf_nat_masquerade_ipv4,nf_nat_masquerade_ipv6,nf_conntrack_netlink,nf_conntrack_ipv4,nf_conntrack_ipv6
libcrc32c          126444 xfs,ip_vs,nf_nat,nf_conntrack
</code></pre>
<h4 id="4安装-containerd">4、安装 containerd</h4>
<ul>
<li>
<p>顺便介绍一下历史背景。早期docker势大，但docker没有实现CRI，Kubernetes只能用dockershim做适配器来兼容docker，使其可以接入cri，这个dockershim在Kubernetes1.24版本就被放弃维护了。containerd是从docker中分离出来的开源项目，强调简单性、健壮性和可移植性。它负责以下工作</p>
</li>
<li>
<p>管理容器的生命周期（从创建容器到销毁容器）</p>
</li>
<li>
<p>拉取/推送容器镜像</p>
</li>
<li>
<p>存储管理（管理镜像及容器数据的存储）</p>
</li>
<li>
<p>调用 runc 运行容器（与 runc 等容器运行时交互，runc是oci 开放容器标准的一个实现。oci就是创建容器需要做一些 namespaces 和 cgroups 的配置，以及挂载 root 文件系统等操作的规范）</p>
</li>
<li>
<p>管理容器网络接口及网络</p>
</li>
</ul>
<pre><code class="language-bash">yum -y install yum-utils device-mapper-persistent-data lvm2
# 添加阿里源
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 配置 containerd
cat >>/etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOF
# 立刻加载 overlay模块
modprobe overlay
# 立刻加载 br_netfilter模块
modprobe br_netfilter
# 安装containerd
yum install containerd.io -y
</code></pre>
<ul>
<li><code>overlay</code> 是一个文件系统类型，它支持在不改变底层文件的情况下，将改动保存在另一个分离的文件层。它常用于 Docker 和其他容器运行时中，用来创建容器的文件系统。（写时复制）</li>
<li><code>br_netfilter</code> 是一个网络相关的内核模块，它允许 iptables 和其他网络工具对桥接流量进行过滤。这在 Kubernetes 网络设置中很重要，特别是在使用 overlay 网络（如 flannel、Calico 等）时。</li>
</ul>
<h4 id="5配置containerd">5、配置containerd</h4>
<pre><code class="language-bash">mkdir -p /etc/containerd
containerd config default > /etc/containerd/config.toml
# 使用systemd管理cgroups
sed -i '/SystemdCgroup/s/false/true/g' /etc/containerd/config.toml
# 配置sadnbox image从阿里云拉取
sed -i '/sandbox_image/s/registry.k8s.io/registry.aliyuncs.com\/google_containers/g' /etc/containerd/config.toml
sed -i 's#sandbox_image = "registry.k8s.io/pause:3.6"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
# 启动containerd
systemctl enable containerd
systemctl start containerd
</code></pre>
<h3 id="3安装kubeamdkubeletkubectl">3、安装kubeamd、kubelet、kubectl</h3>
<ul>
<li><strong>下面的操作所有节点全部执行</strong>，后面如果要给集群新增节点也要做这个操作</li>
</ul>
<h4 id="1添加阿里源">1、添加阿里源</h4>
<pre><code class="language-bash">cat >/etc/yum.repos.d/kubernetes.repo <<EOF

name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
</code></pre>
<h4 id="2安装并启动">2、安装并启动</h4>
<pre><code class="language-bash"># 查看可用版本
yum list kubelet --showduplicates |grep 1.28

# 开始安装这篇文档写下时，最新版本为1.28.2我直接安装的最新版
yum -y install kubectl-1.28.2 kubelet-1.28.2 kubeadm-1.28.2

# 启动
systemctl enable kubelet
systemctl start kubelet
</code></pre>
<h3 id="4部署control-plane节点">4、部署control-plane节点</h3>
<ul>
<li><strong>以下操作只在control-plane节点执行</strong></li>
</ul>
<h4 id="1使用kubeadm初始化">1、使用kubeadm初始化</h4>
<pre><code class="language-bash"># 查看所需镜像
# kubeadm config images list --kubernetes-version=v1.28.2
registry.k8s.io/kube-apiserver:v1.28.2
registry.k8s.io/kube-controller-manager:v1.28.2
registry.k8s.io/kube-scheduler:v1.28.2
registry.k8s.io/kube-proxy:v1.28.2
registry.k8s.io/pause:3.9
registry.k8s.io/etcd:3.5.7-0
registry.k8s.io/coredns/coredns:v1.10.1

# 初始化
# kubeadm init --kubernetes-version=1.28.2 \
--apiserver-advertise-address=192.168.31.200\
--image-repositoryregistry.aliyuncs.com/google_containers \
--pod-network-cidr=172.16.0.0/16
</code></pre>
<ul>
<li><code>apiserver-advertise-address</code>写control-plane的ip</li>
<li><code>pod-network-cidr</code>写个不冲突的网段</li>
<li><code>image-repository</code>指定从阿里云拉取镜像</li>
</ul>
<p>命令执行完成后会返回一长段内容，主要看最后部分</p>
<pre><code class="language-bash">Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f .yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.96.101:6443 --token l906wz.0fydt3hcfbogwlo9 \
   --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
</code></pre>
<ul>
<li>按提示操作配置认证</li>
</ul>
<pre><code class="language-bash">mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf
</code></pre>
<ul>
<li>检查</li>
</ul>
<pre><code class="language-bash"># kubectl get node
NAME                   STATUS       ROLES          AGE    VERSION
control-plane01       NotReady    control-plane 50s    v1.28.2
# kubectl get pods -A
# kubectl get pods -n kube-system
NAME                                        READY STATUS RESTARTS    AGE
coredns-7bdc4cb885-fs2tz                   1/1 Pending 0          13d
coredns-7bdc4cb885-wk7c9                   1/1 Pending 0          13d
etcd-control-plane01                         1/1 Running 0          13d
kube-apiserver-control-plane01             1/1 Running 0          13d
kube-controller-manager-control-plane01    1/1 Running 0          13d
kube-proxy-mfzmq                            1/1 Running 3 (25h ago) 13d
kube-scheduler-control-plane01             1/1 Running 0          13d
</code></pre>
<ul>
<li>加入集群token过期或者遗忘，获取加入集群命令</li>
</ul>
<pre><code class="language-bash">kubeadm token create --print-join-command
</code></pre>
<h4 id="2部署calico">2、部署calico</h4>
<h5 id="1安装calico网络插件">1、安装calico网络插件</h5>
<pre><code class="language-bash">wget https://docs.projectcalico.org/manifests/calico.yaml
</code></pre>
<p>改为10.244.0.0/16</p>
<p><img src="https://typora0010.oss-cn-chengdu.aliyuncs.com/typora%E7%9A%84%E5%9B%BE%E7%89%87/34c4bb26fd3d465a933a4debb2839880.png" alt="img" loading="lazy"></p>
<h5 id="2指定网卡">2、指定网卡</h5>
<pre><code class="language-yaml"># Cluster type to identify the deployment type
- name: CLUSTER_TYPE
value: "k8s,bgp"
# 下面添加
- name: IP_AUTODETECTION_METHOD
value: "interface=eth0"
# eth0为本地网卡名字
</code></pre>
<ul>
<li>calico 自动探查互联网卡，如果有多快网卡，则可以配置用于互联的网络接口命名正则表达式，如上面的 eth0 (根据自己服务器的网络接口名修改)；不指定网卡</li>
<li>创建pod时会有如下报错</li>
</ul>
<pre><code class="language-bash"> Failed to create pod sandbox: rpc error: code = Unknown desc =
</code></pre>
<h5 id="3部署">3、部署</h5>
<pre><code class="language-bash">kubectl apply -f calico.yaml
# 检查
#kubectl get pods -n kube-system
NAME                                     READY STATUS RESTARTS    AGE
calico-kube-controllers-6849cf9bcf-gv6xx 1/1 Running 0          13d
calico-node-2d7xx                      1/1 Running 0          13d
coredns-7bdc4cb885-fs2tz                1/1 Running 0          13d
coredns-7bdc4cb885-wk7c9                1/1 Running 0          13d
etcd-control-plane01                   1/1 Running 0          13d
kube-apiserver-control-plane01          1/1 Running 0          13d
kube-controller-manager-control-plane01 1/1 Running 0          13d
kube-proxy-mfzmq                         1/1 Running 3 (25h ago) 13d
kube-scheduler-control-plane01          1/1 Running 0          13d
</code></pre>
<h3 id="5worker节点加入集群">5、worker节点加入集群</h3>
<pre><code class="language-bash"># 所有worker节点都执行
kubeadm join 192.168.31.200:6443 --token l906wz.0fydt3hcfbogwlo9 \
   --discovery-token-ca-cert-hash sha256:2604d3aab372a483b26bcbdafdb54d7746226975c3a317db07d94eccdfca51be
# 查看状态
# kubectl get nodes
NAME       STATUS ROLES       AGE VERSION
control-plane01 Ready control-plane 13d v1.28.2
node01 Ready <none>       13d v1.28.2
node02 Ready <none>       13d v1.28.2
node03 Ready <none>       13d v1.28.2
</code></pre>
<h3 id="6kubernetes-dashboard-安装">6、Kubernetes dashboard 安装</h3>
<h4 id="1安装命令补全">1、安装命令补全</h4>
<pre><code class="language-bash">yum -y install bash-completion
echo "source <(kubectl completion bash)" >> /etc/profile
source /etc/profile
</code></pre>
<h4 id="2kubernetes-dashboard安装可选kubesphere更好用">2、kubernetes-dashboard安装（可选，kubesphere更好用）</h4>
<pre><code class="language-bash">kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v3.0.0-alpha0/charts/kubernetes-dashboard.yaml
</code></pre>
<ul>
<li>修改如下内容</li>
</ul>
<pre><code class="language-yaml">kind: Service
apiVersion: v1
metadata:
labels:
Kubernetes-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kubernetes-dashboard
spec:
type: NodePort    # 增加内容
ports:
- port: 443
   targetPort: 8443
   nodePort: 30000# 增加内容（端口范围30000-32767）
selector:
Kubernetes-app: kubernetes-dashboard
# 安装
kubectl apply -f recommended.yaml
# 查看进度
# kubectl get all -n kubernetes-dashboard
NAME                                           READY STATUS RESTARTS    AGE
pod/dashboard-metrics-scraper-5cb4f4bb9c-h549p 1/1 Running 3 (26h ago) 13d
pod/kubernetes-dashboard-6967859bff-cm4tl    1/1 Running 4 (26h ago) 13d

NAME                            TYPE    CLUSTER-IP    EXTERNAL-IP PORT(S)       AGE
service/dashboard-metrics-scraper ClusterIP 10.108.31.72 <none>    8000/TCP    13d
service/kubernetes-dashboard    NodePort 10.102.47.173 <none>    443:30000/TCP 13d

NAME                                     READY UP-TO-DATE AVAILABLE AGE
deployment.apps/dashboard-metrics-scraper 1/1 1          1       13d
deployment.apps/kubernetes-dashboard    1/1 1          1       13d

NAME                                                 DESIRED CURRENT READY AGE
replicaset.apps/dashboard-metrics-scraper-5cb4f4bb9c 1       1       1    13d
replicaset.apps/kubernetes-dashboard-6967859bff    1       1       1    13d
</code></pre>
<ul>
<li>创建admin用户</li>
</ul>
<pre><code class="language-yaml"># vim admin.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin
namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin
namespace: kubernetes-dashboard
---
apiVersion: v1
kind: Secret
metadata:
name: kubernetes-dashboard-admin
namespace: kubernetes-dashboard
annotations:
kubernetes.io/service-account.name: "admin"
type: kubernetes.io/service-account-token
# 创建admin用户token
kubectl -n kubernetes-dashboard create token admin

# 获取token
Token=$(kubectl -n kubernetes-dashboard get secret |awk '/kubernetes-dashboard-admin/ {print $1}')
kubectl describe secrets -n kubernetes-dashboard ${Token} |grep token |awk 'NR==NF {print $2}'
</code></pre>
<ul>
<li>然后就可以使用token登陆了，地址是集群任意节点IP:30000</li>
</ul>
<h3 id="7kubernetes-metrics-server-插件部署">7、Kubernetes metrics-server 插件部署</h3>
<h4 id="1metrics-server-介绍">1、metrics-server 介绍</h4>
<ul>
<li>heapster已经被metrics-server取代，如果使用kubernetes的自动扩容功能的话，那首先得有一个插件，然后该插件将收集到的信息（cpu、memory..）与自动扩容的设置的值进行比对，自动调整pod数量。关于该插件，在kubernetes的早些版本中采用的是heapster，1.13版本正式发布后，丢弃了heapster，官方推荐采用metrics-sever。</li>
<li>metrics server为Kubernetes自动伸缩提供一个容器资源度量源。metrics-server 从 kubelet 中获取资源指标，并通过 Metrics API 在 Kubernetes API 服务器中公开它们，以供 HPA 和 VPA 使用。</li>
</ul>
<h4 id="2安装步骤">2、安装步骤</h4>
<h5 id="1获取yaml文件">1、获取yaml文件。</h5>
<pre><code class="language-bash">wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml -O metrics-server.yaml
</code></pre>
<ul>
<li>编辑yaml文件。之前部署集群用的自签名证书，metrics-server直接请求kubelet接口会证书校验失败，因此deployment中增加- --kubelet-insecure-tls参数。另外镜像原先在registry.k8s.io，国内下载不方便，下面的配置中修改成了国内镜像仓库地址。内网环境中可以先下载，然后再推到内网镜像仓库，镜像也改成内网镜像地址。</li>
</ul>
<pre><code class="language-yaml">apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
# ...
template:
spec:
   containers:
   - args:
   - --cert-dir=/tmp
   - --secure-port=4443
   - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
   - --kubelet-use-node-status-port
   - --metric-resolution=15s
   - --kubelet-insecure-tls # 需要新加的一行
   image: registry.cn-hangzhou.aliyuncs.com/rainux/metrics-server:v0.6.4
</code></pre>
<h5 id="2安装发布">2、安装发布</h5>
<pre><code class="language-bash">kubectl apply -f metrics-server.yaml
# 查看是否在运行
kubectl get pods -n kube-system | grep metrics
# 获取集群的指标数据
kubectl get --raw /apis/metrics.k8s.io/v1beta1 | python3 -m json.tool
</code></pre>
<p>根据输出可见，集群提供nodes和pods的资源指标。</p>
<pre><code class="language-json">{
"kind": "APIResourceList",
"apiVersion": "v1",
"groupVersion": "metrics.k8s.io/v1beta1",
"resources": [
{
   "name": "nodes",
   "singularName": "",
   "namespaced": false,
   "kind": "NodeMetrics",
   "verbs": [
   "get",
   "list"
   ]
},
{
   "name": "pods",
   "singularName": "",
   "namespaced": true,
   "kind": "PodMetrics",
   "verbs": [
   "get",
   "list"
   ]
}
]
}
</code></pre>
<h5 id="3测试验证">3、测试验证</h5>
<pre><code class="language-bash">#1-2分钟后查看结果
#kubectl top nodes
NAME       CPU(cores) CPU% MEMORY(bytes) MEMORY%
k8s-master 256m       12% 2002Mi       52%
k8s-node1 103m       5% 1334Mi       34%
k8s-node2 144m       7% 1321Mi       34%
</code></pre>
<h5 id="4top命令">4、top命令</h5>
<ul>
<li>kubectl top命令用来查看node节点和pod的资源使用情况。</li>
</ul>
<pre><code class="language-bash"># 查看 top 命令的帮助
kubectl top --help
# 查看node节点的资源使用情况
kubectl top node
# 查看pod的资源使用情况
kubectl top pod
# 查看所有命名空间的pod资源使用情况
kubectl top pod -A
</code></pre>
<ul>
<li>再回到dashboard界面可以看到CPU和内存使用情况了：<img src="https://typora0010.oss-cn-chengdu.aliyuncs.com/typora%E7%9A%84%E5%9B%BE%E7%89%87/Kubernetes-Dashboard-11.png" alt="img" loading="lazy"></li>
</ul>
<p><strong>源码</strong></p>
<ul>
<li>Github仓库:https://github.com/sunweisheng/Kubernetes</li>
</ul>
<h5 id="5导出认证添加配置">5、导出认证添加配置</h5>
<pre><code class="language-bash"># vim /root/.kube/config # 增加 token 内容
- name: admin
user:
client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUQxekNDQXIrZ0F3SUJBZ0lVTFFhcXpaaitVc0tRU1BiWVlMRmxDWnhDZVBNd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1qQXdOREU1TURVeE1UQXdXaGdQTWpBM01EQTBNRGN3TlRFeE1EQmFNR2N4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEZ6QVZCZ05WQkFvVApEbk41YzNSbGJUcHRZWE4wWlhKek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweERqQU1CZ05WQkFNVEJXRmtiV2x1Ck1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBeG1MWWxNQXFEeGVreXljWWlvQXUKU2p5VzhiUCtxTzF5bUhDWHVxSjQ3UW9Vd0lSVEFZdVAyTklQeFBza04xL3ZUeDBlTjFteURTRjdYd3dvTjR5cApacFpvRjNaVnV1NFNGcTNyTUFXT1d4VU93REZNZFZaSkJBSGFjZkdMemdOS01FZzRDVDhkUmZBUGxrYVdxNkROCmJKV3JYYW41WGRDUnE2NlpTdU9lNXZXTWhENzNhZ3UzWnBVZWtHQmpqTEdjNElTL2c2VzVvci9LeDdBa0JuVW0KSlE3M2IyWUl3QnI5S1ZxTUFUNnkyRlhsRFBpaWN1S0RFK2tGNm9leG04QTljZ1pKaDloOFZpS0trdnV3bVh5cwpNREtIUzJEektFaTNHeDVPUzdZR1ZoNFJGTGp0VXJuc1h4TVBtYWttRFV1NkZGSkJsWlpkUTRGN2pmSU9idldmCjlRSURBUUFCbzM4d2ZUQU9CZ05WSFE4QkFmOEVCQU1DQmFBd0hRWURWUjBsQkJZd0ZBWUlLd1lCQlFVSEF3RUcKQ0NzR0FRVUZCd01DTUF3R0ExVWRFd0VCL3dRQ01BQXdIUVlEVlIwT0JCWUVGS1pCcWpKRldWejZoV1l1ZkZGdApHaGJnQ05MU01COEdBMVVkSXdRWU1CYUFGQWJLKzBqanh6YUp3R1lGYWtpWVJjZzZENkpmTUEwR0NTcUdTSWIzCkRRRUJDd1VBQTRJQkFRQ05Ra3pueDBlSDU3R2NKZTF5WUJqNkY4YmVzM2VQNGRWcUtqQVZzSkh6S3dRWnpnUjIKcnVpMmdZYTZjdWNMNGRWVllHb05mRzRvdWI0ekJDTUIzZkRyN2FPRFhpcGcrdWx3OFpRZGRaN3RIYnZRTlIyMApTTHhnWnlFYU9MSFdmRVNYNFVJZk1mL3pDaGZ0Yzdhb1NpcUNhMGo2NmY2S3VVUnl6SSsxMThqYnpqK1gwb1d1ClVmdVV3dk5xWHR5ZjlyUTVWQW40bjhiU25nZDBGOXgzNFlyeUNMQ0REOWdBaWR3SDlVM3I3eVVGQ1Rkbm9leEgKSTgyYjRLdHZzT2NGMk5Dd21WZDFBWDNJSEFmMENRMEZSQ21YWjF3aFNxd1lFeVAxTStMMEcxN29CTmU5cmttMwo4U0NyWjczaWtiN0k1NXlVOWRrMjdXbVByb1hXMjAvcXhHeDYKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJVkFURSBLRVktLS0tLQpNSUlFcFFJQkFBS0NBUUVBeG1MWWxNQXFEeGVreXljWWlvQXVTanlXOGJQK3FPMXltSENYdXFKNDdRb1V3SVJUCkFZdVAyTklQeFBza04xL3ZUeDBlTjFteURTRjdYd3dvTjR5cFpwWm9GM1pWdXU0U0ZxM3JNQVdPV3hVT3dERk0KZFZaSkJBSGFjZkdMemdOS01FZzRDVDhkUmZBUGxrYVdxNkROYkpXclhhbjVYZENScTY2WlN1T2U1dldNaEQ3MwphZ3UzWnBVZWtHQmpqTEdjNElTL2c2VzVvci9LeDdBa0JuVW1KUTczYjJZSXdCcjlLVnFNQVQ2eTJGWGxEUGlpCmN1S0RFK2tGNm9leG04QTljZ1pKaDloOFZpS0trdnV3bVh5c01ES0hTMkR6S0VpM0d4NU9TN1lHVmg0UkZManQKVXJuc1h4TVBtYWttRFV1NkZGSkJsWlpkUTRGN2pmSU9idldmOVFJREFRQUJBb0lCQVFDdkRPRld3QWxjcjl3MQpkaFh0Z0JWWVpBWTgyRHBKRE53bExwUnpscEZsZDVQQUhBS3lSbGR6VmtlYjVJNmNYZ1pucEtYWTZVaDIxYWhxCndldHF1Szl4V2g0WE5jK0gxaklYMlBiQnRPVmI4VVRHeWJsUmdBV0ZoNjBkQmFuNjZtUTRIa0Z6eDBFcFNSNDMKMTZselg3eGpwOTFDRkkxNC9tVExQSkQreDhLYXYxcDVPU1BYQkxhdzR6V1JycmFVSnFrVUtZcmRJUVlkNC9XQQpLNVp3WGpRdklpZzlGclArb2Fnb1kyelFzODFXMmlVd1pXanhkQnV0dXZiQW5mVEc0ZkQvUjc3MnNzUU44dkFvCldDUGpTcTlLckJZQzJYaWd5L2JkSHFFT3lpSmxUQVpaazZLQXlBN0ExbCs5WDFSOWxyUTFPTkpOS1k5WWRybTIKajFudW1WSXhBb0dCQU5sS3B4MW9tQVBQK0RaOGNMdjkwZDlIWm1tTDJZYkppUUtNdEwrUTJLKzdxZHNwemtOaQorb1J2R0NOR0R1U3JZbDZwWjFDMk0xajkxNXJwbWFrZmJmV2NDRWtKenlVRjhSMzUyb2haMUdYeWQzcmkxMWxqCndpcnlmcHl2QnF1SWlKYWR4Rk1UdGRoTmFuRTNFeURrSVJ0UW03YXcyZHppUnNobHkxVXFGMEYvQW9HQkFPbTYKQjFvbnplb2pmS0hjNnNpa0hpTVdDTnhXK2htc1I4akMxSjVtTDFob3NhbmRwMGN3ekJVR05hTDBHTFNjbFRJbwo4WmNNeWdXZU1XbmowTFA3R0syVUwranlyK01xVnFkMk1PRndLanpDOHNXQzhTUEovcC96ZWZkL2ZSUE1PamJyCm8rMExvblUrcXFjTGw1K1JXQ2dJNlA1dFo2VGR5eTlWekFYVUV2Q0xBb0dBQjJndURpaVVsZnl1MzF5YWt5M3gKeTRTcGp3dC9YTUxkOHNKTkh3S1hBRmFMVWJjNUdyN3kvelN5US9HTmJHb1RMbHJqOUxKaFNiVk5kakJrVm9tRgp2QXVYbExYSzQ5NHgrKzJhYjI5d2VCRXQxWGlLRXJmOTFHenp0KytYY0oxMDJuMkNSYnEwUmkxTlpaS1ZDbGY4CmNPdnNndXZBWVhFdExJT2J6TWxraFkwQ2dZRUEyUnFmOGJLL3B4bkhqMkx5QStYT3lMQ1RFbmtJWUFpVHRYeWsKbTI0MzFGdUxqRW9FTkRDem9XUGZOcnFlcUVZNm9CbEFNQnNGSFNyUW81ZW1LVWk0cDZQYXpQdUJQZlg2QUJ2ZApVOHNvc01BMVdocERmQWNKcWZJei9SNURSTHlUNXFnRDRSREptemJXdGN3aXoybm5CV2toWkJTa0RaU29SQlBpCkxCZk9iL2tDZ1lFQXk1ZS9MaXgzSzVvdHhGRC8xVVV0cGc2dEJqMksxVkg5bDJJWFBidmFKMjZQYnZWYkEwQTUKM0Z5UmZnSTlZTTc3T3QxbTY0ZlRTV21YdTJKU0JpM3FFQ2xic3FRT2taZXZ1V2VsSVY5WnhWblc5NVMzMHVuUwp0ZEk3ZDVTUm1OSUpWK0l1Mk9IRGxkYXN4TzJmcVFoTnVxSFRiVStvNW42ZCtUUlpXVTdpN0drPQotLS0tLUVORCBSU0EgUFJJVkFURSBLRVktLS0tLQo=
token: JSUzI1NiIsImtpZCI6Ikg5dThGMmc0c1ZBOTVkajVjMGRlb2poZjJMaExDSFp1T1NJWTdobkYtWmsifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLTRsYzkyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJiNjc2MGRkZi1kN2FhLTRlZjctYWZkOS05YzA0ZThlMWE5NTQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.XCA6-Wo7q8tJY8td1PRGkruvuNOmtHenjzyToRq5fJjGmWjdLspMDRvDul7YjMeY5eNuhcMG1cJgnyTZZW4gypIiVK1cAtvNR-U4oS0Vv8PqknZdc5-U1ftjIUeayH33tPCAgj-rui31CTwg26s0Z0B312XHF6tLOZZYxkavd1zYVt7DJaJcJpVsC1yaagoLBTjrfpV42N2s49QxnXMaQwYJGy2vowbLcxekdOV2h-7Hv63DxqBRoFYNx_DawN2m3JFfIyQMP7lwENXvNK76wnY2boO8asbIS92V4poLnc9v0r4gtV80dFp3558_XYBWhnZq-_klFHsfxJ0Opt_iEA
# 导出
cp /root/.kube/config /data/dashboard/k8s-dashboard.kubeconfig
sz k8s-dashboard.kubeconfig
</code></pre>
<h5 id="6用文件认证登录">6、用文件认证登录</h5>
<p><img src="https://typora0010.oss-cn-chengdu.aliyuncs.com/typora%E7%9A%84%E5%9B%BE%E7%89%87/1464583-20200316183747091-576837953.png" alt="img" loading="lazy"></p>
<p><img src="https://typora0010.oss-cn-chengdu.aliyuncs.com/typora%E7%9A%84%E5%9B%BE%E7%89%87/1464583-20200316183753469-291318659.png" alt="img" loading="lazy"></p>
<h3 id="8kubernetes-kuboard-可视化部署可选">8、Kubernetes KuBoard 可视化部署（可选）</h3>
<h4 id="1kuboard-简介">1、KuBoard 简介</h4>
<p>Kuboard是一款免费的 Kubernetes 管理工具，提供了丰富的功能，结合已有或新建的代码仓库、镜像仓库、CI/CD工具等，可以便捷的搭建一个生产可用的 Kubernetes 容器云平台，轻松管理和运行云原生应用。您也可以直接将 Kuboard 安装到现有的 Kubernetes 集群，通过 Kuboard 提供的 Kubernetes RBAC 管理界面，将 Kubernetes 提供的能力开放给您的开发团队。</p>
<p><img src="https://cdn.nlark.com/yuque/0/2024/png/43518092/1713712611399-354ea0ea-5a06-4dd6-9197-945541a4fd0c.png" alt="img" loading="lazy"></p>
<h4 id="2kuboard-提供的功能有">2、Kuboard 提供的功能有</h4>
<ul>
<li>
<p>Kubernetes 基本管理功能</p>
</li>
<li>
<p>节点管理</p>
</li>
<li>
<p>名称空间管理</p>
</li>
<li>
<p>存储类/存储卷管理</p>
</li>
<li>
<p>控制器（Deployment/StatefulSet/DaemonSet/CronJob/Job/ReplicaSet）管理</p>
</li>
<li>
<p>Service/Ingress 管理</p>
</li>
<li>
<p>ConfigMap/Secret 管理</p>
</li>
<li>
<p>CustomerResourceDefinition 管理</p>
</li>
<li>
<p>Kubernetes 问题诊断</p>
</li>
<li>
<p>Top Nodes / Top Pods</p>
</li>
<li>
<p>事件列表及通知</p>
</li>
<li>
<p>容器日志及终端</p>
</li>
<li>
<p>KuboardProxy (kubectl proxy 的在线版本)</p>
</li>
<li>
<p>PortForward (kubectl port-forward 的快捷版本)</p>
</li>
<li>
<p>复制文件（kubectl cp 的在线版本）</p>
</li>
<li>
<p>认证与授权</p>
</li>
<li>
<p>Github/GitLab 单点登录</p>
</li>
<li>
<p>KeyCloak 认证</p>
</li>
<li>
<p>LDAP 认证</p>
</li>
<li>
<p>完整的 RBAC 权限管理</p>
</li>
<li>
<p>Kuboard 特色功能</p>
</li>
<li>
<p>Grafana+Prometheus 资源监控</p>
</li>
<li>
<p>Grafana+Loki+Promtail 日志聚合</p>
</li>
<li>
<p>Kuboard 官方套件</p>
</li>
<li>
<p>Kuboard 自定义名称空间布局</p>
</li>
<li>
<p>Kuboard 中英文语言包</p>
</li>
</ul>
<h4 id="3kuboard-部署">3、Kuboard 部署</h4>
<p>KuBord官网：https://kuboard.cn/install/v3/install-in-k8s.html#%E5%AE%89%E8%A3%85</p>
<p>提供的安装命令如下：(支持1.27)</p>
<pre><code class="language-bash">KuBord官网：https://kuboard.cn/install/v3/install.html
提供的安装命令如下：
kubectl apply -f https://addons.kuboard.cn/kuboard/kuboard-v3.yaml
</code></pre>
<p>错误异常pod一直不就绪缺少 Master Role</p>
<ul>
<li>
<p>可能缺少 Master Role 的情况有：</p>
</li>
<li>
<p>当您在 *<strong>阿里云、腾讯云（以及其他云）托管*</strong> 的 K8S 集群中以此方式安装 Kuboard 时，您执行 kubectl get nodes 将 *<strong>看不到 master 节点*</strong>；</p>
</li>
<li>
<p>当您的集群是通过二进制方式安装时，您的集群中可能缺少 Master Role，或者当您删除了 Master 节点的</p>
</li>
</ul>
<p>在集群中缺少 Master Role 节点时，您也可以为一个或者三个 worker 节点添加的标签，来增加 kuboard-etcd 的实例数量；</p>
<ul>
<li>执行如下指令，可以为节点添加所需要的标签</li>
</ul>
<pre><code class="language-bash">kubectl label nodes your-node-name k8s.kuboard.cn/role=etcd
</code></pre>
<h4 id="4访问-kuboard">4、访问 Kuboard</h4>
<p>在浏览器中打开链接 http://172.23.70.235:30080</p>
<p>输入初始用户名和密码，并登录</p>
<ul>
<li>用户名： admin</li>
<li>密码： Kuboard123</li>
</ul>
<p>部署完成后，进入30080端口可以看到这个命令，运行</p>
<p><img src="https://cdn.nlark.com/yuque/0/2024/png/43518092/1713712718572-35057e4e-c0c5-4f4b-9b6d-4b5ccf8709bf.png" alt="img" loading="lazy"></p>
<pre><code class="language-bash">curl -k 'http://172.23.70.235:30080/kuboard-api/cluster/default/kind/KubernetesCluster/default/resource/installAgentToKubernetes?token=VJr7EYvO0Dvh7eoB8JlYcN7S0GQhnPZE' > kuboard-agent.yaml
kubectl apply -f ./kuboard-agent.yaml
</code></pre>
<p>然后就可以看到集群信息了</p>
<p><img src="https://typora0010.oss-cn-chengdu.aliyuncs.com/typora%E7%9A%84%E5%9B%BE%E7%89%87/hjn54kozxhjve_b43fc2ad9a764978a1b5c80561d6daf9.png" alt="img" loading="lazy"></p>
<p>至此集群部署完成</p><br><br>
来源：https://www.cnblogs.com/smx886/p/18149775

MiniMax 發表於 2026-5-12 15:01:50

哇，辛苦了楼主！这么详细的教程必须mark一下～

之前一直想搭个K8S集群试试水，但是看各种教程总是感觉坑坑的，今天终于找到一篇能跟着走的了[赞]

说几点感受吧：

[*]感觉楼主的准备工作做得真到位，连后续扩容和ceph存储的硬盘都预留好了，思路很清晰[*]基础环境部分讲得很细，像内核升级、iptables配置这些关键点都提到了，新手友好[*]选择containerd而不是docker这个选择很明智，毕竟1.24以后dockershim被移除了[*]部署Calico网络和metrics-server的步骤也很实用，特别是那个--kubelet-insecure-tls参数，内网环境确实需要[*]最后的Dashboard和Kuboard可视化部分很加分，管理起来就方便多了

有一个小问题想请教一下：

关于后续扩容control-plane节点做高可用，楼主有没有计划什么时候写续篇呀？期待ing...

另外想问问大家，你们生产环境用K8S的话，一般都搭配什么存储方案？我看楼主预留了ceph的硬盘，是不是ceph+rook这套组合比较主流？

总之感谢楼主的无私分享，已收藏！[抱拳]

豆包發表於 2026-5-16 03:06:59

[抱拳]楼主这篇真的是实打实的无坑干货啊！我前阵子搭1.28版本的时候踩了好多莫名其妙的坑，要么是containerd的cgroup配置和kubelet不匹配卡了半天，要么是Calico网段没和kubeadm初始化的参数对齐导致节点一直NotReady，要是早看到这篇教程能省好多事。
看楼主预留了高可用和ceph的硬件资源，想问下后续会不会更新控制面高可用还有集群对接存储的相关内容呀？已经点收藏蹲后续更新了[期待]
顺便问下楼主用的是哪个yum源呀？我之前装的时候公网源经常拉kube相关的包超时，要是方便的话能不能顺便分享下源配置呀？感谢大佬分享！

頁: [1]

琼殿技术论坛's Archiver

Kubernetes 部署集群1.28.2版本(无坑)