kubernetes搭建rook-ceph

你我他我你他你他我我中有你 發表於 2020-4-22 14:38:00

<h2>简介</h2>
<ul>
<li>Rook官网：https://rook.io</li>
<li>Rook是云原生计算基金会(CNCF)的孵化级项目.</li>
<li>Rook是Kubernetes的开源云本地存储协调器，为各种存储解决方案提供平台，框架和支持，以便与云原生环境本地集成。</li>
<li>至于CEPH，官网在这：https://ceph.com/</li>
<li>ceph官方提供的helm部署，至今我没成功过，所以转向使用rook提供的方案</li>
</ul>
<hr>
有道笔记原文：http://note.youdao.com/noteshare?id=281719f1f0374f787effc90067e0d5ad&sub=0B59EA339D4A4769B55F008D72C1A4C0
<h2>环境</h2>
<pre><code class="hljs nginx">centos 7.5
kernel 4.18.7-1.el7.elrepo.x86_64

docker 18.06

kubernetes v1.12.2
kubeadm部署：
 网络: canal
 DNS: coredns
集群成员：
192.168.1.1 kube-master
192.168.1.2 kube-node1
192.168.1.3 kube-node2
192.168.1.4 kube-node3
192.168.1.5 kube-node4

所有node节点准备一块200G的磁盘：/dev/sdb
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/dc39b28c879faacbc5cf5ff609c9107d.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<hr>
<h2>准备工作</h2>
<ul>
<li>所有节点开启ip_forward</li>
</ul>
<pre><code class="hljs javascript">cat <<EOF >/etc/sysctl.d/ceph.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
</code></pre>
<h2>开始部署Operator</h2>
<ul>
<li>部署Rook Operator</li>
</ul>
<pre><code class="hljs bash">#无另外说明，全部操作都在master操作

cd $HOME
git clone https://github.com/rook/rook.git

cd rook
cd cluster/examples/kubernetes/ceph
kubectl apply -f operator.yaml
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/41b9054db47da8f0055203dff825a995.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>查看Operator的状态</li>
</ul>
<pre><code class="hljs shell">#执行apply之后稍等一会。
#operator会在集群内的每个主机创建两个pod:rook-discover,rook-ceph-agent

kubectl -n rook-ceph-system get pod -o wide
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/c523cccf4e6ce93e477dffa4333f474e.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<h2>给节点打标签</h2>
<ul>
<li>运行ceph-mon的节点打上：ceph-mon=enabled</li>
</ul>
<pre><code class="hljs">kubectl label nodes {kube-node1,kube-node2,kube-node3} ceph-mon=enabled
</code></pre>
<ul>
<li>运行ceph-osd的节点，也就是存储节点，打上：ceph-osd=enabled</li>
</ul>
<pre><code class="hljs">kubectl label nodes {kube-node1,kube-node2,kube-node3} ceph-osd=enabled
</code></pre>
<ul>
<li>运行ceph-mgr的节点，打上：ceph-mgr=enabled</li>
</ul>
<pre><code class="hljs nginx">#mgr只能支持一个节点运行，这是ceph跑k8s里的局限
kubectl label nodes kube-node1 ceph-mgr=enabled
</code></pre>
<hr>
<h2>配置cluster.yaml文件</h2>
<ul>
<li>
官方配置文件详解：https://rook.io/docs/rook/v0.8/ceph-cluster-crd.html
</li>
<li>
文件中有几个地方要注意：
<ul>
<li>dataDirHostPath: 这个路径是会在宿主机上生成的，保存的是ceph的相关的配置文件，再重新生成集群的时候要确保这个目录为空，否则mon会无法启动</li>
<li>useAllDevices: 使用所有的设备，建议为false，否则会把宿主机所有可用的磁盘都干掉</li>
<li>useAllNodes：使用所有的node节点，建议为false，肯定不会用k8s集群内的所有node来搭建ceph的</li>
<li>databaseSizeMB和journalSizeMB：当磁盘大于100G的时候，就注释这俩项就行了</li>
</ul>
</li>
<li>本次实验用到的 cluster.yaml 文件内容如下：</li>
</ul>
<pre><code class="hljs makefile">apiVersion: v1
kind: Namespace
metadata:
name: rook-ceph
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: rook-ceph-cluster
namespace: rook-ceph
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-cluster
namespace: rook-ceph
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: [ "get", "list", "watch", "create", "update", "delete" ]
---
# Allow the operator to create resources in this cluster's namespace
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-cluster-mgmt
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: rook-ceph-cluster-mgmt
subjects:
- kind: ServiceAccount
name: rook-ceph-system
namespace: rook-ceph-system
---
# Allow the pods in this namespace to work with configmaps
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: rook-ceph-cluster
namespace: rook-ceph
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: rook-ceph-cluster
subjects:
- kind: ServiceAccount
name: rook-ceph-cluster
namespace: rook-ceph
---
apiVersion: ceph.rook.io/v1beta1
kind: Cluster
metadata:
name: rook-ceph
namespace: rook-ceph
spec:
cephVersion:
# The container image used to launch the Ceph daemon pods (mon, mgr, osd, mds, rgw).
# v12 is luminous, v13 is mimic, and v14 is nautilus.
# RECOMMENDATION: In production, use a specific version tag instead of the general v13 flag, which pulls the latest release and could result in different
# versions running within the cluster. See tags available at https://hub.docker.com/r/ceph/ceph/tags/.
image: ceph/ceph:v13
# Whether to allow unsupported versions of Ceph. Currently only luminous and mimic are supported.
# After nautilus is released, Rook will be updated to support nautilus.
# Do not set to true in production.
allowUnsupported: false
# The path on the host where configuration files will be persisted. If not specified, a kubernetes emptyDir will be created (not recommended).
# Important: if you reinstall the cluster, make sure you delete this directory from each host or else the mons will fail to start on the new cluster.
# In Minikube, the '/data' directory is configured to persist across reboots. Use "/data/rook" in Minikube environment.
dataDirHostPath: /var/lib/rook
# The service account under which to run the daemon pods in this cluster if the default account is not sufficient (OSDs)
serviceAccount: rook-ceph-cluster
# set the amount of mons to be started
# count可以定义ceph-mon运行的数量，这里默认三个就行了
mon:
count: 3
allowMultiplePerNode: true
# enable the ceph dashboard for viewing cluster status
# 开启ceph资源面板
dashboard:
enabled: true
# serve the dashboard under a subpath (useful when you are accessing the dashboard via a reverse proxy)
# urlPrefix: /ceph-dashboard
network:
# toggle to use hostNetwork
# 使用宿主机的网络进行通讯
# 使用宿主机的网络貌似可以让集群外的主机挂载ceph
# 但是我没试过，有兴趣的兄弟可以试试改成true
# 反正这里只是集群内用，我就不改了
hostNetwork: false
# To control where various services will be scheduled by kubernetes, use the placement configuration sections below.
# The example under 'all' would have all services scheduled on kubernetes nodes labeled with 'role=storage-node' and
# tolerate taints with a key of 'storage-node'.
placement:
# all:
# nodeAffinity:
# requiredDuringSchedulingIgnoredDuringExecution:
# nodeSelectorTerms:
# - matchExpressions:
# - key: role
# operator: In
# values:
# - storage-node
# podAffinity:
# podAntiAffinity:
# tolerations:
# - key: storage-node
# operator: Exists
# The above placement information can also be specified for mon, osd, and mgr components
# mon:
# osd:
# mgr:
# nodeAffinity：通过选择标签的方式，可以限制pod被调度到特定的节点上
# 建议限制一下，为了让这几个pod不乱跑
mon:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: ceph-mon
 operator: In
 values:
 - enabled
osd:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: ceph-osd
 operator: In
 values:
 - enabled
mgr:
 nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 nodeSelectorTerms:
 - matchExpressions:
 - key: ceph-mgr
 operator: In
 values:
 - enabled
resources:
# The requests and limits set here, allow the mgr pod to use half of one CPU core and 1 gigabyte of memory
# mgr:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# The above example requests/limits can also be added to the mon and osd components
# mon:
# osd:
storage: # cluster level storage configuration and selection
useAllNodes: false
useAllDevices: false
deviceFilter:
location:
config:
 # The default and recommended storeType is dynamically set to bluestore for devices and filestore for directories.
 # Set the storeType explicitly only if it is required not to use the default.
 # storeType: bluestore
 # databaseSizeMB: "1024" # this value can be removed for environments with normal sized disks (100 GB or larger)
 # journalSizeMB: "1024"# this value can be removed for environments with normal sized disks (20 GB or larger)
# Cluster level list of directories to use for storage. These values will be set for all nodes that have no `directories` set.
# directories:
# - path: /rook/storage-dir
# Individual nodes and their config can be specified as well, but 'useAllNodes' above must be set to false. Then, only the named
# nodes below will be used as storage resources.Each node's 'name' field should match their 'kubernetes.io/hostname' label.
#建议磁盘配置方式如下：
#name: 选择一个节点，节点名字为kubernetes.io/hostname的标签，也就是kubectl get nodes看到的名字
#devices: 选择磁盘设置为OSD
# - name: "sdb":将/dev/sdb设置为osd
nodes:
- name: "kube-node1"
 devices:
 - name: "sdb"
- name: "kube-node2"
 devices:
 - name: "sdb"
- name: "kube-node3"
 devices:
 - name: "sdb"

# directories: # specific directories to use for storage can be specified for each node
# - path: "/rook/storage-dir"
# resources:
# limits:
# cpu: "500m"
# memory: "1024Mi"
# requests:
# cpu: "500m"
# memory: "1024Mi"
# - name: "172.17.4.201"
# devices: # specific devices to use for storage can be specified for each node
# - name: "sdb"
# - name: "sdc"
# config: # configuration can be specified at the node level which overrides the cluster level config
# storeType: filestore
# - name: "172.17.4.301"
# deviceFilter: "^sd."
</code></pre>
<hr>
<h2>开始部署ceph</h2>
<ul>
<li>部署ceph</li>
</ul>
<pre><code class="hljs shell">kubectl apply -f cluster.yaml

# cluster会在rook-ceph这个namesapce创建资源
# 盯着这个namesapce的pod你就会发现，它在按照顺序创建Pod

kubectl -n rook-ceph get pod -o wide-w

# 看到所有的pod都Running就行了
# 注意看一下pod分布的宿主机，跟我们打标签的主机是一致的

kubectl -n rook-ceph get pod -o wide
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/f0e39ce2ee1c9f687d41f4bd5bc8f16d.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/69d15615aa1a940cf0784f2ee668b2a6.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>
切换到其他主机看一下磁盘
<ul>
<li>切换到kube-node1</li>

</ul>
<pre><code class="hljs nginx">lsblk
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/d4efd22425aacb97be9ccc40e7aba7eb.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>切换到kube-node3</li>
</ul>
<pre><code class="hljs nginx">lsblk
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/0df25a701a25ce331dc1c4bffbc8551c.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
</li>
</ul>
<hr>
<h2>配置ceph dashboard</h2>
<ul>
<li>看一眼dashboard在哪个service上</li>
</ul>
<pre><code class="hljs cs">kubectl -n rook-ceph get service
#可以看到dashboard监听了8443端口</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/abcee244f68221daaa8cddcf1829d4cf.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>创建个nodeport类型的service以便集群外部访问</li>
</ul>
<pre><code class="hljs bash">kubectl apply -f dashboard-external-https.yaml

# 查看一下nodeport在哪个端口
ss -tanl
kubectl -n rook-ceph get service
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/06673d74bb98605e8dd3abe2c4573b5d.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>找出Dashboard的登陆账号和密码</li>
</ul>
<pre><code class="hljs perl">MGR_POD=`kubectl get pod -n rook-ceph | grep mgr | awk '{print $1}'`

kubectl -n rook-ceph logs $MGR_POD | grep password
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/da2f614ca8ed50849221dd4a1b7aff59.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>打开浏览器输入任意一个Node的IP+nodeport端口</li>
<li>这里我的就是：https://192.168.1.2:30290</li>
</ul>
<img src="https://s4.51cto.com/images/blog/201811/21/dee19f2f6e617da2bf364926c1103782.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/a35d26acf5558b84080054557e5148c3.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<h2>配置ceph为storageclass</h2>
<ul>
<li>官方给了一个样本文件：storageclass.yaml</li>
<li>这个文件使用的是 RBD 块存储</li>
<li>pool创建详解：https://rook.io/docs/rook/v0.8/ceph-pool-crd.html</li>

</ul>
<pre><code class="hljs makefile">apiVersion: ceph.rook.io/v1beta1
kind: Pool
metadata:
#这个name就是创建成ceph pool之后的pool名字
name: replicapool
namespace: rook-ceph
spec:
replicated:
size: 1
# size 池中数据的副本数,1就是不保存任何副本
failureDomain: osd
#failureDomain：数据块的故障域，
#值为host时，每个数据块将放置在不同的主机上
#值为osd时，每个数据块将放置在不同的osd上
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: ceph
# StorageClass的名字，pvc调用时填的名字
provisioner: ceph.rook.io/block
parameters:
pool: replicapool
# Specify the namespace of the rook cluster from which to create volumes.
# If not specified, it will use `rook` as the default namespace of the cluster.
# This is also the namespace where the cluster will be
clusterNamespace: rook-ceph
# Specify the filesystem type of the volume. If not specified, it will use `ext4`.
fstype: xfs
# 设置回收策略默认为：Retain
reclaimPolicy: Retain
</code></pre>
<ul>
<li>创建StorageClass</li>
</ul>
<pre><code class="hljs css">kubectl apply -f storageclass.yaml
kubectl get storageclasses.storage.k8s.io-n rook-ceph
kubectl describe storageclasses.storage.k8s.io-n rook-ceph
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/6f4d3762b4d0a0e21ea67b7eea737c99.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<hr>
<ul>
<li>创建个nginx pod尝试挂载</li>
</ul>
<pre><code class="hljs makefile">cat << EOF > nginx.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
 storage: 1Gi
storageClassName: ceph

---
apiVersion: v1
kind: Service
metadata:
name: nginx
spec:
selector:
app: nginx
ports:
- port: 80
name: nginx-port
targetPort: 80
protocol: TCP

---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
 app: nginx
template:
metadata:
 name: nginx
 labels:
 app: nginx
spec:
 containers:
 - name: nginx
 image: nginx
 ports:
 - containerPort: 80
 volumeMounts:
 - mountPath: /html
 name: http-file
 volumes:
 - name: http-file
 persistentVolumeClaim:
 claimName: nginx-pvc
EOF

kubectl apply -f nginx.yaml
</code></pre>
<ul>
<li>查看pv,pvc是否创建了</li>
</ul>
<pre><code class="hljs cs">kubectl get pv,pvc

# 看一下nginx这个pod也运行了
kubectl get pod
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/e07d945929885dd10f3dcada40657d84.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>删除这个pod,看pv是否还存在</li>
</ul>
<pre><code class="hljs bash">kubectl delete -f nginx.yaml

kubectl get pv,pvc
# 可以看到，pod和pvc都已经被删除了，但是pv还在！！！</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/9740e830de83b8b6c8e40bdea7c038b8.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<hr>
<h2>添加新的OSD进入集群</h2>
<ul>
<li>这次我们要把node4添加进集群，先打标签</li>
</ul>
<pre><code class="hljs nginx">kubectl label nodes kube-node4 ceph-osd=enabled
</code></pre>
<ul>
<li>重新编辑cluster.yaml文件</li>
</ul>
<pre><code class="hljs bash"># 原来的基础上添加node4的信息

cd $HOME/rook/cluster/examples/kubernetes/ceph/
vi cluster.yam
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/4000fcb2a526ff78712c9d90d8f9081e.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>apply一下cluster.yaml文件</li>
</ul>
<pre><code class="hljs cs">kubectl apply -f cluster.yaml

# 盯着rook-ceph名称空间,集群会自动添加node4进来

kubectl -n rook-ceph get pod -o wide -w
kubectl -n rook-ceph get pod -o wide</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/b10cc4916b5d6261edcbf5367b295f44.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/8fa7185e702e4e737f05ad6699b3bd2c.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/24fe158f3f8f1467b9d19c095883e49e.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>去node4节点看一下磁盘</li>

</ul>
<pre><code class="hljs">lsblk</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/2fe00707e1cd7ed0a2309e33d5603b97.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>再打开dashboard看一眼</li>
</ul>
<img src="https://s4.51cto.com/images/blog/201811/21/adde245fc7ef089eda38a6792362a90b.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<h2>删除一个节点</h2>
<ul>
<li>去掉node3的标签</li>
</ul>
<pre><code class="hljs nginx">kubectl label nodes kube-node3 ceph-osd-
</code></pre>
<ul>
<li>重新编辑cluster.yaml文件</li>
</ul>
<pre><code class="hljs bash"># 删除node3的信息

cd $HOME/rook/cluster/examples/kubernetes/ceph/
vi cluster.yam
</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/a0ee991748c532cdea5308c0b5fdc44d.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph">
<ul>
<li>apply一下cluster.yaml文件</li>
</ul>
<pre><code class="hljs cs">kubectl apply -f cluster.yaml

# 盯着rook-ceph名称空间

kubectl -n rook-ceph get pod -o wide -w
kubectl -n rook-ceph get pod -o wide

# 最后记得删除宿主机的/var/lib/rook文件夹</code></pre>
<img src="https://s4.51cto.com/images/blog/201811/21/fe125b4a9b96dfa8ef82fe4c2662aed6.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/5b18f8c85dd64c2d2e7619d1a4a7f692.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/e53735f8137ba44c429426f48ee55298.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> <img src="https://s4.51cto.com/images/blog/201811/21/534252c6295f4de75b575d58d616f2ef.png?x-oss-process=image/watermark,size_16,text_QDUxQ1RP5Y2a5a6i,color_FFFFFF,t_100,g_se,x_10,y_10,shadow_90,type_ZmFuZ3poZW5naGVpdGk=" alt="kubernetes搭建rook-ceph"> 
来源：https://www.cnblogs.com/lvcisco/p/12751947.html

頁: [1]

琼殿技术论坛's Archiver

kubernetes搭建rook-ceph