Ubuntu18.04上部署Ceph(基于debian-pacific）
<p><span style="font-size: 18px"><strong>Ceph基础：</strong></span></p><div><strong><span style="font-family: 宋体">1.Ceph 是一个开源的分布式存储系统，同时支持对象存储、块设备、文件系统.</span></strong></div>
<div><strong><span style="font-family: 宋体">2.ceph 是一个对象(object)式存储系统，它把每一个待管理的数据流(文件等数据)切分为一到多个固定大小(默认 4 兆)的对象数据，并以其为原子单元(原子是构成元素的最小单元)完成数据的读写。</span></strong></div>
<div><strong><span style="font-family: 宋体">3.对象数据的底层存储服务是由多个存储主机(host)组成的存储集群，该集群也被称之为RADOS(reliable automatic distributed object store)存储集群，即可靠的、自动化的、分布式的对象存储系统。</span></strong></div>
<div><strong><span style="font-family: 宋体">4.librados 是 RADOS 存储集群的 API，支持 C/C++/JAVA/python/ruby/php/go等编程语言客户端。</span></strong></div>
<div> </div>
<div> </div>
<div><span style="font-size: 18px"><strong><span style="font-family: 宋体">Ceph涉及组件以及作用：</span></strong></span></div>
<div> </div>
<div>
<p> <span style="font-family: 宋体"><strong>1. Ceph监视器(ceph-mon)：用于维护集群状态映射(maintains maps of the cluster state)，比如 ceph 集群中有多少存储池、每个存储池有多少 PG 以及存储池和 PG的映射关系等，monitor map, manager map, the OSD map, the MDS map, and theCRUSH map，这些映射是 Ceph 守护程序相互协调所需的关键群集状态，此外监视器还负责管理守护程序和客户端之间的身份验证(认证使用 cephX 协议)。通常至少需要三个监视器才能实现冗余和高可用性</strong></span></p>
<p><span style="font-family: 宋体"><strong> 2. Ceph管理器(ceph-mgr)：ceph manager守护进程负责跟踪运行时指标和ceph集群当前的状态，包括存储利用率，当前性能指标和系统负载等，ceph-mgr还托管一些python模块，以实现基于web的ceph仪表盘和rest api，通常要实现高可用至少需要2个ceph-mgr进程，通常ceph-mon和ceph-mgr个数相同，1个ceph-mon同节点会伴随1个ceph-mgr守护进程。</strong></span></p>
<p><span style="font-family: 宋体"><strong> 3. Ceph对象存储(ceph-osd)：提供存储数据，操作系统上的一个磁盘就是一个 OSD 守护程序，OSD 用于处理 ceph集群数据复制，恢复，重新平衡，并通过检查其他 Ceph OSD 守护程序的心跳来向 Ceph监视器和管理器提供一些监视信息。通常至少需要 3 个 Ceph OSD 才能实现冗余和高可用性。</strong></span></p>
<p><span style="font-family: 宋体"><strong> 4. Ceph元数据服务(ceph-mds)：Ceph MDS为ceph文件系统存储元数据，注意Ceph块设备和对象存储不用mds存储元数据，Ceph MDS允许POSIX文件系统用户执行基本命令，而不会将压力都集中到Ceph OSD集群上，通常mds可以选择部署至少2个节点，可以和其他组件一起也可以分开。</strong></span></p>
<p> </p>
<p>基于以上，本次部署采用Ubuntu 18.04.5-server版本，Ceph采用pacific版</p>
<p>环境准备：计划采用9台虚拟机机分开部署，1台用于部署，3台监视，2台管理，最后3台存储节点，2个网段，一个用cluster，一个用于public</p>
<p>Cluster网段：172.16.1.0/24</p>
<p>Public网段： 192.168.79.0/24</p>
<p> </p>
<p>具体规划如下：</p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818172528371-685407326.png"></p>
<p> </p>
<p> </p>
<p><span style="font-size: 14pt"><strong><span style="color: rgba(255, 0, 0, 1)">基础配置：</span></strong></span>（这里可以先初始化一台模板，配置完后在克隆）：</p>
<p> </p>
<p><span style="color: rgba(255, 0, 0, 1)"><strong>1.更换源，这里采用清华镜像地址：</strong></span></p>
<div> wget -q -O- 'https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc' | apt-key add -</div>
<div>
<div> </div>
<div>Ubuntu18.04.x:</div>
<div> </div>
<div>cat /etc/apt/sources.list</div>
<div> </div>
<div># 默认注释了源码镜像以提高 apt update 速度，如有需要可自行取消注释</div>
<div>deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse</div>
<div># deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic main restricted universe multiverse</div>
<div>deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse</div>
<div># deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-updates main restricted universe</div>
<div>multiverse</div>
<div>deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse</div>
<div># deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-backports main restricted universe</div>
<div>multiverse</div>
<div>deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe multiverse</div>
<div># deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu/ bionic-security main restricted universe</div>
<div>multiverse</div>
<div>deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main</div>
<div># echo "deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic main"</div>
<div>>> /etc/apt/sources.list </div>
<div> </div>
<div>更换完成后，执行apt-get update && apt-get upgrade</div>
</div>
<p> </p>
<p> </p>
<p><span style="color: rgba(255, 0, 0, 1)"><strong>2.时间同步：</strong></span></p>
<p>apt update</p>
<p>apt install chrony -y</p>
<div> vim /etc/chrony/chrony.conf</div>
<div># 修改为阿⾥云时钟同步服务器</div>
<div># 公⽹</div>
<div>server ntp.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp2.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp3.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp4.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp5.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp6.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div>server ntp7.aliyun.com minpoll 4 maxpoll 10 iburst</div>
<div># 重启服务</div>
<div>systemctl restart chrony</div>
<div>systemctl status chrony</div>
<div>systemctl enable chrony</div>
<div># 查看时钟同步状态</div>
<div> timedatectl status</div>
<div># 写⼊系统时钟</div>
<div>hwclock -w</div>
<div># 重启服务器</div>
<div> reboot</div>
<div> </div>
<div><strong><span style="color: rgba(255, 0, 0, 1)">3.更改IP地址</span></strong></div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818173612005-1732513441.png">
<p> </p>
<p> 配置完成后，执行netplan apply </p>
<p><span style="color: rgba(255, 0, 0, 1)"><strong>4.添加一个Ceph账户并给与权限，后期OSD挂载可能涉及</strong></span></p>
<div>groupadd -r -g 2022 ceph && useradd -r -m-s /bin/bash -u 2022 -g 2022 ceph && echo ceph:123456 | chpasswd</div>
<div>
<div>echo "ceph ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers</div>
</div>
<p><strong><span style="color: rgba(255, 0, 0, 1)"> 5.配置/etc/hosts主机解析</span></strong></p>
<p><strong><span style="color: rgba(255, 0, 0, 1)"><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818174211706-391392136.png"></span></strong></p>
<p> </p>
<p> </p>
<p> 在上面全部配置完成后，按规划地址克隆其他服务器，并更改IP</p>
<p> </p>
<p><span style="font-size: 18px"><strong><span style="color: rgba(255, 0, 0, 1)">集群部署：</span></strong></span></p>
<p><strong><span style="color: rgba(255, 0, 0, 1)">1.配置免秘钥登录</span>：</strong></p>
<div>在 ceph-deploy 节点配置允许以非交互的方式登录到各节点</div>
<div> </div>
<div>root@ceph-deploy:~# ssh-keygen -t rsa</div>
<div>root@ceph-deploy:~# ssh-copy-id 172.16.1.161</div>
</div>
<div>root@ceph-deploy:~# ssh-copy-id 172.16.1.162</div>
<div>略（这里可以写脚本，批量执行）</div>
<div> </div>
<div><span style="color: rgba(255, 0, 0, 1)"><strong>2.安装 ceph 部署工具：</strong></span></div>
<div> </div>
<div>
<div>root@ceph-deploy:~# apt-cache madison ceph-deploy</div>
<div>回显如下：</div>
<div>ceph-deploy | 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main amd64 Packages</div>
<div>ceph-deploy | 2.0.1 | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific bionic/main i386 Packages</div>
<div>ceph-deploy |1.5.38-0ubuntu1 | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe amd64 Packages</div>
<div>ceph-deploy | 1.5.38-0ubuntu | https://mirrors.tuna.tsinghua.edu.cn/ubuntu bionic/universe i386 Packages </div>
</div>
<p><strong><span style="background-color: rgba(136, 136, 136, 1); color: rgba(255, 0, 0, 1)">3.初始化 mon 节点：</span></strong></p>
<div># mkdir ceph-cluster </div>
<div># cd ceph-cluster/</div>
<div> </div>
<div>
<div>初始化 mon 节点过程如下:</div>
<div>Ubuntu 各服务器需要单独安装 Python2：</div>
<div>
<div>root@ceph-mon1:~# apt install python2.7 -y</div>
<div>root@ceph-mon1:~# ln -sv /usr/bin/python2.7 /usr/bin/python2</div>
<div> </div>
<div>完成后执行：ceph-deploy new --cluster-network 172.16.1.0/24 --public-network 192.168.79.0/24 ceph-mon1.example.com</div>
</div>
</div>
<p>验证初始化：</p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818175522889-994847581.png"></p>
<p> </p>
<p> </p>
<p> </p>
<p><strong><span style="color: rgba(255, 0, 0, 1)">4.初始化 ceph 存储节点：</span></strong></p>
<div># ceph-deploy install --no-adjust-repos --nogpgcheck ceph-node1 ceph-node2 ceph-node3</div>
<div> </div>
<div><strong><span style="color: rgba(255, 0, 0, 1)">5.配置 mon 节点并生成及同步秘钥（注意：在mon节点执行):</span></strong></div>
<div> </div>
<div>
<div>root@ceph-mon1:~# apt install ceph-mon</div>
<div>root@ceph-mon2:~# apt install ceph-mon</div>
<div>root@ceph-mon3:~# apt install ceph-mon</div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818180028317-1063231119.png">
<p> </p>
<p> </p>
<p> </p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818180123103-1411319027.png"></p>
<p> </p>
<p> </p>
<p>一路回车，3台mon节点一样操作</p>
<p>完成后，在deploy部署节点，执行</p>
<div># ceph-deploy mon create-initial</div>
<div> </div>
<div>在mon节点验证：</div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818180511330-2120325649.png">
<p> </p>
<p> </p>
</div>
<div> </div>
<div><strong><span style="color: rgba(255, 0, 0, 1)">6.分发 admin 秘钥：</span></strong></div>
<div> </div>
<div>
<div>root@ceph-deploy:~# apt install ceph-common #先安装 ceph 的公共组件</div>
<div>root@ceph-node1:~# apt install ceph-common -y</div>
<div>root@ceph-node2:~# apt install ceph-common -y</div>
<div>root@ceph-node3:~# apt install ceph-common -y</div>
<div>root@ceph-node4:~# apt install ceph-common -y</div>
<div> </div>
<div>
<div>root@ceph-deploy:/home/ceph/ceph-cluster# ceph-deploy admin ceph-node1 ceph-node2 ceph-node3</div>
<div> </div>
<div>如下图：</div>
</div>
</div>
<p> <img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818180724015-1656248747.png"></p>
<p> </p>
<p>node节点验证：</p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818180844976-144062354.png"></p>
<p> </p>
<p> </p>
<p> <span style="color: rgba(255, 0, 0, 1)"><strong>7.配置 manager 节点：</strong></span></p>
</div>
</div>
<div>
<div>初始化 ceph-mgr 节点：</div>
<div>#apt install ceph-mgr</div>
<div> </div>
<div>2台mgr节点一样操作：</div>
<div>
<div> </div>
<div>完成后执行：</div>
<div> </div>
<div># ceph-deploy mgr create ceph-mgr1</div>
</div>
</div>
<div> </div>
<p>mgr节点验证：</p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818181133183-275177841.png"></p>
<p> </p>
<p> </p>
<p><strong><span style="color: rgba(255, 0, 0, 1)">8.ceph-deploy 管理 ceph 集群：</span></strong></p>
<div>在 ceph-deploy 节点配置一下系统环境，以方便后期可以执行 ceph 管理命令。</div>
<p> </p>
<div>root@ceph-deploy:~# apt install ceph-common</div>
<div>$ ceph-deploy admin ceph-deploy #推送正证书给自己 </div>
<div> </div>
<div>执行：root@ceph-deploy:~/ceph-cluster# ceph -s</div>
<div> </div>
<div>若显示有如下所示：</div>
<div> </div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818181506959-802889204.png">
<p> </p>
<p> </p>
<p>执行：root@ceph-deploy:~/ceph-cluster# ceph config set mon auth_allow_insecure_global_id_reclaim false</p>
<p> </p>
<p><span style="color: rgba(255, 0, 0, 1)"><strong>9:准备 OSD 节点：</strong></span></p>
<div>擦除磁盘之前通过 deploy节点对 node 节点执行安装 ceph 基本运行环境</div>
<div> </div>
<div># ceph-deploy install --release pacific ceph-node1</div>
<div># ceph-deploy install --release pacific ceph-node2</div>
<div># ceph-deploy install --release pacific ceph-node3</div>
</div>
<p> </p>
<p>查看列出磁盘信息：</p>
<div># ceph-deploy disk list ceph-node1 #列出远端存储 node1 节点的磁盘信息 </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818182132955-60968489.png">
<p> </p>
<p> </p>
<div>使用 ceph-deploy disk zap 擦除各 ceph node 的 ceph 数据磁盘：</div>
<div>
<p>#ceph-deploy disk zap ceph-node1 /dev/sdb<br>#ceph-deploy disk zap ceph-node1 /dev/sdc <br>#ceph-deploy disk zap ceph-node1 /dev/sdd</p>
<p> </p>
<p>注：所有node节点执行同样操作</p>
<p> </p>
<p><strong><span style="color: rgba(255, 0, 0, 1)">10.添加OSD</span></strong></p>
<p>在部署节点执行：</p>
<p>ceph-deploy osd create ceph-node1 --data /dev/sdb<br>ceph-deploy osd create ceph-node1 --data /dev/sdc<br>ceph-deploy osd create ceph-node1 --data /dev/sdd</p>
<p>ceph-deploy osd create ceph-node2 --data /dev/sdb<br>ceph-deploy osd create ceph-node2 --data /dev/sdc<br>ceph-deploy osd create ceph-node2 --data /dev/sdd</p>
<p>ceph-deploy osd create ceph-node3 --data /dev/sdb<br>ceph-deploy osd create ceph-node3 --data /dev/sdc<br>ceph-deploy osd create ceph-node3 --data /dev/sdd</p>
<p> </p>
<p>在node 节点验证：</p>
<p> </p>
<p><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818182658767-1148243367.png"></p>
<p> </p>
<p> </p>
<p><span style="color: rgba(255, 0, 0, 1)"><strong>11.扩容mod节点：</strong></span></p>
<p> </p>
<div>
<div># ceph-deploy mon add ceph-mon2 </div>
<div># ceph-deploy mon add ceph-mon3</div>
<div> </div>
<div><strong><span style="color: rgba(255, 0, 0, 1)">12.扩容mgr节点：</span></strong></div>
<div> </div>
<div>
<div># ceph-deploy mgr create ceph-mgr2</div>
<div># ceph-deploy admin ceph-mgr2 #同步配置文件到ceph-mg2 节点 </div>
<div> </div>
<div><strong><span style="color: rgba(255, 0, 0, 1)">13.最后验证：</span></strong></div>
<div> </div>
<div> </div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818183207366-1014966829.png">
<p> </p>
<p> </p>
<p>至此，集群部署完成</p>
<p> </p>
<p> </p>
<p>客户端验证：</p>
<p>准备一台 centos的虚拟机，配置好Yum源，</p>
<div>在客户端安装 ceph-common:</div>
<p>执行：</p>
<div>yum install https://mirrors.aliyun.com/ceph/rpm-octopus/el7/noarch/ceph-release-1-1.el7.noarch.rpm -y</div>
<div>
<div>yum install ceph-common</div>
<div> </div>
<div>完成后，从ceph部署节点执行scp配合和验证文件</div>
<div> </div>
<div>
<div># scp ceph.conf ceph.client.admin.keyring root@172.16.1.XX:/etc/ceph/</div>
<div> </div>
<div> </div>
<div> </div>
<div>Ceph集群块设备 RBD准备：</div>
<div> </div>
<div>
<div>创建 RBD：</div>
<div> </div>
<div>
<div>#ceph osd pool create myrbd1 64 64 #创建存储池,指定 pg 和 pgp 的数量，pgp 是对存在于 pg 的数据进行组合存储，pgp 通常等于 pg 的值</div>
<div>pool 'myrdb1' created</div>
<div># ceph osd pool --help</div>
<div> </div>
<div># ceph osd pool application enable myrbd1 rbd #对存储池启用 RBD 功能</div>
<div>enabled application 'rbd' on pool 'myrdb1'</div>
<div> </div>
<div># rbd pool init -p myrbd1 #通过 RBD 命令对存储池初始化</div>
</div>
</div>
</div>
</div>
<div> </div>
<div>创建并验证 img：</div>
<div> </div>
<div>
<div>#rbd create myimg1 --size 5G --pool myrbd1</div>
<div> </div>
<div># rbd create myimg2 --size 3G --pool myrbd1 --image-format 2 --image-feature layering</div>
<div>后续步骤会使用 myimg2 ，由于 centos 系统内核较低无法挂载使用，因此只开启部分特</div>
<div>性。除了 layering 其他特性需要高版本内核支持</div>
<div> </div>
<div>#rbd ls --pool myrbd1 #列出指定的 pool 中所有的 img</div>
<div>myimg1</div>
<div>myimg2 </div>
</div>
<p> <img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818184105406-939837384.png"></p>
<p> </p>
<p> </p>
<p>完成上述操作后：</p>
<div>客户端映射 img：</div>
<div> </div>
<div>在cenos上执行：</div>
<div> </div>
<div>
<div># rbd -p myrbd1 map myimg2</div>
<div> </div>
<div>出现如下所示：</div>
<div> </div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>rbd: sysfs write failed</div>
<div>RBD image feature set mismatch. You can disable features unsupported by the kernel</div>
<div>with "rbd feature disable myrdb1/myimg1 object-map fast-diff deep-flatten".</div>
<div>In some cases useful info is found in syslog - try "dmesg | tail".</div>
<div>rbd: map failed: (6) No such device or address</div>
<div> </div>
<div>执行括号中的rbd feature disable myrdb1/myimg1 object-map fast-diff deep-flatten</div>
<div> </div>
<div>再执行 lsblk查看</div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818184514056-1335700762.png">
<p> </p>
<p> 已发现rbd0</p>
<p>创建一个挂载目录test，并对rbd0执行mkfs.xfs /dev/rdb0格式化 ，并挂载到test目录下</p>
<p> </p>
<p>写入一个文件测试：</p>
<div>dd if=/dev/zero of=/test/ceph-test-file bs=1MB count=300</div>
<div>300+0 records in</div>
<div>300+0 records out</div>
<div>300000000 bytes (300 MB) copied, 3.49849 s, 85.8 MB/s </div>
<div> </div>
<div> </div>
<div> </div>
<div>最后到ceph端验证：</div>
<div> </div>
<div><img src="https://img2020.cnblogs.com/blog/2509111/202108/2509111-20210818184851254-821574818.png">
<p> </p>
<p> </p>
</div>
<p>可以看到容量已经成功完成，完</p>
</div>
</div>
</div>
<div> </div>
<p> </p>
<p> </p>
<p> </p>
</div>
<div> </div><br><br>
来源：https://www.cnblogs.com/N56041/p/15158348.html
頁:
[1]