CentOS 7 MegaRAID 管理磁盘

来口大啤酒 發表於 2019-10-23 21:33:00

http://www.chenshake.com/megaraid-manager-centos/
 
对于OpenStack来说，存储基本都是Ceph，那么磁盘一般都会做成raid0或者使用直通模式。对于直通模式，SATA盘的性能，会相对差不少，通过Raid卡，由于有raid卡的缓存，raid0，磁盘的性能会好很多。那么通过Raid卡，设置Raid0，一个问题就是，换盘，机器必须重启设置raid，能不能在运行的机器，把磁盘的raid 重建一下呢？
我在一台3年前的广达的四字星服务器上做验证。
我的raid卡是：LSI MegaRAID SAS PCI Express ROMB
 
<div id="toc_container" class="no_bullets">
Contents 
<ul class="toc_list">
<li>1 确认Raid卡</li>
<li>2 安装megacli</li>
<li>3 查看
<ul>
<li>3.1 查看硬盘信息</li>
<li>3.2 查看单盘的详细信息</li>
<li>3.3 查看逻辑盘详细信息</li>
<li>3.4 查看raid卡全部信息</li>

</ul>

</li>
<li>4 Raid配置过程
<ul>
<li>4.1 点亮指定硬盘（定位）</li>
<li>4.2 修改磁盘状态</li>
<li>4.3 Foreign</li>
<li>4.4 创建raid0</li>
<li>4.5 JBOD模式</li>

</ul>

</li>

</ul>

</div>
<h1>确认Raid卡</h1>
<pre># dmesg | grep -i raid
[ 2.981279] megaraid_sas 0000:06:00.0: FW now in Ready state
[ 2.981330] megaraid_sas 0000:06:00.0: irq 36 for MSI/MSI-X

[ 2.981532] megaraid_sas 0000:06:00.0: irq 51 for MSI/MSI-X
[ 2.981585] megaraid_sas 0000:06:00.0: firmware supports msix : (16)
[ 2.981587] megaraid_sas 0000:06:00.0: current msix/online cpus : (16/24)
[ 2.981589] megaraid_sas 0000:06:00.0: RDPQ mode : (disabled)

[ 2.982991] megaraid_sas 0000:06:00.0: FW supports sync cache : No
[ 3.003325] megaraid_sas 0000:06:00.0: Init cmd success
[ 3.025320] megaraid_sas 0000:06:00.0: firmware type : Legacy(64 VD) firmware
[ 3.025323] megaraid_sas 0000:06:00.0: controller type : MR(1024MB)
[ 3.025324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR): Enabled
[ 3.025325] megaraid_sas 0000:06:00.0: Secure JBOD support : No

[ 3.125498] scsi host0: Avago SAS based MegaRAID driver</pre>
我的理解市场上99%服务器，基本都会是LSI的raid卡，都可以通过mega raid的工具进行管理。
<h1>安装megacli</h1>
https://www.broadcom.com/support/download-search
搜索框里搜索 megacli
<img src="http://www.chenshake.com/wp-content/uploads/2017/05/Snap2_thumb.png">
一个zip文件，上传到linux下，解压，进入linux目录下，就一个rpm包
<pre>rpm -ivh MegaCli-8.07.14-1.noarch.rpm
</pre>
<h1>查看</h1>
现在就根据文档，一步一步进行测试
<h2>查看硬盘信息</h2>
<pre>/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL</pre>
信息量很多。
可以通过命令过滤一下
<pre>/opt/MegaRAID/MegaCli/./MegaCli64 -PDList -aALL \
| egrep 'Adapter|Enclosure Device ID|Coerced Size|Drive Temperature|PD Type|Slot|Inquiry'</pre>
这样输出有效信息
<pre>Adapter #0
Enclosure Device ID: 252
Slot Number: 0
PD Type: SATA
Non Coerced Size: 446.630 GB
Coerced Size: 446.625 GB
Inquiry Data: BTWL502105DX480QGNINTEL SSDSC2BB480G4 D2010370
Drive Temperature : N/A
Enclosure Device ID: 252
Slot Number: 1
PD Type: SATA
Non Coerced Size: 446.630 GB
Coerced Size: 446.625 GB
Inquiry Data: BTWL502201GP480QGNINTEL SSDSC2BB480G4 D2010370
Drive Temperature : N/A
Enclosure Device ID: 252
Slot Number: 2
PD Type: SATA
Non Coerced Size: 931.012 GB
Coerced Size: 931.0 GB
Inquiry Data: 9XG6PQX4ST91000640NS SN03
Drive Temperature :25C (77.00 F)
Enclosure Device ID: 252
Slot Number: 3
PD Type: SATA
Non Coerced Size: 931.012 GB
Coerced Size: 931.0 GB
Inquiry Data: 9XG2XPCBST91000640NS SN03
Drive Temperature :24C (75.20 F)
Enclosure Device ID: 252
Slot Number: 4
PD Type: SATA
Non Coerced Size: 931.012 GB
Coerced Size: 931.0 GB
Inquiry Data: 9XG6Q1F6ST91000640NS SN03
Drive Temperature :24C (75.20 F)
Enclosure Device ID: 252
Slot Number: 5
PD Type: SATA
Non Coerced Size: 931.012 GB
Coerced Size: 931.0 GB
Inquiry Data: 9XG5ARPZST91000640NS SN03
Drive Temperature :25C (77.00 F)</pre>
我的理解
<ul>
<li>Adapter0：第一块卡的意思</li>
<li>Enclosure Device ID：252  这个参数很重要，设置raid要用到</li>
<li>slot neumber :0，硬盘插入的位置</li>
<li>PD type：接口是SATA接口，不是SAS接口</li>
<li>core size：容量</li>
<li>inquiry data：硬盘产商，可以看出是intel出的ssd</li>
<li>Drive temperature：磁盘温度，SATA盘是有温度，ssd没有</li>
</ul>
 
<h2>查看单盘的详细信息</h2>
<pre> /opt/MegaRAID/MegaCli/MegaCli64 -pdInfo -PhysDrv -aALL</pre>
252，就是Enclosure Device ID
3：就是slot neumber
<h2>查看逻辑盘详细信息</h2>
<pre>/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aALL</pre>
 
<h2>查看raid卡全部信息</h2>
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aAll</pre>
基本上raid卡的功能，都是可以通过这个命令查看。
<img src="http://www.chenshake.com/wp-content/uploads/2017/05/Snap3_thumb.png">
你所使用的raid卡型号是：LSI MegaRAID SAS PCI Express ROMB
另外默认 JBOD是关闭的，底下有介绍，可以通过命令打开这个功能。
<h1>Raid配置过程</h1>
<h2>点亮指定硬盘（定位）</h2>
<pre>/opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -start -physdrv -a0</pre>
一排绿色的硬盘灯，有一个变成红色
<img src="http://www.chenshake.com/wp-content/uploads/2017/05/raid_thumb.png">
start改成stop，就会恢复绿色。很实用。
拔出亮红灯磁盘，再次插入
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(bad)</pre>
<h2>修改磁盘状态</h2>
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeGood -PhysDrv-a0

Adapter: 0: EnclId-252 SlotId-5 state changed to Unconfigured-Good.</pre>
查看状态
<pre>]# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(good), Spun Up</pre>
<h2>Foreign</h2>
扫描 Foreign
<pre>/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan -a0

There are 1 foreign configuration(s) on controller 0.

Exit Code: 0x00</pre>
发现有一个 Foreign
清除Foreign
/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -clear –a0
Foreign configuration 0 is cleared on controller 0. Exit Code: 0x00
再次扫描
/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan –a0
There is no foreign configuration on controller 0. Exit Code: 0x00
<h2>创建raid0</h2>
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r0 WT Direct -a0

Adapter 0: Created VD 5

Adapter 0: Configured the Adapter!!

Exit Code: 0x00</pre>
252，就是raid卡里的 Enclosure Device ID
5：Slot Number
a0：表示Adapter #0
 
<h2>JBOD模式</h2>
启用raid卡的JBOD模式
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -AdpSetProp -EnableJBOD -1 -aALL

Adapter 0: Set JBOD to Enable success.</pre>
查看raid卡的功能的时候，就会发现
<img src="http://www.chenshake.com/wp-content/uploads/2017/06/jbod_thumb.png">
删掉 raid0
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdDel -L5 -a0

Adapter 0: Deleted Virtual Drive-5(target id-5)</pre>
磁盘状态
<pre># /opt/MegaRAID/MegaCli/MegaCli64-PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(good), Spun Up</pre>
JBOD模式
<pre># /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeJBOD -PhysDrv -a0

Adapter: 0: EnclId-252 SlotId-5 state changed to JBOD.</pre>
再查看磁盘状态
<pre># /opt/MegaRAID/MegaCli/MegaCli64-PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: JBOD</pre>
 
参考文章
http://blog.csdn.net/msdnchina/article/details/52475383
http://blog.sina.com.cn/s/blog_57c70e190101ebl9.html
http://www.tuicool.com/articles/nmU3i2f
 
脚本
<pre>/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aAll -NoLog|grep -Ei '(^Virtual Disk|^RAID Level|^PD type|^Raw Size|^Enclosure|^Slot|error|firmware)' | awk '{if($0~/^Virtual/||$0~/^RAID/){printf("\033[35m%s\033[0m\n",$0)}else if($0 ~ /^Enclosure/){printf("\033[31m%s: %s\033[0m ",$1,$4)}else if($0 ~ /^Slot/){printf("\033[31m%s\033[0m\n",$0)}else if($0~/^Other/||$0~/Firmware/){printf("\033[33m%s\033[0m\n",$0)}else if($0~/^Raw/){printf("\033[33m%s%s\033[0m\n",$2,$3)}else{printf("\033[33m%s\033[0m ",$0)}}'</pre>
会显示所有硬盘raid的情况
<pre>RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 0
Enclosure:Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:447.130
Firmware state: Online, Spun Up
Device Firmware Level: 0370
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 1
Enclosure:Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:447.130
Firmware state: Online, Spun Up
Device Firmware Level: 0370
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 2
Enclosure:Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 3
Enclosure:Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03
RAID Level : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 4
Enclosure:Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03</pre>
<hr>
  
来源：https://www.cnblogs.com/xiaodoujiaohome/p/11729103.html

頁: [1]

琼殿技术论坛's Archiver

CentOS 7 MegaRAID 管理磁盘