May 312017
 

对于OpenStack来说,存储基本都是Ceph,那么磁盘一般都会做成raid0或者使用直通模式。对于直通模式,SATA盘的性能,会相对差不少,通过Raid卡,由于有raid卡的缓存,raid0,磁盘的性能会好很多。那么通过Raid卡,设置Raid0,一个问题就是,换盘,机器必须重启设置raid,能不能在运行的机器,把磁盘的raid 重建一下呢?

我在一台3年前的广达的四字星服务器上做验证。

我的raid卡是:LSI MegaRAID SAS PCI Express ROMB

确认Raid卡

# dmesg | grep -i raid 
[    2.981279] megaraid_sas 0000:06:00.0: FW now in Ready state
[    2.981330] megaraid_sas 0000:06:00.0: irq 36 for MSI/MSI-X

[    2.981532] megaraid_sas 0000:06:00.0: irq 51 for MSI/MSI-X
[    2.981585] megaraid_sas 0000:06:00.0: firmware supports msix        : (16)
[    2.981587] megaraid_sas 0000:06:00.0: current msix/online cpus      : (16/24)
[    2.981589] megaraid_sas 0000:06:00.0: RDPQ mode     : (disabled)

[    2.982991] megaraid_sas 0000:06:00.0: FW supports sync cache        : No
[    3.003325] megaraid_sas 0000:06:00.0: Init cmd success
[    3.025320] megaraid_sas 0000:06:00.0: firmware type : Legacy(64 VD) firmware
[    3.025323] megaraid_sas 0000:06:00.0: controller type       : MR(1024MB)
[    3.025324] megaraid_sas 0000:06:00.0: Online Controller Reset(OCR)  : Enabled
[    3.025325] megaraid_sas 0000:06:00.0: Secure JBOD support   : No

[    3.125498] scsi host0: Avago SAS based MegaRAID driver

我的理解市场上99%服务器,基本都会是LSI的raid卡,都可以通过mega raid的工具进行管理。

安装megacli

https://www.broadcom.com/support/download-search

搜索框里搜索 megacli

Snap2

一个zip文件,上传到linux下,解压,进入linux目录下,就一个rpm包

rpm -ivh MegaCli-8.07.14-1.noarch.rpm

查看

现在就根据文档,一步一步进行测试

查看硬盘信息

/opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL

信息量很多。

可以通过命令过滤一下

/opt/MegaRAID/MegaCli/./MegaCli64 -PDList -aALL \
| egrep 'Adapter|Enclosure Device ID|Coerced Size|Drive Temperature|PD Type|Slot|Inquiry'

这样输出有效信息

Adapter #0
Enclosure Device ID: 252
Slot Number: 0
PD Type: SATA
Non Coerced Size: 446.630 GB [0x37d436b0 Sectors]
Coerced Size: 446.625 GB [0x37d40000 Sectors]
Inquiry Data: BTWL502105DX480QGN  INTEL SSDSC2BB480G4                     D2010370
Drive Temperature : N/A
Enclosure Device ID: 252
Slot Number: 1
PD Type: SATA
Non Coerced Size: 446.630 GB [0x37d436b0 Sectors]
Coerced Size: 446.625 GB [0x37d40000 Sectors]
Inquiry Data: BTWL502201GP480QGN  INTEL SSDSC2BB480G4                     D2010370
Drive Temperature : N/A
Enclosure Device ID: 252
Slot Number: 2
PD Type: SATA
Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
Coerced Size: 931.0 GB [0x74600000 Sectors]
Inquiry Data:             9XG6PQX4ST91000640NS                            SN03    
Drive Temperature :25C (77.00 F)
Enclosure Device ID: 252
Slot Number: 3
PD Type: SATA
Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
Coerced Size: 931.0 GB [0x74600000 Sectors]
Inquiry Data:             9XG2XPCBST91000640NS                            SN03    
Drive Temperature :24C (75.20 F)
Enclosure Device ID: 252
Slot Number: 4
PD Type: SATA
Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
Coerced Size: 931.0 GB [0x74600000 Sectors]
Inquiry Data:             9XG6Q1F6ST91000640NS                            SN03    
Drive Temperature :24C (75.20 F)
Enclosure Device ID: 252
Slot Number: 5
PD Type: SATA
Non Coerced Size: 931.012 GB [0x74606db0 Sectors]
Coerced Size: 931.0 GB [0x74600000 Sectors]
Inquiry Data:             9XG5ARPZST91000640NS                            SN03    
Drive Temperature :25C (77.00 F)

我的理解

  • Adapter0:第一块卡的意思
  • Enclosure Device ID:252  这个参数很重要,设置raid要用到
  • slot neumber :0,硬盘插入的位置
  • PD type:接口是SATA接口,不是SAS接口
  • core size:容量
  • inquiry data:硬盘产商,可以看出是intel出的ssd
  • Drive temperature:磁盘温度,SATA盘是有温度,ssd没有

 

查看单盘的详细信息

 /opt/MegaRAID/MegaCli/MegaCli64 -pdInfo -PhysDrv[252:3] -aALL

252,就是Enclosure Device ID

3:就是slot neumber

查看逻辑盘详细信息

/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aALL

 

查看raid卡全部信息

# /opt/MegaRAID/MegaCli/MegaCli64 -AdpAllInfo -aAll

基本上raid卡的功能,都是可以通过这个命令查看。

Snap3

你所使用的raid卡型号是:LSI MegaRAID SAS PCI Express ROMB

另外默认 JBOD是关闭的,底下有介绍,可以通过命令打开这个功能。

Raid配置过程

点亮指定硬盘(定位)

/opt/MegaRAID/MegaCli/MegaCli64 -PdLocate -start -physdrv[252:5] -a0

一排绿色的硬盘灯,有一个变成红色

raid

start改成stop,就会恢复绿色。很实用。

拔出亮红灯磁盘,再次插入

# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(bad)

修改磁盘状态

# /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeGood -PhysDrv[252:5]-a0
                                     
Adapter: 0: EnclId-252 SlotId-5 state changed to Unconfigured-Good.

查看状态

]# /opt/MegaRAID/MegaCli/MegaCli64 -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(good), Spun Up

Foreign

扫描 Foreign

/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan -a0
                                     
There are 1 foreign configuration(s) on controller 0.

Exit Code: 0x00

发现有一个 Foreign

清除Foreign

/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -clear –a0

Foreign configuration 0 is cleared on controller 0. Exit Code: 0x00

再次扫描

/opt/MegaRAID/MegaCli/MegaCli64 -cfgforeign -scan –a0

There is no foreign configuration on controller 0. Exit Code: 0x00

创建raid0

# /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r0[252:5] WT Direct -a0 
                                     
Adapter 0: Created VD 5

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

252,就是raid卡里的 Enclosure Device ID

5:Slot Number

a0:表示Adapter #0

 

JBOD模式

启用raid卡的JBOD模式

# /opt/MegaRAID/MegaCli/MegaCli64 -AdpSetProp -EnableJBOD -1 -aALL
                                     
Adapter 0: Set JBOD to Enable success.

查看raid卡的功能的时候,就会发现

jbod

删掉 raid0

# /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdDel -L5 -a0
                                     
Adapter 0: Deleted Virtual Drive-5(target id-5)

磁盘状态

# /opt/MegaRAID/MegaCli/MegaCli64  -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Unconfigured(good), Spun Up

JBOD模式

# /opt/MegaRAID/MegaCli/MegaCli64 -PDMakeJBOD -PhysDrv[252:5] -a0 
                                     
Adapter: 0: EnclId-252 SlotId-5 state changed to JBOD.

再查看磁盘状态

# /opt/MegaRAID/MegaCli/MegaCli64  -PDList -aALL -Nolog|grep '^Firm'
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: Online, Spun Up
Firmware state: JBOD

 

参考文章

http://blog.csdn.net/msdnchina/article/details/52475383

http://blog.sina.com.cn/s/blog_57c70e190101ebl9.html

http://www.tuicool.com/articles/nmU3i2f

 

脚本

/opt/MegaRAID/MegaCli/MegaCli64 -LdPdInfo -aAll -NoLog|grep -Ei '(^Virtual Disk|^RAID Level|^PD type|^Raw Size|^Enclosure|^Slot|error|firmware)' | awk '{if($0~/^Virtual/||$0~/^RAID/){printf("\033[35m%s\033[0m\n",$0)}else if($0 ~ /^Enclosure/){printf("\033[31m%s: %s\033[0m ",$1,$4)}else if($0 ~ /^Slot/){printf("\033[31m%s\033[0m\n",$0)}else if($0~/^Other/||$0~/Firmware/){printf("\033[33m%s\033[0m\n",$0)}else if($0~/^Raw/){printf("\033[33m%s%s\033[0m\n",$2,$3)}else{printf("\033[33m%s\033[0m ",$0)}}'

会显示所有硬盘raid的情况

RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 0
Enclosure:  Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:447.130
Firmware state: Online, Spun Up
Device Firmware Level: 0370
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 1
Enclosure:  Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:447.130
Firmware state: Online, Spun Up
Device Firmware Level: 0370
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 2
Enclosure:  Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 3
Enclosure:  Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03
RAID Level          : Primary-0, Secondary-0, RAID Level Qualifier-0
Enclosure: 252 Slot Number: 4
Enclosure:  Media Error Count: 0 Other Error Count: 0
PD Type: SATA Size:931.512
Firmware state: Online, Spun Up
Device Firmware Level: SN03

 Leave a Reply

(required)

(required)