企业存储技术 · 2020年08月24日

Multipath切换分布式存储iSCSI主路径

本文转载自一位专家朋友Bean Li的博客,原文链接见文末。

前言

本文记录multipath相关指令, Multipath的path path\_grouping\_policy – Paths are grouped into path groups. The policy determines how path groups are formed. There are five different policies.

  • failover: One path per priority group
  • multibus: All paths in one priority group. This is the default.
  • group\_by\_serial: One priority group per storage controller (serial number)
  • group\_by\_prio: One priority group per priority value
  • group\_by\_node\_name: One priority group per target node name

有时候使用failover,需要切换主路径,本文记录切换主路径的方法

查看当前配置

root@host243:~# multipath -ll2c2c5d201aba719ee dm-0 Bigtera ,VirtualStor_Scalsize=5.0T features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 49:0:0:0 sdg 8:96 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 48:0:0:0 sdf 8:80 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 46:0:0:0 sdd 8:48 active ready running`-+- policy='round-robin 0' prio=1 status=enabled  `- 47:0:0:0 sde 8:64 active ready running

灰色部分按住可左右拖动,以下同

切换主路径

========

从上面的输出可以看出,sdd/sde/sdf/sdg 都是同一个设备的4条不同的路径。如果设备是分布式存储提供的iSCSI设备,每条路径可能对应一个Gateway。有时候可能需要切换主路径,比如某个存储节点要关机维护,这种情况,要怎么做呢?

首先执行如下语句,进入交互页面:

multipathd -k 

通过如下指令可以实时查看某个multipath的多个group信息:

multipathd> show topology2c2c5d201aba719ee dm-0 Bigtera ,VirtualStor_Scalsize=5.0T features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 49:0:0:0 sdg 8:96 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 48:0:0:0 sdf 8:80 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 46:0:0:0 sdd 8:48 active ready running`-+- policy='round-robin 0' prio=1 status=enabled  `- 47:0:0:0 sde 8:64 active ready runningmultipathd>

对于我这个环境比较简单,如果存在多个,可以通过如下指令指定:

multipathd> show map 2c2c5d201aba719ee topology2c2c5d201aba719ee dm-0 Bigtera ,VirtualStor_Scalsize=5.0T features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=active| `- 49:0:0:0 sdg 8:96 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 48:0:0:0 sdf 8:80 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 46:0:0:0 sdd 8:48 active ready running`-+- policy='round-robin 0' prio=1 status=enabled  `- 47:0:0:0 sde 8:64 active ready runningmultipathd>

从上面的输出可以看出,我们设备下有多条路径,第一条sdg 为group 1,第二个sdf为group 2, 如果想切换主路径到sdd,那么需要执行:

image.png

执行完毕后,可以执行quit 退出multipathd的交互,然后执行multipath -ll来确认主路径是否发生了变化。

有些时候,没有I/O流量的情况下,multipath -ll的输出并不会及时发生变化,不过没关系,引入一些IO,就可以看到active 路径发生了变化。

oot@host243:~# multipath -ll2c2c5d201aba719ee dm-0 Bigtera ,VirtualStor_Scalsize=5.0T features='0' hwhandler='0' wp=rw|-+- policy='round-robin 0' prio=1 status=enabled| `- 49:0:0:0 sdg 8:96 active ready running|-+- policy='round-robin 0' prio=1 status=enabled| `- 48:0:0:0 sdf 8:80 active ready running|-+- policy='round-robin 0' prio=1 status=active| `- 46:0:0:0 sdd 8:48 active ready running`-+- policy='round-robin 0' prio=1 status=enabled  `- 47:0:0:0 sde 8:64 active ready running

验证

对于分布式存储的iSCSI,每一条路径对应某一个存储Gateway,如何判断sdX和存储Gateway的映射关系呢?

root@host243:~# iscsiadm  -m session -P 3iSCSI Transport Class version 2.0-870version 2.0-873Target: iqn.2020-01.lenovo:test    Current Portal: 10.16.172.105:3260,1    Persistent Portal: 10.16.172.105:3260,1        **********        Interface:        **********        Iface Name: default        Iface Transport: tcp        Iface Initiatorname: iqn.1993-08.org.debian:01:f077d70d498        Iface IPaddress: 10.16.172.243        Iface HWaddress: <empty>        Iface Netdev: <empty>        SID: 16        iSCSI Connection State: LOGGED IN        iSCSI Session State: LOGGED_IN        Internal iscsid Session State: NO CHANGE        *********        Timeouts:        *********        Recovery Timeout: 120        Target Reset Timeout: 30        LUN Reset Timeout: 30        Abort Timeout: 15        *****        CHAP:        *****        username: <empty>        password: ********        username_in: <empty>        password_in: ********        ************************        Negotiated iSCSI params:        ************************        HeaderDigest: None        DataDigest: None        MaxRecvDataSegmentLength: 262144        MaxXmitDataSegmentLength: 1048576        FirstBurstLength: 262144        MaxBurstLength: 1048576        ImmediateData: Yes        InitialR2T: No        MaxOutstandingR2T: 1        ************************        Attached SCSI devices:        ************************        Host Number: 46    State: running        scsi46 Channel 00 Id 0 Lun: 0            Attached scsi disk sdd        State: running    Current Portal: 10.16.172.106:3260,1    Persistent Portal: 10.16.172.106:3260,1        **********        Interface:        **********        Iface Name: default        Iface Transport: tcp        Iface Initiatorname: iqn.1993-08.org.debian:01:f077d70d498        Iface IPaddress: 10.16.172.243        Iface HWaddress: <empty>        Iface Netdev: <empty>        SID: 17        iSCSI Connection State: LOGGED IN        iSCSI Session State: LOGGED_IN        Internal iscsid Session State: NO CHANGE        *********        Timeouts:        *********        Recovery Timeout: 120        Target Reset Timeout: 30        LUN Reset Timeout: 30        Abort Timeout: 15        *****        CHAP:        *****        username: <empty>        password: ********        username_in: <empty>        password_in: ********        ************************        Negotiated iSCSI params:        ************************        HeaderDigest: None        DataDigest: None        MaxRecvDataSegmentLength: 262144        MaxXmitDataSegmentLength: 1048576        FirstBurstLength: 262144        MaxBurstLength: 1048576        ImmediateData: Yes        InitialR2T: No        MaxOutstandingR2T: 1        ************************        Attached SCSI devices:        ************************        Host Number: 47    State: running        scsi47 Channel 00 Id 0 Lun: 0            Attached scsi disk sde        State: running    Current Portal: 10.16.172.107:3260,1    Persistent Portal: 10.16.172.107:3260,1        **********        Interface:        **********        Iface Name: default        Iface Transport: tcp        Iface Initiatorname: iqn.1993-08.org.debian:01:f077d70d498        Iface IPaddress: 10.16.172.243        Iface HWaddress: <empty>        Iface Netdev: <empty>        SID: 18        iSCSI Connection State: LOGGED IN        iSCSI Session State: LOGGED_IN        Internal iscsid Session State: NO CHANGE        *********        Timeouts:        *********        Recovery Timeout: 120        Target Reset Timeout: 30        LUN Reset Timeout: 30        Abort Timeout: 15        *****        CHAP:        *****        username: <empty>        password: ********        username_in: <empty>        password_in: ********        ************************        Negotiated iSCSI params:        ************************        HeaderDigest: None        DataDigest: None        MaxRecvDataSegmentLength: 262144        MaxXmitDataSegmentLength: 1048576        FirstBurstLength: 262144        MaxBurstLength: 1048576        ImmediateData: Yes        InitialR2T: No        MaxOutstandingR2T: 1        ************************        Attached SCSI devices:        ************************        Host Number: 48    State: running        scsi48 Channel 00 Id 0 Lun: 0            Attached scsi disk sdf        State: running    Current Portal: 10.16.172.108:3260,1    Persistent Portal: 10.16.172.108:3260,1        **********        Interface:        **********        Iface Name: default        Iface Transport: tcp        Iface Initiatorname: iqn.1993-08.org.debian:01:f077d70d498        Iface IPaddress: 10.16.172.243        Iface HWaddress: <empty>        Iface Netdev: <empty>        SID: 19        iSCSI Connection State: LOGGED IN        iSCSI Session State: LOGGED_IN        Internal iscsid Session State: NO CHANGE        *********        Timeouts:        *********        Recovery Timeout: 120        Target Reset Timeout: 30        LUN Reset Timeout: 30        Abort Timeout: 15        *****        CHAP:        *****        username: <empty>        password: ********        username_in: <empty>        password_in: ********        ************************        Negotiated iSCSI params:        ************************        HeaderDigest: None        DataDigest: None        MaxRecvDataSegmentLength: 262144        MaxXmitDataSegmentLength: 1048576        FirstBurstLength: 262144        MaxBurstLength: 1048576        ImmediateData: Yes        InitialR2T: No        MaxOutstandingR2T: 1        ************************        Attached SCSI devices:        ************************        Host Number: 49    State: running        scsi49 Channel 00 Id 0 Lun: 0            Attached scsi disk sdg        State: runningroot@host243:~#

通过上面的输出,可以看到 /dev/sdd对应的是10.16.172.105:3260 ,因此找到了对应关系。

修改完毕了主路径,如何从网路上判定已经发生了切换呢?

iftop -i bond0

通过iftop,查看网络流向,确定是否流向了正确的Gateway。

image.png

点开后双击放大显示

原文链接:https://bean-li.github.io/mul...

推荐阅读

本文转载自企业存储技术微信公众号原文链接点这里

注:本文只代表作者个人观点,与任何组织机构无关,如有错误和不足之处欢迎在留言中批评指正。 进一步交流技术可以加我的微信/QQ:490834312。
尊重知识,转载时请保留全文,并包括本行及如下二维码。感谢您的阅读和支持!《企业存储技术》微信公众号:HL_Storage,也欢迎关注企业存储技术极术专栏,定期更新。
42.jpg

推荐阅读
关注数
5555
内容数
236
关注存储、服务器、图形工作站、AI硬件等方面技术。WeChat:490834312
目录
极术微信服务号
关注极术微信号
实时接收点赞提醒和评论通知
安谋科技学堂公众号
关注安谋科技学堂
实时获取安谋科技及 Arm 教学资源
安谋科技招聘公众号
关注安谋科技招聘
实时获取安谋科技中国职位信息