Chapter 24.  Online Storage Management

24.1. Target Setup
24.1.1. Install and Run targetcli
24.1.2. Create a Backstore
24.1.3. Create an iSCSI Target
24.1.4. Configure an iSCSI Portal
24.1.5. Configure LUNs
24.1.6. Configure ACLs
24.1.7. Fibre Channel over Ethernet (FCoE) Target Setup
24.1.8. Remove objects with targetcli
24.1.9. targetcli References
24.2. Creating an iSCSI Initiator
24.3. Fibre Channel
24.3.1. Fibre Channel API
24.3.2. Native Fibre Channel Drivers and Capabilities
24.4. Configuring a Fibre Channel over Ethernet Interface
24.5. Configuring an FCoE Interface to Automatically Mount at Boot
24.6. iSCSI
24.6.1. iSCSI API
24.7. Persistent Naming
24.7.1. /dev/sd* and their Major and Minor Numbers
24.7.2. WWID
24.7.3. Device Names Managed by the udev mechanism (/dev/disk/by-*)
24.8. Removing a Storage Device
24.9. Removing a Path to a Storage Device
24.10. Adding a Storage Device or Path
24.11. Scanning Storage Interconnects
24.12. iSCSI Discovery Configuration
24.13. Configuring iSCSI Offload and Interface Binding
24.13.1. Viewing Available iface Configurations
24.13.2. Configuring an iface for Software iSCSI
24.13.3. Configuring an iface for iSCSI Offload
24.13.4. Binding/Unbinding an iface to a Portal
24.14. Scanning iSCSI Interconnects
24.15. Logging in to an iSCSI Target
24.16. Resizing an Online Logical Unit
24.16.1. Resizing Fibre Channel Logical Units
24.16.2. Resizing an iSCSI Logical Unit
24.16.3. Updating the Size of Your Multipath Device
24.16.4. Changing the Read/Write State of an Online Logical Unit
24.17. Adding/Removing a Logical Unit Through rescan-scsi-bus.sh
24.18. Modifying Link Loss Behavior
24.18.1. Fibre Channel
24.18.2. iSCSI Settings With dm-multipath
24.18.3. iSCSI Root
24.19. Controlling the SCSI Command Timer and Device Status
24.20. Online Storage Configuration Troubleshooting
24.21. Configuring Maximum Time for Error Recovery with eh_deadline
It is often desirable to add, remove or re-size storage devices while the operating system is running, and without rebooting. This chapter outlines the procedures that may be used to reconfigure storage devices on Red Hat Enterprise Linux 7 host systems while the system is running. It covers iSCSI and Fibre Channel storage interconnects; other interconnect types may be added it the future.
This chapter focuses on adding, removing, modifying, and monitoring storage devices. It does not discuss the Fibre Channel or iSCSI protocols in detail. For more information about these protocols, refer to other documentation.
This chapter makes reference to various sysfs objects. Red Hat advises that the sysfs object names and directory structure are subject to change in major Red Hat Enterprise Linux releases. This is because the upstream Linux kernel does not provide a stable internal API. For guidelines on how to reference sysfs objects in a transportable way, refer to the document /usr/share/doc/kernel-doc-version/Documentation/sysfs-rules.txt in the kernel source tree for guidelines.

Warning

Online storage reconfiguration must be done carefully. System failures or interruptions during the process can lead to unexpected results. Red Hat advises that you reduce system load to the maximum extent possible during the change operations. This will reduce the chance of I/O errors, out-of-memory errors, or similar errors occurring in the midst of a configuration change. The following sections provide more specific guidelines regarding this.
In addition, Red Hat recommends that you back up all data before reconfiguring online storage. 

24.1. Target Setup

Red Hat Enterprise Linux 7 uses the targetcli shell as a front end for viewing, editing, and saving the configuration of the Linux-IO Target without the need to manipulate the kernel target's configuration files directly. The targetcli tool is a command-line interface that allows an administrator to export local storage resources, which are backed by either files, volumes, local SCSI devices, or RAM disks, to remote systems. The targetcli tool has a tree-based layout, includes built-in tab completion, and provides full auto-complete support and inline documentation.
The hierarchy of targetcli does not always match the kernel interface exactly because targetcli is simplified where possible.

Important

To ensure that the changes made in targetcli are persistent, start and enable the target service:
# systemctl start target
# systemctl enable target

24.1.1. Install and Run targetcli

To install targetcli, use:
# yum install targetcli
Start the target service:
# systemctl start target
Configure target to start at boot time:
# systemctl enable target
Use the targetcli command, and then use the ls command for the layout of the tree interface:
# targetcli
:
/> ls
o- /........................................[...]
  o- backstores.............................[...]
  | o- block.................[Storage Objects: 0]           
  | o- fileio................[Storage Objects: 0]       
  | o- pscsi.................[Storage Objects: 0]         
  | o- ramdisk...............[Storage Ojbects: 0]          
  o- iscsi...........................[Targets: 0]   
  o- loopback........................[Targets: 0]

Note

In Red Hat Enterprise Linux 7.0, using the targetcli command from Bash, for example, targetcli iscsi/ create, does not work and does not return an error. Starting with Red Hat Enterprise Linux 7.1, an error status code is provided to make using targetcli with shell scripts more useful.

24.1.2. Create a Backstore

Backstores enable support for different methods of storing an exported LUN's data on the local machine. Creating a storage object defines the resources the backstore uses.

Note

In Red Hat Enterprise Linux 6, the term 'backing-store' is used to refer to the mappings created. However, to avoid confusion between the various ways 'backstores' can be used, in Red Hat Enterprise Linux 7 the term 'storage objects' refers to the mappings created and 'backstores' is used to describe the different types of backing devices.
The backstore devices that LIO supports are:
FILEIO (Linux file-backed storage)
FILEIO storage objects can support either write_back or write_thru operation. The write_back enables the local file system cache. This improves performance but increases the risk of data loss. It is recommended to use write_back=false to disable write_back in favor of write_thru.
To create a fileio storage object, run the command /backstores/fileio create file_name file_location file_size write_back=false. For example:
/> /backstores/fileio create file1 /tmp/disk1.img 200M write_back=false
Created fileio file1 with size 209715200
BLOCK (Linux BLOCK devices)
The block driver allows the use of any block device that appears in the /sys/block to be used with LIO. This includes physical devices (for example, HDDs, SSDs, CDs, DVDs) and logical devices (for example, software or hardware RAID volumes, or LVM volumes).

Note

BLOCK backstores usually provide the best performance.
To create a BLOCK backstore using the /dev/sdb block device, use the following command:
/> /backstores/block create name=block_backend dev=/dev/sdb
Generating a wwn serial.
Created block storage object block_backend using /dev/sdb.
PSCSI (Linux pass-through SCSI devices)
Any storage object that supports direct pass-through of SCSI commands without SCSI emulation, and with an underlying SCSI device that appears with lsscsi in /proc/scsi/scsi (such as a SAS hard drive) can be configured as a backstore. SCSI-3 and higher is supported with this subsystem.

Warning

PSCSI should only be used by advanced users. Advanced SCSI commands such as for Aysmmetric Logical Unit Assignment (ALUAs) or Persistent Reservations (for example, those used by VMware ESX, and vSphere) are usually not implemented in the device firmware and can cause malfunctions or crashes. When in doubt, use BLOCK for production setups instead.
To create a PSCSI backstore for a physical SCSI device, a TYPE_ROM device using /dev/sr0 in this example, use:
/> backstores/pscsi/ create name=pscsi_backend dev=/dev/sr0
Generating a wwn serial.
Created pscsi storage object pscsi_backend using /dev/sr0
Memory Copy RAM disk (Linux RAMDISK_MCP)
Memory Copy RAM disks (ramdisk) provide RAM disks with full SCSI emulation and separate memory mappings using memory copy for initiators. This provides capability for multi-sessions and is particularly useful for fast, volatile mass storage for production purposes.
To create a 1GB RAM disk backstore, use the following command:
/> backstores/ramdisk/ create name=rd_backend size=1GB
Generating a wwn serial.
Created rd_mcp ramdisk rd_backend with size 1GB.

24.1.3. Create an iSCSI Target

To create an iSCSI target:

Procedure 24.1. Create an iSCSI target

  1. Run targetcli.
  2. Move into the iSCSI configuration path:
    /> iscsi/

    Note

    The cd command is also accepted to change directories, as well as simply listing the path to move into.
  3. Create an iSCSI target using a default target name.
    /iscsi> create 
    Created target 
    iqn.2003-01.org.linux-iscsi.hostname.x8664:sn.78b473f296ff
    Created TPG1
    Or create an iSCSI target using a specified name.
    /iscsi > create iqn.2006-04.com.example:444
    Created target iqn.2006-04.com.example:444
    Created TPG1
  4. Verify that the newly created target is visible when targets are listed with ls.
    /iscsi > ls
    o- iscsi.......................................[1 Target]
        o- iqn.2006-04.com.example:444................[1 TPG] 
            o- tpg1...........................[enabled, auth]
                o- acls...............................[0 ACL]
                o- luns...............................[0 LUN]
                o- portals.........................[0 Portal]
    

Note

As of Red Hat Enterprise Linux 7.1, whenever a target is created, a default portal is also created.

24.1.4. Configure an iSCSI Portal

To configure an iSCSI portal, an iSCSI target must first be created and associated with a TPG. For instructions on how to do this, refer to Section 24.1.3, “Create an iSCSI Target”.

Note

As of Red Hat Enterprise Linux 7.1 when an iSCSI target is created, a default portal is created as well. This portal is set to listen on all IP addresses with the default port number (that is, 0.0.0.0:3260). To remove this and add only specified portals, use /iscsi/iqn-name/tpg1/portals delete ip_address=0.0.0.0 ip_port=3260 then create a new portal with the required information.

Procedure 24.2. Create an iSCSI Portal

  1. Move into the TPG.
    /iscsi> iqn.2006-04.example:444/tpg1/
  2. There are two ways to create a portal: create a default portal, or create a portal specifying what IP address to listen to.
    Creating a default portal uses the default iSCSI port 3260 and allows the target to listen on all IP addresses on that port.
    /iscsi/iqn.20...mple:444/tpg1> portals/ create
    Using default IP port 3260
    Binding to INADDR_Any (0.0.0.0)
    Created network portal 0.0.0.0:3260
    
    To create a portal specifying what IP address to listen to, use the following command.
    /iscsi/iqn.20...mple:444/tpg1> portals/ create 192.168.122.137
    Using default IP port 3260
    Created network portal 192.168.122.137:3260
    
  3. Verify that the newly created portal is visible with the ls command.
    /iscsi/iqn.20...mple:444/tpg1> ls
    o- tpg.................................. [enambled, auth]  
        o- acls ......................................[0 ACL]
        o- luns ......................................[0 LUN]
        o- portals ................................[1 Portal]
            o- 192.168.122.137:3260......................[OK]
    

24.1.5. Configure LUNs

To configure LUNs, first create storage objects. See Section 24.1.2, “Create a Backstore” for more information.

Procedure 24.3. Configure LUNs

  1. Create LUNs of already created storage objects.
    /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/ramdisk/ramdisk1
    Created LUN 0.
    
    /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/block/block1
    Created LUN 1.
    
    /iscsi/iqn.20...mple:444/tpg1> luns/ create /backstores/fileio/file1
    Created LUN 2.
    
  2. Show the changes.
    /iscsi/iqn.20...mple:444/tpg1> ls
    o- tpg.................................. [enambled, auth]
        o- acls ......................................[0 ACL]
        o- luns .....................................[3 LUNs]
        |  o- lun0.........................[ramdisk/ramdisk1]
        |  o- lun1.................[block/block1 (/dev/vdb1)]
        |  o- lun2...................[fileio/file1 (/foo.img)]
        o- portals ................................[1 Portal]
            o- 192.168.122.137:3260......................[OK]
    

    Note

    Be aware that the default LUN name starts at 0, as opposed to 1 as was the case when using tgtd in Red Hat Enterprise Linux 6.

Important

By default, LUNs are created with read-write permissions. In the event that a new LUN is added after ACLs have been created that LUN will be automatically mapped to all available ACLs. This can cause a security risk. Use the following procedure to create a LUN as read-only.

Procedure 24.4. Create a read-only LUN

  1. To create a LUN with read-only permissions, first use the following command:
    /> set global auto_add_mapped_luns=false
    Parameter auto_add_mapped_luns is now 'false'.
    
    This prevents the auto mapping of LUNs to existing ACLs allowing the manual mapping of LUNs.
  2. Next, manually create the LUN with the command iscsi/target_iqn_name/tpg1/acls/initiator_iqn_name/ create mapped_lun=next_sequential_LUN_number tpg_lun_or_backstore=backstore write_protect=1.
    /> iscsi/iqn.2015-06.com.redhat:target/tpg1/acls/iqn.2015-06.com.redhat:initiator/ create mapped_lun=1 tpg_lun_or_backstore=/backstores/block/block2 write_protect=1
    Created LUN 1.
    Created Mapped LUN 1.
    /> ls
    o- / ...................................................... [...]
      o- backstores ........................................... [...]
      <snip>
      o- iscsi ......................................... [Targets: 1]
      | o- iqn.2015-06.com.redhat:target .................. [TPGs: 1]
      |   o- tpg1 ............................ [no-gen-acls, no-auth]
      |     o- acls ....................................... [ACLs: 2]
      |     | o- iqn.2015-06.com.redhat:initiator .. [Mapped LUNs: 2]
      |     | | o- mapped_lun0 .............. [lun0 block/disk1 (rw)]
      |     | | o- mapped_lun1 .............. [lun1 block/disk2 (ro)]
      |     o- luns ....................................... [LUNs: 2]
      |     | o- lun0 ...................... [block/disk1 (/dev/vdb)]
      |     | o- lun1 ...................... [block/disk2 (/dev/vdc)]
      <snip>
    
    The mapped_lun1 line now has (ro) at the end (unlike mapped_lun0's (rw)) stating that it is read-only.

24.1.6. Configure ACLs

Create an ACL for each initiator that will be connecting. This enforces authentication when that initiator connects, allowing only LUNs to be exposed to each initiator. Usually each initator has exclusive access to a LUN. Both targets and initiators have unique identifying names. The initiator's unique name must be known to configure ACLs. For open-iscsi initiators, this can be found in /etc/iscsi/initiatorname.iscsi.

Procedure 24.5. Configure ACLs

  1. Move into the acls directory.
    /iscsi/iqn.20...mple:444/tpg1> acls/
  2. Create an ACL. Either use the initiator name found in /etc/iscsi/initiatorname.iscsi on the initiator, or if using a name that is easier to remember, refer to Section 24.2, “Creating an iSCSI Initiator” to ensure ACL matches the initiator. For example:
    /iscsi/iqn.20...444/tpg1/acls> create iqn.2006-04.com.example.foo:888
    Created Node ACL for iqn.2006-04.com.example.foo:888
    Created mapped LUN 2.
    Created mapped LUN 1.
    Created mapped LUN 0.
    

    Note

    The above example's behavior depends on the setting used. In this case, the global setting auto_add_mapped_luns is used. This automatically maps LUNs to any created ACL.
  3. Show the changes.
    /iscsi/iqn.20...444/tpg1/acls> ls
    o- acls .................................................[1 ACL]
        o- iqn.2006-04.com.example.foo:888 ....[3 Mapped LUNs, auth]
            o- mapped_lun0 .............[lun0 ramdisk/ramdisk1 (rw)]
            o- mapped_lun1 .................[lun1 block/block1 (rw)]
            o- mapped_lun2 .................[lun2 fileio/file1 (rw)]
    

24.1.7. Fibre Channel over Ethernet (FCoE) Target Setup

In addition to mounting LUNs over FCoE, as described in Section 24.4, “Configuring a Fibre Channel over Ethernet Interface”, exporting LUNs to other machines over FCoE is also supported with the aid of targetcli.

Important

Before proceeding, refer to Section 24.4, “Configuring a Fibre Channel over Ethernet Interface” and verify that basic FCoE setup is completed, and that fcoeadm -i displays configured FCoE interfaces.

Procedure 24.6. Configure FCoE target

  1. Setting up an FCoE target requires the installation of the targetcli package, along with its dependencies. Refer to Section 24.1, “Target Setup” for more information on targetcli basics and set up.
  2. Create an FCoE target instance on an FCoE interface.
    /> tcm_fc/ create 00:11:22:33:44:55:66:77
    If FCoE interfaces are present on the system, tab-completing after create will list available interfaces. If not, ensure fcoeadm -i shows active interfaces.
  3. Map a backstore to the target instance.

    Example 24.1. Example of mapping a backstore to the target instance

    /> tcm_fc/00:11:22:33:44:55:66:77
    /> luns/ create /backstores/fileio/example2
  4. Allow access to the LUN from an FCoE initiator.
    /> acls/ create 00:99:88:77:66:55:44:33
    The LUN should now be accessible to that initiator.
  5. To make the changes persistant across reboots, use the saveconfig command and type yes when prompted. If this is not done the configuration will be lost after rebooting.
  6. Exit targetcli by typing exit or entering ctrl+D.

24.1.8. Remove objects with targetcli

To remove an backstore use the command:
/> /backstores/backstore-type/backstore-name
To remove parts of an iSCSI target, such as an ACL, use the following command:
/> /iscsi/iqn-name/tpg/acls/ delete iqn-name
To remove the entire target, including all ACLs, LUNs, and portals, use the following command:
/> /iscsi delete iqn-name

24.1.9. targetcli References

For more information on targetcli, refer to the following resources:
man targetcli
The targetcli man page. It includes an example walk through.
The Linux SCSI Target Wiki
Screencast by Andy Grover

Note

This was uploaded on February 28, 2012. As such, the service name has changed from targetcli to target.