Installing:

1. On the iSCSI gateway nodes, enable the Red Hat Ceph Storage 3 Tools repository. For details, see the Enabling the Red Hat Ceph Storage Repositories section in the Installation Guide for Red Hat Enterprise Linux.

   1. Install the ceph-iscsi-config package:

      # yum install ceph-iscsi-config

2. On the Ansible administration node, do the following steps, as the root user:

   1. Enable the Red Hat Ceph Storage 3 Tools repository. For details, see the Enabling the Red Hat Ceph Storage Repositories section in the Installation Guide for Red Hat Enterprise Linux.

   2. Install the ceph-ansible package:

      # yum install ceph-ansible

   3. Add an entry in /etc/ansible/hosts file for the gateway group:

      [iscsigws]
      ceph-igw-1
      ceph-igw-2

      Note

      If colocating the iSCSI gateway with an OSD node, add the OSD node to the [iscsigws] section.

1. Execute the Ansible playbook:

   # cd /usr/share/ceph-ansible
   # ansible-playbook site.yml

   Note

   The Ansible playbook will handle RPM dependencies, RBD creation and Linux iSCSI target configuration.

   Warning

   On stand-alone iSCSI gateway nodes, verify that the correct Red Hat Ceph Storage 3.0 software repositories are enabled. If they are unavailable, then the wrong packages will be installed.

2. Verify the configuration by running the following command:

   # gwcli ls

   Important

   Do not use the targetcli utility to change the configuration, this will result in the following issues: ALUA misconfiguration and path failover problems. There is the potential to corrupt data, to have mismatched configuration across iSCSI gateways, and to have mismatched WWN information, which will lead to client pathing problems.

1. Restart the ceph-mon service, as the root user:

   # systemctl restart ceph-mon@$MONITOR_HOST_NAME

   For example:

   # systemctl restart ceph-mon@monitor1

1. If the Ceph iSCSI gateway is not colocated on an OSD node, then copy the Ceph configuration files, located in /etc/ceph/, from a running Ceph node in the storage cluster to the iSCSI Gateway node. The Ceph configuration files must exist on the iSCSI gateway node under /etc/ceph/.
2. Install and configure the Ceph command-line interface.For details, see the Installing the Ceph Command Line Interface chapter in the Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux.

3. Enable the Ceph tools repository:

   # subscription-manager repos --enable=rhel-7-server-ceph-3-tools-rpms

4. If needed, open TCP ports 3260 and 5000 on the firewall.

5. Create a new or use an existing RADOS Block Device (RBD).

   1. See Section 2.1, “Prerequisites” for more details.

1. Install the ceph-iscsi-cli package:

   # yum install ceph-iscsi-cli

2. Install the tcmu-runner package:

   # yum install tcmu-runner

3. If needed, install the openssl package:

   # yum install openssl

   1. On the primary iSCSI gateway node, create a directory to hold the SSL keys:

      # mkdir ~/ssl-keys
      # cd ~/ssl-keys

   2. On the primary iSCSI gateway node, create the certificate and key files:

      # openssl req -newkey rsa:2048 -nodes -keyout iscsi-gateway.key -x509 -days 365 -out iscsi-gateway.crt

      Note

      You will be prompted to enter the environmental information.

   3. On the primary iSCSI gateway node, create a PEM file:

      # cat iscsi-gateway.crt iscsi-gateway.key > iscsi-gateway.pem

   4. On the primary iSCSI gateway node, create a public key:

      # openssl x509 -inform pem -in iscsi-gateway.pem -pubkey -noout > iscsi-gateway-pub.key

   5. From the primary iSCSI gateway node, copy the iscsi-gateway.crt, iscsi-gateway.pem, iscsi-gateway-pub.key, and iscsi-gateway.key files to the /etc/ceph/ directory on the other iSCSI gateway nodes.

4. Create a file named iscsi-gateway.cfg in the /etc/ceph/ directory:

   # touch /etc/ceph/iscsi-gateway.cfg

   1. Edit the iscsi-gateway.cfg file and add the following lines:

      Syntax

      [config]
      cluster_name = <ceph_cluster_name>
      gateway_keyring = <ceph_client_keyring>
      api_secure = true
      trusted_ip_list = <ip_addr>,<ip_addr>

      Example

      [config]
      cluster_name = ceph
      gateway_keyring = ceph.client.admin.keyring
      api_secure = true
      trusted_ip_list = 192.168.0.10,192.168.0.11

      See Tables 8.1 and 8.2 in the Requirements: for more details on these options.

      Important

      The iscsi-gateway.cfg file must be identical on all iSCSI gateway nodes.

   2. Copy the iscsi-gateway.cfg file to all iSCSI gateway nodes.

5. Enable and start the API service:

   # systemctl enable rbd-target-api
   # systemctl start rbd-target-api

Configuring:

1. Start the iSCSI gateway command-line interface:

   # gwcli

2. Creating the iSCSI gateways:

   Syntax

   >/iscsi-target create iqn.2003-01.com.redhat.iscsi-gw:<target_name>
   > goto gateways
   > create <iscsi_gw_name> <IP_addr_of_gw>
   > create <iscsi_gw_name> <IP_addr_of_gw>

   Example

   >/iscsi-target create iqn.2003-01.com.redhat.iscsi-gw:ceph-igw
   > goto gateways
   > create ceph-gw-1 10.172.19.21
   > create ceph-gw-2 10.172.19.22

3. Adding a RADOS Block Device (RBD):

   Syntax

   > cd /disks
   >/disks/ create <pool_name> image=<image_name> size=<image_size>m|g|t max_data_area_mb=<buffer_size>

   Example

   > cd /disks
   >/disks/ create rbd image=disk_1 size=50g max_data_area_mb=32

   Important

   There can not be any periods (.) in the pool name or in the image name.

   Note

   The max_data_area_mb option controls the amount of memory in megabytes that each image can use to pass SCSI command data between the iSCSI target and the Ceph cluster. If this value is too small, then it can result in excessive queue full retries which will affect performance. If the value is too large, then it can result in one disk using too much of the system’s memory, which can cause allocation failures for other subsystems. The default value is 8.

   This value can be changed using the reconfigure command The image must not be in use by an iSCSI initiator for this command to take effect.

   Syntax

   >/disks/ reconfigure max_data_area_mb <new_buffer_size>

   Example

   >/disks/ reconfigure max_data_area_mb 64

4. Creating a client:

   Syntax

   > goto hosts
   > create iqn.1994-05.com.redhat:<client_name>
   > auth chap=<user_name>/<password>

   Example

   > goto hosts
   > create iqn.1994-05.com.redhat:rh7-client
   > auth chap=iscsiuser1/temp12345678

   Important

   Disabling CHAP is only supported on Red Hat Ceph Storage 3.1 or higher. Red Hat does not support mixing clients, some with CHAP enabled and some CHAP disabled. All clients must have either CHAP enabled or have CHAP disabled. The default behavior is to only authenticate an initiator by its initiator name.

   If initiators are failing to log into the target, then the CHAP authentication might be a misconfigured for some initiators.

   Example

   o- hosts ................................ [Hosts: 2: Auth: MISCONFIG]

   Do the following command at the hosts level to reset all the CHAP authentication:

   /> goto hosts
   /iscsi-target...csi-igw/hosts> auth nochap
   ok
   ok
   /iscsi-target...csi-igw/hosts> ls
   o- hosts ................................ [Hosts: 2: Auth: None]
     o- iqn.2005-03.com.ceph:esx ........... [Auth: None, Disks: 4(310G)]
     o- iqn.1994-05.com.redhat:rh7-client .. [Auth: None, Disks: 0(0.00Y)]

5. Adding disks to a client:

   Syntax

   >/iscsi-target..eph-igw/hosts> cd iqn.1994-05.com.redhat:<client_name>
   > disk add <pool_name>.<image_name>

   Example

   >/iscsi-target..eph-igw/hosts> cd iqn.1994-05.com.redhat:rh7-client
   > disk add rbd.disk_1

6. To confirm that the API is using SSL correctly, look in the /var/log/rbd-target-api.log file for https, for example:

   Aug 01 17:27:42 test-node.example.com python[1879]:  * Running on https://0.0.0.0:5001/

7. The next step is to configure an iSCSI initiator. See Section 8.3, “Configuring the iSCSI Initiator” for more information on configuring an iSCSI initiator.

Verifying

1. To verify if the iSCSI gateways are working:

   Example

   /> goto gateways
   /iscsi-target...-igw/gateways> ls
   o- gateways ............................ [Up: 2/2, Portals: 2]
     o- ceph-gw-1  ........................ [ 10.172.19.21 (UP)]
     o- ceph-gw-2  ........................ [ 10.172.19.22 (UP)]

   Note

   If the status is UNKNOWN, then check for network issues and any misconfigurations. If using a firewall, then check if the appropriate TCP port is open. Check if the iSCSI gateway is listed in the trusted_ip_list option. Verify that the rbd-target-api service is running on the iSCSI gateway node.

2. To verify if the initiator is connecting to the iSCSI target, you will see the initiator LOGGED-IN:

   Example

   /> goto hosts
   /iscsi-target...csi-igw/hosts> ls
   o- hosts .............................. [Hosts: 1: Auth: None]
     o- iqn.1994-05.com.redhat:rh7-client  [LOGGED-IN, Auth: None, Disks: 0(0.00Y)]

3. To verify if LUNs are balanced across iSCSI gateways:

   /> goto hosts
   /iscsi-target...csi-igw/hosts> ls
   o- hosts ................................. [Hosts: 2: Auth: None]
     o- iqn.2005-03.com.ceph:esx ............ [Auth: None, Disks: 4(310G)]
     | o- lun 0 ............................. [rbd.disk_1(100G), Owner: ceph-gw-1]
     | o- lun 1 ............................. [rbd.disk_2(10G), Owner: ceph-gw-2]

   When creating a disk, the disk will be assigned an iSCSI gateway as its Owner based on the initiator’s multipath layer. The initiator’s multipath layer will reported as being in ALUA Active-Optimized (AO) state. The other paths will be reported as being in the ALUA Active-non-Optimized (ANO) state.

   If the AO path fails one of the other iSCSI gateways will be used. The ordering for the failover gateway depends on the initiator’s multipath layer, where normally, the order is based on which path was discovered first.

   Currently, the balancing of LUNs is not dynamic. The owning iSCSI gateway is selected at disk creation time and is not changeable.

1. Install the iSCSI initiator and multipath tools:

   # yum install iscsi-initiator-utils
   # yum install device-mapper-multipath

1. Edit the /etc/iscsi/initiatorname.iscsi file.

   Note

   The initiator name must match the initiator name used in the Ansible client_connections option or what was used during the initial setup using gwcli.

1. Create the default /etc/multipath.conf file and enable the multipathd service:

   # mpathconf --enable --with_multipathd y

2. Add the following to /etc/multipath.conf file:

   devices {
           device {
                   vendor                 "LIO-ORG"
                   hardware_handler       "1 alua"
                   path_grouping_policy   "failover"
                   path_selector          "queue-length 0"
                   failback               60
                   path_checker           tur
                   prio                   alua
                   prio_args              exclusive_pref_bit
                   fast_io_fail_tmo       25
                   no_path_retry          queue
           }
   }

3. Restart the multipathd service:

   # systemctl reload multipathd

1. Provide a CHAP username and password by updating the /etc/iscsi/iscsid.conf file accordingly.

   Example

   node.session.auth.authmethod = CHAP
   node.session.auth.username = user
   node.session.auth.password = password

   Note

   If you update these options, then you must rerun the iscsiadm discovery command.

2. Discover the target portals:

   # iscsiadm -m discovery -t st -p 192.168.56.101
   192.168.56.101:3260,1 iqn.2003-01.org.linux-iscsi.rheln1
   192.168.56.102:3260,2 iqn.2003-01.org.linux-iscsi.rheln1

3. Login to target:

   # iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.rheln1 -l

1. Click the Storage resource tab to list the existing storage domains.
2. Click the New Domain button to open the New Domain window.
3. Enter the Name of the new storage domain.
4. Use the Data Center drop-down menu to select an data center.
5. Use the drop-down menus to select the Domain Function and the Storage Type. The storage domain types that are not compatible with the chosen domain function are not available.
6. Select an active host in the Use Host field. If this is not the first data domain in a data center, you must select the data center’s SPM host.

7. The New Domain window automatically displays known targets with unused LUNs when iSCSI is selected as the storage type. If the target that you are adding storage from is not listed then you can use target discovery to find it, otherwise proceed to the next step.

   1. Click Discover Targets to enable target discovery options. When targets have been discovered and logged in to, the New Domain window automatically displays targets with LUNs unused by the environment.

      Note

      LUNs external to the environment are also displayed.

      You can use the Discover Targets options to add LUNs on many targets, or multiple paths to the same LUNs.

   2. Enter the fully qualified domain name or IP address of the iSCSI host in the Address field.
   3. Enter the port to connect to the host on when browsing for targets in the Port field. The default is 3260.
   4. If the Challenge Handshake Authentication Protocol (CHAP) is being used to secure the storage, select the User Authentication check box. Enter the CHAP user name and CHAP password.
   5. Click the Discover button.

   6. Select the target to use from the discovery results and click the Login button. Alternatively, click the Login All to log in to all of the discovered targets.

      Important

      If more than one path access is required, ensure to discover and log in to the target through all the required paths. Modifying a storage domain to add additional paths is currently not supported.

8. Click the + button next to the desired target. This will expand the entry and display all unused LUNs attached to the target.
9. Select the check box for each LUN that you are using to create the storage domain.

10. Optionally, you can configure the advanced parameters.

    1. Click Advanced Parameters.
    2. Enter a percentage value into the Warning Low Space Indicator field. If the free space available on the storage domain is below this percentage, warning messages are displayed to the user and logged.
    3. Enter a GB value into the Critical Space Action Blocker field. If the free space available on the storage domain is below this value, error messages are displayed to the user and logged, and any new action that consumes space, even temporarily, will be blocked.
    4. Select the Wipe After Delete check box to enable the wipe after delete option. This option can be edited after the domain is created, but doing so will not change the wipe after delete property of disks that already exist.
    5. Select the Discard After Delete check box to enable the discard after delete option. This option can be edited after the domain is created. This option is only available to block storage domains.
11. Click OK to create the storage domain and close the window.

1. Install the iSCSI initiator driver and MPIO tools.
2. Launch the MPIO program, click on the Discover Multi-Paths tab, check the Add support for iSCSI devices box, and click Add. This change will require a reboot.

3. On the iSCSI Initiator Properties window, on the Discovery tab , add a target portal. Enter the IP address or DNS name and Port of the Ceph iSCSI gateway:

   

4. On the Targets tab , select the target and click on Connect :

   

5. On the Connect To Target window, select the Enable multi-path option , and click the Advanced button :

   

6. Under the Connect using section, select a Target portal IP . Select the Enable CHAP login on and enter the Name and Target secret values from the Ceph iSCSI Ansible client credentials section, and click OK :

   
   Important

   Windows Server 2016 does not accept a CHAP secret less than 12 bytes.

7. Repeat steps 5 and 6 for each target portal defined when setting up the iSCSI gateway.

8. If the initiator name is different than the initiator name used during the initial setup, then rename the initiator name. From iSCSI Initiator Properties window, on the Configuration tab , click the Change button to rename the initiator name.

   

1. Set the failover policy

1. Verify the failover policy

1. Using the iSCSI Initiator tool, from the Targets tab click on the Devices…​ button :

   

2. From the Devices window, select a disk and click the MPIO…​ button :

   

3. On the Device Details window the paths to each target portal is displayed. If using the ceph-ansible setup method, the iSCSI gateway will use ALUA to tell the iSCSI initiator which path and iSCSI gateway should be used as the primary path. The Load Balancing Policy Fail Over Only must be selected

   
4. From PowerShell, to view the multipath configuration

1. Disable HardwareAcceleratedMove (XCOPY):

   # esxcli system settings advanced set --int-value 0 --option /DataMover/HardwareAcceleratedMove

2. Enable the iSCSI software. From Navigator pane, click on Storage . Select the Adapters tab . Click on Confgure iSCSI :

   

3. Verify the initiator name in the Name & alias section .

   
   Note

   If the initiator name is different than the initiator name used when creating the client during the initial setup using gwcli or if the initiator name used in the Ansible client_connections: client variable is different, then follow this procedure to change the initiator name. From the VMware ESX host, run these esxcli commands.

   1. Get the adapter name for the iSCSI software:

      > esxcli iscsi adapter list
      > Adapter  Driver     State   UID            Description
      > -------  ---------  ------  -------------  ----------------------
      > vmhba64  iscsi_vmk  online  iscsi.vmhba64  iSCSI Software Adapter

   2. Set the initiator name:

      Syntax

      > esxcli iscsi adapter set -A <adaptor_name> -n <initiator_name>

      Example

      > esxcli iscsi adapter set -A vmhba64 -n iqn.1994-05.com.redhat:rh7-client

4. Configure CHAP. Expand the CHAP authentication section . Select “Do not use CHAP unless required by target” . Enter the CHAP Name and Secret credentials that were used in the initial setup, whether using the gwcli auth command or the Ansible client_connections: credentials variable. Verify the Mutual CHAP authentication section has “Do not use CHAP” selected.

   
   Warning

   There is a bug in the vSphere Web Client where the CHAP settings are not used initially. On the Ceph iSCSI gateway node, in kernel logs, you will see the following errors as an indication of this bug:

   > kernel: CHAP user or password not set for Initiator ACL
   > kernel: Security negotiation failed.
   > kernel: iSCSI Login negotiation failed.

   To workaround this bug, configure the CHAP settings using the esxcli command. The authname argument is the Name in the vSphere Web Client:

   > esxcli iscsi adapter auth chap set --direction=uni --authname=myiscsiusername --secret=myiscsipassword --level=discouraged -A vmhba64

5. Configure the iSCSI settings. Expand Advanced settings . Set the RecoveryTimeout value to 25 .

   

6. Set the discovery address. In the Dynamic targets section , click Add dynamic target . Under Address add an IP addresses for one of the Ceph iSCSI gateways. Only one IP address needs to be added. Finally, click the Save configuration button . From the main interface, on the Devices tab, you will see the RBD image.

   
   Note

   Configuring the LUN will be done automatically, using the ALUA SATP and MRU PSP. Other SATPs and PSPs must not be used. This can be verified with the esxcli command:

   esxcli storage nmp path list -d eui.$DEVICE_ID

   Replace $DEVICE_ID with the appropriate device identifier.

7. Verify that multipathing has been setup correctly.

   1. List the devices:

      Example

      # esxcli storage nmp device list | grep iSCSI
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405057360ba9b4c434daa3c6770c)

   2. Get the multipath information for the Ceph iSCSI disk from the previous step:

      Example

      # esxcli storage nmp path list -d naa.6001405f8d087846e7b4f0e9e3acd44b
      
      iqn.2005-03.com.ceph:esx1-00023d000001,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,1-naa.6001405f8d087846e7b4f0e9e3acd44b
         Runtime Name: vmhba64:C0:T0:L0
         Device: naa.6001405f8d087846e7b4f0e9e3acd44b
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Group State: active
         Array Priority: 0
         Storage Array Type Path Config: {TPG_id=1,TPG_state=AO,RTP_id=1,RTP_health=UP}
         Path Selection Policy Path Config: {current path; rank: 0}
      
      iqn.2005-03.com.ceph:esx1-00023d000002,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,t,2-naa.6001405f8d087846e7b4f0e9e3acd44b
         Runtime Name: vmhba64:C1:T0:L0
         Device: naa.6001405f8d087846e7b4f0e9e3acd44b
         Device Display Name: LIO-ORG iSCSI Disk (naa.6001405f8d087846e7b4f0e9e3acd44b)
         Group State: active unoptimized
         Array Priority: 0
         Storage Array Type Path Config: {TPG_id=2,TPG_state=ANO,RTP_id=2,RTP_health=UP}
         Path Selection Policy Path Config: {non-current path; rank: 0}

      From the example output, each path has an iSCSI/SCSI name with the following parts:

      Initiator name = iqn.2005-03.com.ceph:esx1 ISID = 00023d000002 Target name = iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw Target port group = 2 Device id = naa.6001405f8d087846e7b4f0e9e3acd44b

      The Group State value of active indicates this is the Active-Optimized path to the iSCSI gateway. The gwcli command lists the active as the iSCSI gateway owner. The rest of the paths will have the Group State value of unoptimized and will be the failover path, if the active path goes into a dead state.

8. To match all paths to their respective iSCSI gateways, run the following command:

   # esxcli iscsi session connection list

   Example output

   vmhba64,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000001,0
      Adapter: vmhba64
      Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
      ISID: 00023d000001
      CID: 0
      DataDigest: NONE
      HeaderDigest: NONE
      IFMarker: false
      IFMarkerInterval: 0
      MaxRecvDataSegmentLength: 131072
      MaxTransmitDataSegmentLength: 262144
      OFMarker: false
      OFMarkerInterval: 0
      ConnectionAddress: 10.172.19.21
      RemoteAddress: 10.172.19.21
      LocalAddress: 10.172.19.11
      SessionCreateTime: 08/16/18 04:20:06
      ConnectionCreateTime: 08/16/18 04:20:06
      ConnectionStartTime: 08/16/18 04:30:45
      State: logged_in
   
   vmhba64,iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw,00023d000002,0
      Adapter: vmhba64
      Target: iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw
      ISID: 00023d000002
      CID: 0
      DataDigest: NONE
      HeaderDigest: NONE
      IFMarker: false
      IFMarkerInterval: 0
      MaxRecvDataSegmentLength: 131072
      MaxTransmitDataSegmentLength: 262144
      OFMarker: false
      OFMarkerInterval: 0
      ConnectionAddress: 10.172.19.22
      RemoteAddress: 10.172.19.22
      LocalAddress: 10.172.19.12
      SessionCreateTime: 08/16/18 04:20:06
      ConnectionCreateTime: 08/16/18 04:20:06
      ConnectionStartTime: 08/16/18 04:30:41
      State: logged_in

   Match the path name with the ISID value, and the RemoteAddress value is the IP address of the owning iSCSI gateway.