How many LUNs can be mapped from targetcli/LIO based targets?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)
    • 7
    • 8
  • Linux IO target (LIO)
  • targetcli

Issue

  • Getting following error while exporting more than 256 LUNs using iSCSI target configured in targetcli:

    /iscsi/iqn.20...0bc/tpg1/luns> create /backstores/fileio/file255
    Created LUN 255.
    Cannot map new lun 255 into ACL iqn.1994-05.com.redhat:2012b45c62c5
    
  • How many LUNs can be mapped from targetcli/LIO based targets?

Resolution

  • RHEL 7.5 kernel-3.10.0-862.el7 and later kernels no longer have a 256 LUNs limit. However, the limit remains in the userspace tooling targetcli/ftslib for creating and managing LUNs.
  • Red Hat is aware of a feature request to remove the limit in targetcli/ftslib, currently under review in RFE RHBZ#2149425. If you require this feature, open a support case in the Red Hat Customer Portal referring to this solution.

Root Cause

  • The original RFC 3720 for iSCSI describes a LUN ID field as being 64-bit identifier. It is formatted in accordance with SAM-2. So, there is no iSCSI protocol limitation which restricts the maximum number of mapped_luns to 256.

    RFC 3720
    10.2.1.7.  LUN
    
       Some opcodes operate on a specific Logical Unit.  The Logical Unit
       Number (LUN) field identifies which Logical Unit.  If the opcode does
       not relate to a Logical Unit, this field is either ignored or may be
       used in an opcode specific way.  The LUN field is 64-bits and should
       be formatted in accordance with [SAM2].  For example, LUN[0] from
       [SAM2] is BHS byte 8 and so on up to LUN[7] from [SAM2], which is BHS
       byte 15
    
  • The older LIO target code limited the max number of mapped LUNs to 'TRANSPORT_MAX_LUNS_PER_TPG' count of 256.

    include/target/target_core_base.h
            /* Maximum Number of LUNs per Target Portal Group */
            /* Don't raise above 511 or REPORT_LUNS needs to handle >1 page */
            #define TRANSPORT_MAX_LUNS_PER_TPG             256          <------
    

    Above limitation was removed through following upstream patch:
    target: Remove TARGET_MAX_LUNS_PER_TRANSPORT

    This patch was back-ported to RHEL 7.5 kernel-3.10.0-862.el7 through private BZ#1366062

  • While the initial 'TRANSPORT_MAX_LUNS_PER_TPG' limitation in LIO target is now removed, the targetcli code still uses MAX_LUN count of 255. So, the logical units can have values of 0-255, which results in max 256 total logical units:

    rtslib-fb/rtslib/target.py
            class MappedLUN(CFSNode):
                '''  
                This is an interface to RTS Target Mapped LUNs.
                A MappedLUN is a mapping of a TPG LUN to a specific initiator node, and is
                part of a NodeACL. It allows the initiator to actually access the TPG LUN
                if ACLs are enabled for the TPG. The initial TPG LUN will then be seen by
                the initiator node as the MappedLUN.
                '''
    
                MAX_LUN = 255
    
                # MappedLUN private stuff
               [...]
    
  • This limit may be increased in future. If there is a requirement to export more than 256 LUNs through targetcli/LIO based target, then please reach out to your Red Hat support representative and RFE (Request for Feature Enhancement) could be created for the same.

Diagnostic Steps

  • Verify on target server by using targetcli ls command and look for below-highlighted numbers-

    o- iscsi .............................................................. [Targets: 2]
      | o- iqn.2003-01.org.linux-iscsi.server.x8664:sn.699f8cd480bc ....... [TPGs: 1]  <- 1 Target
      | | o- tpg1 ......................................................... [no-gen-acls, no-auth]
      | |   o- acls ....................................................... [ACLs: 1]
      | |   | o- iqn.1994-05.com.redhat:2012b45c62c5 .... [Mapped LUNs: 255]  <- 256 mapped luns (0-255) per initiator
      | |   o- luns ....................................................... [LUNs: 261]  <- LUN per target is under 65K limit
    
  • To verify on the client side, use /proc/scsi/scsi command:

    Host: scsi12 Channel: 00 Id: 00 Lun: 02
      Vendor: LIO-ORG  Model: file2            Rev: 4.0 
      Type:   Direct-Access                    ANSI  SCSI revision: 05
    
    #cat /proc/scsi/scsi | grep -i "Vendor: LIO-ORG"| wc -l
    255
    
  • Trying to map a LUN that is greater than 255 will get errors like Cannot map new lun 255 into ACL iqn.

    /iscsi/iqn.20...0bc/tpg1/luns> create /backstores/fileio/file255
    Created LUN 255.
    Cannot map new lun 255 into ACL iqn.1994-05.com.redhat:2012b45c62c5
    /iscsi/iqn.20...0bc/tpg1/luns> create /backstores/fileio/file256 
    Created LUN 256.
    Cannot map new lun 256 into ACL iqn.1994-05.com.redhat:2012b45c62c5
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments