FTP client running on a mainframe fails when connecting via SSL

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 5
  • Red Hat Enterprise Linux (RHEL) 4
  • vsftpd
  • FTP client on an IBM mainframe zOS

Issue

  • An FTP client running on a mainframe fails to connect to a vsftpd server via SSL. The client replies "410 (SSL message format is incorrect)".
  • An FTP client running on a mainframe fails to connect to a vsftpd server via SSL. The client replies "SSL connection failed; session reuse required; see require_ssl_reuse option"

Resolution

  1. The issue is caused by SSL incompatibility between zOS and Linux implementations. The issue can be fixed by switching the following vsftpd option to no. Per `man vsftpd.conf

    ssl_request_cert
              If  enabled,  vsftpd  will  request  (but  not  necessarily  require; see require_cert) a certificate on
              incoming SSL connections. Normally this should not cause  any trouble  at  all,  but IBM zOS seems to have
              issues.  (New in v2.0.7).
    
              Default: YES
    
  2. Unfortunately, the issue can persist even after this configuration change. The client might still fail while getting a file, returning the following message:

    522 SSL connection failed; session reuse required; see require_ssl_reuse option
    
  • Examples of the two above parameters to set in /etc/vsftpd/vsftpd.conf:

    ssl_request_cert=no
    require_ssl_reuse=no
    

Root Cause

The described issue is caused by incompatibility of OpenSSL implementations between the IBM mainframe zOS ftp client and Linux implementations.

See also the following references:

Diagnostic Steps

  • Mainframe FTP clients fail when trying to create an SSL session with a vsftpd server running on a RHEL system. The log of the FTP client looks like the following snippet:

    FC0914 authServer: secure_socket_open()
    FC0981 authServer: secure_socket_init()
    FC0994 authServer: secure_socket_init failed with rc = 410 (SSL message format is incorrect)
    FC1349 endSecureConn: entered
    EZA2897I Authentication negotiation failed
    FC1381 endSecureEnv: entered
    CZ0655 SETCEC code = 17
    EZA2898I Unable to successfully negotiate required authentication
    EZA1460I Command:
    EZA1618I Unknown command: 'testftps'
    EZA1619I For a list of the available commands, say HELP
    EZA1460I Command:
    EZA1736I popkin22
    EZA1618I Unknown command: 'popkin22'
    
  • This issue on the client side causes exiting vsftpd child processes handling the mainframe FTP client with the following message, which can be found in the vsftpd strace log:

    12213 write(0, "500 OOPS: ", 10)        = 10
    12213 write(0, "error:14094417:SSL routines:SSL3"..., 73) = 73
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.