HPE Cinder 3Par Driver Error ImageCopyFailure

Solution In Progress - Updated -

Issue

  • We have a fairly big OSP10 install base, with hundreds of Compute nodes and our hardware topology is standardized on HPE Gen9 and Gen10 Blades and HPE 3Par Storage connected using FCoE.

  • Let me start with a strong and important statement, our OSP10 production environments are rock-solid!

  • We are in the process of migrating to OSP13 and during the testing phase we found instability with the Cinder 3Par driver. The current OSP13 test environment mimics our production layout (same blades, same hardware, same uEFI config, same storage, same storage config, only reduced quantities). This is ensured thanks to our OneView Server profiles.

  • Right now we run the latest software stack from Red Hat with OSP13z11.async using RHEL 7.8 (latest GA kernel - 3.10.0-1127.10.1 - at the time of writing) but the error was also present with OSP13z11 without the 7.8 async and was originally discovered in March 2020. We also run the latest HPE 3Par container (currently version 1-20) but the problem was originally discovered with version 1-17 (we never tested any earlier release). In other words while we went through quite a few system updates (RHEL 7.7 -> 7.8, multiple OSP13Z11 containers tags, multiple HPE 3Par container tags) the error has been a constant.

  • The error happens only on the Controller nodes when copying images to volumes. The frequency is quite random, it can fail right away or after some volume creation. We have created a script to generate 200 volumes (here attached) as reproduced. This script creates 2 volumes in parallel but the error can happen with a single volume and sometimes we can create 50 volumes in parallel without troubles. Hence the error is not dependent on how many volumes are generated at a given time.

  • The error is similar (but not complete the same) to the one reported in the following article.

  • Multipath is enabled in Cinder Volume (as the above article suggests - but it doesn't really solve the issue).

  • Follows the extract of the error:

2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc [req-6fe0a3e5-3957-4e6d-a861-1ba55acdf4e9 8fff3b9af5364b1893702c71c4786dfa 59bfce3d280d4f839756ae97ca6f8e08 - default default] Create FC host caught HTTP conflict code: 16: HTTPConflict: Conflict (HTTP 409) 16 - host exists
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc Traceback (most recent call last):
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_fc.py", line 336, in _create_3par_fibrechan_host
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     'persona': persona_id})
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/hpe3parclient/client.py", line 1684, in createHost
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     response, body = self.http.post('/hosts', body=info)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 376, in post
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     return self._cs_request(url, 'POST', **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 321, in _cs_request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 297, in _time_request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     resp, body = self.request(url, method, **kwargs)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc   File "/usr/lib/python2.7/site-packages/hpe3parclient/http.py", line 262, in request
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc     raise exceptions.from_response(resp, body)
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc HTTPConflict: Conflict (HTTP 409) 16 - host exists
2020-06-19 11:38:24.085 34 ERROR cinder.volume.drivers.hpe.hpe_3par_fc
2020-06-19 11:38:24.087 34 DEBUG cinder.volume.drivers.hpe.hpe_3par_common [req-6fe0a3e5-3957-4e6d-a861-1ba55acdf4e9 8fff3b9af5364b1893702c71c4786dfa 59bfce3d280d4f839756ae97ca6f8e08 - default default] Disconnect from 3PAR REST and SSH 91fa4337-abd2-4a68-8654-4a4f48c6da1f client_logout /usr/lib/python2.7/site-packages/cinder/volume/drivers/hpe/hpe_3par_common.py:412
  • We need your help to understand the problem and fix it (whether is a config problem, driver bug, something in RHEL etc)

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In