Fast Forward Upgrade (FFU) from RHOSP10 to RHOSP13 fails due to IO hang

Solution Unverified - Updated -

Issue

  • Fast Forward Upgrade (FFU) from RHOSP10 to RHOSP13 fails due to IO hang

    $ openstack overcloud upgrade run --nodes Controller --skip-tags validation
    

    It looks like it got hung at this stage:

    TASK [Debug output for task: Run docker-puppet tasks (generate config) during step 3] ***
    Tuesday 07 January 2020  16:12:19 +0000 (0:00:00.158)       0:20:32.463 *******
    skipping: [overcloud-controller-proddevb-1] => {}
    skipping: [overcloud-controller-proddevb-0] => {}
    skipping: [overcloud-controller-proddevb-2] => {}
    
    TASK [Start containers for step 3] *********************************************
    Tuesday 07 January 2020  16:12:19 +0000 (0:00:00.158)       0:20:32.622 *******
    ok: [overcloud-controller-proddevb-2] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
    ok: [overcloud-controller-proddevb-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
    
    [Errno 104] Connection reset by peer
    
    • At the time of above failure CPU utilization was very high, it maybe an I/O problem.
    • The xfsaild process was consuming most of the CPU, and gradually the system started becoming unresponsive. The moment we stop docker containers, the system starts to recover back.

Environment

  • Red Hat OpenStack Platform 13.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In