Fast Forward Upgrade (FFU) from RHOSP10 to RHOSP13 fails due to IO hang

Issue

• Fast Forward Upgrade (FFU) from RHOSP10 to RHOSP13 fails due to IO hang

  $ openstack overcloud upgrade run --nodes Controller --skip-tags validation
  

  It looks like it got hung at this stage:

  TASK [Debug output for task: Run docker-puppet tasks (generate config) during step 3] ***
  Tuesday 07 January 2020  16:12:19 +0000 (0:00:00.158)       0:20:32.463 *******
  skipping: [overcloud-controller-proddevb-1] => {}
  skipping: [overcloud-controller-proddevb-0] => {}
  skipping: [overcloud-controller-proddevb-2] => {}
  
  TASK [Start containers for step 3] *********************************************
  Tuesday 07 January 2020  16:12:19 +0000 (0:00:00.158)       0:20:32.622 *******
  ok: [overcloud-controller-proddevb-2] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
  ok: [overcloud-controller-proddevb-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
  
  [Errno 104] Connection reset by peer
  

  • At the time of above failure CPU utilization was very high, it maybe an I/O problem.
  • The xfsaild process was consuming most of the CPU, and gradually the system started becoming unresponsive. The moment we stop docker containers, the system starts to recover back.