OVN fails to configure after reboot during OSP-13 -> OSP-16.1 FFU

Issue

• After rebooting on the rhel-8, ovn-dbs container fails to start due to an SELinux denial during the deploy:

  2021-08-24 21:55:39,572 p=18856 u=mistral n=ansible | TASK [Start containers for step 3 using paunch] ********************************
  2021-08-24 21:55:39,573 p=18856 u=mistral n=ansible | Tuesday 24 August 2021  21:55:39 +0530 (0:00:00.105)       0:26:52.938 ******** 
  2021-08-24 21:55:39,981 p=18856 u=mistral n=ansible | changed: [controller01] => {"ansible_job_id": "435645617350.231201", "changed": true, "finished": 0, "results_file": "/root/.ansi
  ble_async/435645617350.231201", "started": 1}
  2021-08-24 21:55:40,034 p=18856 u=mistral n=ansible | TASK [Wait for containers to start for step 3 using paunch] ********************
  2021-08-24 21:55:40,034 p=18856 u=mistral n=ansible | Tuesday 24 August 2021  21:55:40 +0530 (0:00:00.461)       0:26:53.400 ******** 
  2021-08-24 22:59:16,751 p=18856 u=mistral n=ansible | fatal: [controller01]: FAILED! => {"ansible_job_id": "435645617350.231201", "attempts": 1200, "changed": false, "finished": 0, "started": 1}
  2021-08-24 22:59:16,751 p=18856 u=mistral n=ansible | NO MORE HOSTS LEFT *************************************************************
  2021-08-24 22:59:16,752 p=18856 u=mistral n=ansible | PLAY RECAP *********************************************************************
  2021-08-24 22:59:16,753 p=18856 u=mistral n=ansible | controller01        : ok=308  changed=168  unreachable=0    failed=1    skipped=145  rescued=0    ignored=0   
  2021-08-24 22:59:16,753 p=18856 u=mistral n=ansible | Tuesday 24 August 2021  22:59:16 +0530 (1:03:36.719)       1:30:30.119 ******** 
  2021-08-24 22:59:16,753 p=18856 u=mistral n=ansible | =============================================================================== 
  

• The SELinux denial, as shown in the /var/log/audit/audit.log:

  type=AVC msg=audit(1629830593.634:35262): avc:  denied  { setattr } for  pid=154289 comm="chown" name=".ovnnb_db.db.tmp.lock" dev="sde2" ino=1170249788 scontext=system_u:system_r:container_t:s0:c402,c949 tcontext=system_u:object_r:openvswitch_var_lib_t:s0 tclass=file permissive=0
  

• One can also check the SELinux labels associated to /var/lib/openvswitch/ovn

  [root@controller-0 ~]# ls -lZ  /var/lib/openvswitch/ovn
  total 38088
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0            21 Aug 24 20:17 ovnnb-active.conf
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0        883326 Aug 24 15:42 ovnnb_db.db
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0       8568547 Aug  4 16:26 ovnnb_db.db.backup5.10.1-64444197
  srwxr-x---. 1 root root system_u:object_r:openvswitch_var_lib_t:s0        0 Feb 22  2019 ovn-northd.205.ctl                     <-----
  srwxr-x---. 1 root root system_u:object_r:openvswitch_var_lib_t:s0        0 Mar  6  2019 ovn-northd.46182.ctl                   <-----
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0            21 Aug 24 20:17 ovnsb-active.conf
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0       6170532 Aug 24 19:33 ovnsb_db.db
  -rw-r-----. 1 root root system_u:object_r:container_file_t:s0      23367412 Aug  4 16:26 ovnsb_db.db.backup1.15.1-1164519396
  [root@controller-0 ~]# ls -dlZ  /var/lib/openvswitch/ovn
  drwxr-xr-x. 2 root root system_u:object_r:openvswitch_var_lib_t:s0 4096 Aug 27 11:53 /var/lib/openvswitch/ovn
  

• In the listing above, note the different type for *ovn-northd.205.ctl* and *ovn-northd.46182.ctl*, as well as the /var/lib/openvswitch/ovn directory.