Repeatedly errors: client_io->complete_request() returned Input/output error

Issue

• We have deployed quite a lot of RHOSP13 clouds. In the recently deployed clouds, we found in /var/log/message on controller there are repeatedly error all the time:

Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.347929 7fafd55a3700  1 ====== starting new request req=0x7fafd559cf70 =====
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349152 7fafd55a3700  0 ERROR: client_io->complete_request() returned Input/output error
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.347929 7fafd55a3700  1 ====== starting new request req=0x7fafd559cf70 =====
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349152 7fafd55a3700  0 ERROR: client_io->complete_request() returned Input/output error
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349173 7fafd55a3700  1 ====== req done req=0x7fafd559cf70 op status=0 http_status=200 ======
Feb 20 16:34:59 overcloud-controller-0 docker: 2020-02-20 16:34:59.349209 7fafd55a3700  1 civetweb: 0x55d75524e000: 172.18.0.11 - - [20/Feb/2020:16:34:59 +0100] "HEAD / HTTP/1.0" 200 182 - -
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349173 7fafd55a3700  1 ====== req done req=0x7fafd559cf70 op status=0 http_status=200 ======
Feb 20 16:34:59 overcloud-controller-0 journal: 2020-02-20 16:34:59.349209 7fafd55a3700  1 civetweb: 0x55d75524e000: 172.18.0.11 - - [20/Feb/2020:16:34:59 +0100] "HEAD / HTTP/1.0" 200 182 - -
Feb 20 16:34:59 overcloud-controller-0 ovs-vswitchd: ovs|347528|netdev_tc_offloads(revalidator603)|ERR|Dropped 17 log messages in last 1 seconds (most recently, 1 seconds ago) due to excessive rate
Feb 20 16:34:59 overcloud-controller-0 ovs-vswitchd: ovs|347529|netdev_tc_offloads(revalidator603)|ERR|dump_create: failed to get ifindex for tapda1682c1-76: Operation not supported
Feb 20 16:35:00 overcloud-controller-0 su: (to rabbitmq) root on none

• The cloud itself seems working fine. Please help to investigate what causes the problem.

• The first cloud with the problem was deployed in the end of January. The problem is seen all the time.