PCSD web UI shows "Unable to connect to cluster" and pcsd memory utilization is high

Issue

• The pcsd ruby process is consuming a lot of memory while the web UI is open. The web UI shows "unknown" or "disconnected" status for nodes or daemons.
• The Resident Set Size (RSS) of the pcsd ruby process slowly and steadily increases over time while a web UI browser session is open.
• The PCSD web UI may have a memory leak.
• "Cannot allocate memory" errors appear in /var/log/messages:

Jul 25 04:00:26 fastvm-rhel-8-0-23 systemd[1]: Starting PCS GUI and remote configuration interface (Ruby)...
..
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: 2020-07-25 08:00:55 - Errno::ENOMEM - Cannot allocate memory - fork(2):
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/open4-1.3.4/lib/open4.rb:62:in `fork'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/open4-1.3.4/lib/open4.rb:62:in `do_popen'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:888:in `run_cmd_options'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:861:in `run_cmd'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:1728:in `is_service_running?'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1182:in `block in load_current_node'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1181:in `each'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1181:in `load_current_node'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:846:in `node_status'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:138:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:138:in `remote'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcsd.rb:154:in `block in <top (required)>'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1636:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1636:in `block in compile!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:987:in `block (3 levels) in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1006:in `route_eval'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:987:in `block (2 levels) in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1035:in `block in process_route'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1033:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1033:in `process_route'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:985:in `block in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:984:in `each'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:984:in `route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1098:in `block in dispatch!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `block in invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1095:in `dispatch!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:919:in `block in call!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `block in invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:919:in `call!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:908:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/common_logger.rb:38:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:231:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/rserver.rb:32:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/xss_header.rb:18:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/path_traversal.rb:16:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/json_csrf.rb:26:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/base.rb:50:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/frame_options.rb:31:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:194:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1951:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1503:in `block in call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1730:in `synchronize'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1503:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:1077:in `block in spawn_threadpool'

• 500 errors are logged in /var/log/pcsd/pcsd.log:

E, [2020-07-25T08:00:55.952 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 1073.59ms
E, [2020-07-25T08:01:17.525 #01714]    ERROR -- : Failed to parse cib.
E, [2020-07-25T08:01:17.526 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 74.21ms
E, [2020-07-25T08:01:39.712 #00000]    ERROR -- : 500 POST /remote/cluster_status (192.168.22.23) 15.11ms
E, [2020-07-25T08:01:40.083 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 63.71ms