PCSD web UI shows "Unable to connect to cluster" and pcsd memory utilization is high

Solution In Progress - Updated -

Issue

  • The pcsd ruby process is consuming a lot of memory while the web UI is open. The web UI shows "unknown" or "disconnected" status for nodes or daemons.
  • The Resident Set Size (RSS) of the pcsd ruby process slowly and steadily increases over time while a web UI browser session is open.
  • The PCSD web UI may have a memory leak.
  • "Cannot allocate memory" errors appear in /var/log/messages:
Jul 25 04:00:26 fastvm-rhel-8-0-23 systemd[1]: Starting PCS GUI and remote configuration interface (Ruby)...
..
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: 2020-07-25 08:00:55 - Errno::ENOMEM - Cannot allocate memory - fork(2):
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/open4-1.3.4/lib/open4.rb:62:in `fork'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/open4-1.3.4/lib/open4.rb:62:in `do_popen'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:888:in `run_cmd_options'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:861:in `run_cmd'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcs.rb:1728:in `is_service_running?'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1182:in `block in load_current_node'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1181:in `each'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/cluster_entity.rb:1181:in `load_current_node'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:846:in `node_status'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:138:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/remote.rb:138:in `remote'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/pcsd.rb:154:in `block in <top (required)>'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1636:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1636:in `block in compile!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:987:in `block (3 levels) in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1006:in `route_eval'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:987:in `block (2 levels) in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1035:in `block in process_route'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1033:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1033:in `process_route'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:985:in `block in route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:984:in `each'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:984:in `route!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1098:in `block in dispatch!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `block in invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1095:in `dispatch!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:919:in `block in call!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `block in invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1072:in `invoke'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:919:in `call!'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:908:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/common_logger.rb:38:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:231:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/lib/pcsd/rserver.rb:32:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/xss_header.rb:18:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/path_traversal.rb:16:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/json_csrf.rb:26:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/base.rb:50:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-protection-2.0.8.1/lib/rack/protection/frame_options.rb:31:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:194:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1951:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1503:in `block in call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1730:in `synchronize'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/sinatra-2.0.8.1/lib/sinatra/base.rb:1503:in `call'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:86:in `block in pre_process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `catch'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:84:in `pre_process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/thin-1.7.2/lib/thin/connection.rb:50:in `block in process'
Jul 25 08:00:55 fastvm-rhel-8-0-23 pcsd[2415238]: #011/usr/share/gems/gems/eventmachine-1.2.7/lib/eventmachine.rb:1077:in `block in spawn_threadpool'
  • 500 errors are logged in /var/log/pcsd/pcsd.log:
E, [2020-07-25T08:00:55.952 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 1073.59ms
E, [2020-07-25T08:01:17.525 #01714]    ERROR -- : Failed to parse cib.
E, [2020-07-25T08:01:17.526 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 74.21ms
E, [2020-07-25T08:01:39.712 #00000]    ERROR -- : 500 POST /remote/cluster_status (192.168.22.23) 15.11ms
E, [2020-07-25T08:01:40.083 #00000]    ERROR -- : 500 GET /remote/status?version=2&operations=1 (192.168.22.24) 63.71ms

Environment

  • Red Hat Enterprise Linux 8 (with the High Availability Add-on)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content