Ceph excessive debug logs in /var/log/messages fill up log file system

Solution In Progress - Updated -

Issue

  • The /var/log/messages file on Ceph nodes grows rapidly with [DBG]messages written every two seconds, eventually filling up the file system.

    Feb 11 13:56:19 st1001 conmon[4055351]: debug 2022-02-11T02:56:19.249+0000 7f11828cd700  0 log_channel(cluster) log [DBG] : pgmap v345081: 33 pgs: 1 active+clean+scrubbing+deep, 32 active+clean; 134 GiB data, 20 TiB used, 349 TiB / 369 TiB avail; 1.7 KiB/s wr, 0 op/s
    Feb 11 13:56:19 st1001 conmon[797842]: :56:19.251166+0000 mgr.st1001.wodfwm (mgr.1391964) 375206 : cluster [DBG] pgmap v345081: 33 pgs: 1 active+clean+scrubbing+deep, 32 active+clean; 134 GiB data, 20 TiB used, 349 TiB / 369 TiB avail; 1.7 KiB/s wr, 0 op/s
    
  • These messages appear despite setting the cluster monitor log level to info.

    # ceph config set mon mon_cluster_log_file_level info
    

Environment

  • Red Hat Ceph Storage 5.0
  • Red Hat Enterprise Linux (RHEL) 8.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content