How to enable debug logs in Rados gateway in Red Hat Ceph Storage?

Solution Verified - Updated -

Environment

  • Red Hat Ceph Storage (RHCS)
  • Rados Gateway

Issue

  • How to enable debug logs in Rados gateway in Red Hat Ceph Storage?

Resolution

There are 3 methods to do this depending on the situation and RHCS version:

Method 1 - RHCS 4.x and earlier - Update the ceph.conf and restarting the RGW daemon:

  • Add the following under the rgw section of the /etc/ceph/ceph.conf file of Rados gateway node:

    debug ms = 1
debug rgw = 20

  • Restart the radosgw service:

    # sudo systemctl list-units | grep -i rgw
# sudo systemctl restart ceph-radosgw@<$service_name>.service

Method 2 - Inject the parameters to the RGW services using it's admin socket (.asok):

  • Inject the debug values:

    $ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.<name>.asok config set debug_rgw 20
$ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.<name>.asok config set debug_ms 1

  • Unset the injected values:

    ⓘ Do not set the default values. Instead unset the manually set configuration variables. Variables set via the admin socket take precedence over variables changed for example via the ceph configuration database. If a variable will be set to default values via the admin socket, future changes via the ceph config set command will not take effect and can be confusing in future troubleshooting sessions.

    $ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.<name>.asok config unset debug_rgw
$ ceph --admin-daemon /var/run/ceph/ceph-client.rgw.<name>.asok config unset debug_ms

  • The logs will be inside /var/log/ceph/ on the Rados Gateway node.

Method 3 - RHCS 5.x and later only - Set debug level in Ceph configuration database:

  • Verify existing debug values for debug_rgw and debug_ms:

    $ ceph config get client.rgw debug_rgw
$ ceph config get client.rgw debug_ms

  • Update debug parameter values:

    $ ceph config set client.rgw debug_rgw 20/5
$ ceph config set client.rgw debug_ms 1/5

  • Reset to default values after testing/debugging:

    $ ceph config rm client.rgw debug_rgw
$ ceph config rm client.rgw debug_ms

  • Logs for the RGW daemons reside in /var/log/ceph/<fsid>/ on the nodes hosting the RGW services.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments