Red Hat Training

A Red Hat training course is available for Red Hat Ceph Storage

Chapter 3. Troubleshooting Networking Issues

This chapter lists basic troubleshooting procedures connected with networking and Network Time Protocol (NTP).

3.1. Basic Networking Troubleshooting

Red Hat Ceph Storage depends heavily on a reliable network connection. Ceph nodes use the network for communicating with each other. Networking issues can cause many problems with OSDs, such as flapping OSD, or OSD incorrectly reported as down. Networking issues can also cause Monitor clock skew errors. In addition, packet loss, high latency, or limited bandwidth can impact the cluster performance and stability.

Procedure: Basic Networking Troubleshooting

  1. Verify that the cluster_network and public_network parameters in the Ceph configuration file include correct values.
  2. Verify that the network interfaces are up. See the Basic Network troubleshooting solution on the Customer Portal for details.
  3. Verify that the Ceph nodes are able to reach each other using their short host names. See the Setting DNS Name Resolution section in the Red Hat Ceph Storage 2 Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu.
  4. If you use a firewall, ensure that Ceph nodes are able to reach other on their appropriate ports. See the Configuring Firewall section in the Red Hat Ceph Storage 2 Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu.
  5. Verify that there are no errors on the interface counters and that the network connectivity between hosts has expected latency and no packet loss. See the What is the "ethtool" command and how can I use it to obtain information about my network devices and interfaces and RHEL network interface dropping packets solutions on the Customer Portal for details.
  6. For performance issues, in addition to the latency checks, also use the iperf utility to verify the network bandwidth between all nodes of the cluster. For details, see the What are the performance benchmarking tools available for Red Hat Ceph Storage? solution on the Customer Portal.
  7. Ensure that all hosts have equal speed network interconnects, otherwise slow attached nodes could slow down the faster connected ones. Also, ensure that the inter switch links can handle the aggregated bandwidth of the attached nodes.

See Also

3.2. Basic NTP Troubleshooting

This section includes basic NTP troubleshooting steps.

Procedure: Basic NTP Troubleshooting

  1. Verify that the ntpd daemon is running on the Monitor hosts: 

    # systemctl status ntpd

  2. If ntpd is not running, enable and start it: 

    # systemctl enable ntpd
# systemctl start ntpd

  3. Ensure that ntpd is synchronizing the clocks correctly: 

    $ ntpq -p
  4. See the How to troubleshoot NTP issues solution on the Red Hat Customer Portal for advanced NTP troubleshooting steps.

See Also