Chapter 3. Troubleshooting Networking Issues
This chapter lists basic troubleshooting procedures connected with networking and Network Time Protocol (NTP).
3.1. Basic Networking Troubleshooting
Red Hat Ceph Storage depends heavily on a reliable network connection. Ceph nodes use the network for communicating with each other. Networking issues can cause many problems with OSDs, such as flapping OSD, or OSD incorrectly reported as
down. Networking issues can also cause Monitor
clock skew errors. In addition, packet loss, high latency, or limited bandwidth can impact the cluster performance and stability.
Procedure: Basic Networking Troubleshooting
Verify that the
public_networkparameters in the Ceph configuration file include correct values.
- Verify that the network interfaces are up. See the Basic Network troubleshooting solution on the Customer Portal for details.
- Verify that the Ceph nodes are able to reach each other using their short host names.
- If you use a firewall, ensure that Ceph nodes are able to reach other on their appropriate ports. See the Configuring Firewall section in the Red Hat Ceph Storage 3 Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu.
- Verify that there are no errors on the interface counters and that the network connectivity between hosts has expected latency and no packet loss. See the What is the "ethtool" command and how can I use it to obtain information about my network devices and interfaces and RHEL network interface dropping packets solutions on the Customer Portal for details.
For performance issues, in addition to the latency checks, also use the
iperfutility to verify the network bandwidth between all nodes of the cluster. For details, see the What are the performance benchmarking tools available for Red Hat Ceph Storage? solution on the Customer Portal.
- Ensure that all hosts have equal speed network interconnects, otherwise slow attached nodes could slow down the faster connected ones. Also, ensure that the inter switch links can handle the aggregated bandwidth of the attached nodes.
3.2. Basic NTP Troubleshooting
This section includes basic NTP troubleshooting steps.
Procedure: Basic NTP Troubleshooting
Verify that the
ntpddaemon is running on the Monitor hosts:
# systemctl status ntpd
ntpdis not running, enable and start it:
# systemctl enable ntpd # systemctl start ntpd
ntpdis synchronizing the clocks correctly:
$ ntpq -p
- See the How to troubleshoot NTP issues solution on the Red Hat Customer Portal for advanced NTP troubleshooting steps.