Is there a feature in Ceph to warn of clock SKEW on OSD daemons?

Solution In Progress - Updated -

Issue

  • Currently, Ceph only warns of clock SKEW between monitor nodes for PAXOS on the monitors to run. But other nodes such as OSD's can be impacted by clock skew's as well and can lead to customer impacting situations such as large amounts of flapping OSD's.

  • Clock sync is required for when cephx rotates keys and must be within the hour of rotation. If there is SKU, OSD logs will start to show :

2022-03-25 19:53:40.299 7f576e5bd700  0 auth: could not find secret_id=3868
2022-03-25 19:53:40.299 7f576e5bd700  0 cephx: verify_authorizer could not get service secret for service osd secret_id=3868
  • A customer replaced an OSD node and the node had the incorrect time set. After about an hour the customer noticed a severe performance degradation and OSD's flapping throughout the cluster.

Environment

  • Red Hat Ceph Storage 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content