Ceph/ODF: MDS service crashes with "MDLog.cc: 651: FAILED ceph_assert(segments.size() >= pre_segments_size)"

Solution Verified - Updated -

Issue

Ceph/ODF: MDS service crashes with "MDLog.cc: 651: FAILED ceph_assert(segments.size() >= pre_segments_size)"

Example from ceph crash info:

sh-5.1$ ceph crash info 2024-05-05T05:23:33.483384Z_38907b28-03d5-4805-9080-69c982f4c381
{
    "assert_condition": "segments.size() >= pre_segments_size",
    "assert_file": "/builddir/build/BUILD/ceph-17.2.6/src/mds/MDLog.cc",
    "assert_func": "void MDLog::trim(int)",
    "assert_line": 651,
    "assert_msg": "/builddir/build/BUILD/ceph-17.2.6/src/mds/MDLog.cc: In function 'void MDLog::trim(int)' thread 7f9d3c208640 time 2024-05-05T05:23:33.482451+0000\n/builddir/build/BUILD/ceph-17.2.6/src/mds/MDLog.cc: 651: FAILED ceph_assert(segments.size() >= pre_segments_size)\n",
    "assert_thread_name": "safe_timer",
    "backtrace": [
        "/lib64/libc.so.6(+0x54db0) [0x7f9d428a7db0]",
        "/lib64/libc.so.6(+0xa154c) [0x7f9d428f454c]",
        "raise()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x188) [0x7f9d42f03ae1]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x142c45) [0x7f9d42f03c45]",
        "(MDLog::trim(int)+0xb06) [0x5611463c02a6]",
        "(MDSRankDispatcher::tick()+0x365) [0x561146142c65]",
        "ceph-mds(+0x11c71d) [0x56114611571d]",
        "(CommonSafeTimer<ceph::fair_mutex>::timer_thread()+0x15e) [0x7f9d42fed4ae]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x22cda1) [0x7f9d42fedda1]",
        "/lib64/libc.so.6(+0x9f802) [0x7f9d428f2802]",
        "/lib64/libc.so.6(+0x3f450) [0x7f9d42892450]"
    ],
    "ceph_version": "17.2.6-170.el9cp",
    "crash_id": "2024-05-05T05:23:33.483384Z_38907b28-03d5-4805-9080-69c982f4c381",
    "entity_name": "mds.ocs-storagecluster-cephfilesystem-b",
    "os_id": "rhel",
    "os_name": "Red Hat Enterprise Linux",
    "os_version": "9.3 (Plow)",
    "os_version_id": "9.3",
    "process_name": "ceph-mds",
    "stack_sig": "21cf82abf00a9a80ef194472005415a53e94d6965c4e910d756a9f711243f498",
    "timestamp": "2024-05-05T05:23:33.483384Z",
    "utsname_hostname": "rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-xxxxxxyyyyyy",
    "utsname_machine": "x86_64",
    "utsname_release": "5.14.0-284.52.1.el9_2.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Tue Jan 30 08:35:38 EST 2024"

Environment

Red Hat OpenShift Container Platform (OCP) 4.x
Red Hat OpenShift Container Storage (OCS) 4.x
Red Hat OpenShift Data Foundation (ODF) 4.x
Red Hat Ceph Storage (RHCS) 5.x
Red Hat Ceph Storage (RHCS) 6.x
Red Hat Ceph Storage (RHCS) 7.x
Ceph File System (CephFS)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content