Ceph: "ceph fs perf stats" returns no metrics, empty or zero values

Solution Verified - Updated -

Issue

The command ceph fs perf stats returns no metrics, empty or zero values

Real world example:

Linux-Prompt# ceph fs perf stats
{"version": 2, "global_counters": ["cap_hit", "read_latency", "write_latency", "metadata_latency", "dentry_lease", "opened_files", "pinned_icaps", "opened_inodes", "read_io_sizes", "write_io_sizes", "avg_read_latency", "stdev_read_latency", "avg_write_latency", "stdev_write_latency", "avg_metadata_latency", "stdev_metadata_latency"], "counters": [], "client_metadata": {"cephfs": {"client.455589863": {"hostname": "node.mydomian.fr", "root": "/xxx/infra/pod", "mount_point": "N/A", "IP": "v1:10.xxx.yyy.197"}}}, "global_metrics": {"cephfs": {"client.455589863": [[0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0], [0, 0]]}}, "metrics": {"delayed_ranks": [], "mds.0": {"client.250909022": [], "client.250909220": [], "client.250909322": [], "client.408451245": [], "client.417148405": [], "client.417167859": [], "client.417168402": [], "client.418624062": [], "client.426968587": [], "client.437329177": [], "client.437340530": [], "client.437340533": [], "client.437340536": [], "client.437356386": [], "client.437356389": [], "client.437356392": [], "client.437478619": [], "client.446964984": [], "client.446965149": [], "client.446965362": [], "client.447627519": [], "client.447627570": [], "client.449114458": [], "client.449219478": [], "client.449219808": [], "client.449225628": [], "client.449226567": [], "client.449229099": [], "client.449229672": [], "client.449230548": [], "client.449235630": [], "client.450093806": [], "client.452885935": [], "client.452939376": [], "client.453175110": [], "client.453177607": [], "client.453635941": [], "client.453635944": [], "client.453677508": [], "client.453763087": [], "client.453819859": [], "client.453918955": [], "client.453938583": [], "client.454022345": [], "client.454022582": [], "client.454040119": [], "client.454111093": [], "client.454111096": [], "client.454431952": [], "client.454432477": [], "client.454458810": [], "client.454458813": [], "client.454462851": [], "client.454673390": [], "client.454673393": [], "client.455434600": [], "client.455434603": [], "client.455460352": [], "client.455467746": [], "client.455467749": [], "client.455545351": [], "client.455546530": [], "client.455549149": [], "client.455559413": [], "client.455560393": [], "client.455562934": [], "client.455564389": [], "client.455565559": [], "client.455566660": [], "client.455569255": [], "client.455582814": [], "client.455585385": [], "client.455586678": [], "client.455589525": [], "client.455589863": [], "client.455595930": [], "client.455598213": []}, "mds.1": {"client.250909022": [], "client.250909220": [], "client.250909322": [], "client.408451245": [], "client.417148405": [], "client.417167859": [], "client.417168402": [], "client.418624062": [], "client.426968587": [], "client.437329177": [], "client.437340530": [], "client.437340533": [], "client.437340536": [], "client.437356386": [], "client.437356389": [], "client.437356392": [], "client.437478619": [], "client.446964984": [], "client.446965149": [], "client.446965362": [], "client.447627519": [], "client.447627570": [], "client.449114458": [], "client.449219478": [], "client.449219808": [], "client.449225628": [], "client.449226567": [], "client.449229099": [], "client.449229672": [], "client.449230548": [], "client.449235630": [], "client.450093806": [], "client.452885935": [], "client.452939376": [], "client.453175110": [], "client.453177607": [], "client.453635941": [], "client.453635944": [], "client.453677508": [], "client.453763087": [], "client.453819859": [], "client.453918955": [], "client.453938583": [], "client.454022345": [], "client.454022582": [], "client.454040119": [], "client.454111093": [], "client.454111096": [], "client.454431952": [], "client.454432477": [], "client.454458810": [], "client.454458813": [], "client.454462851": [], "client.454673390": [], "client.454673393": [], "client.455434600": [], "client.455434603": [], "client.455460352": [], "client.455467746": [], "client.455467749": [], "client.455545351": [], "client.455546530": [], "client.455549149": [], "client.455559413": [], "client.455560393": [], "client.455562934": [], "client.455564389": [], "client.455565559": [], "client.455566660": [], "client.455569255": [], "client.455582814": [], "client.455585385": [], "client.455586678": [], "client.455589525": [], "client.455589863": [], "client.455595930": [], "client.455598213": []}}}

The same processed to pretty print JSON data:
(Both commands work the same)

Linux-Prompt# ceph fs perf stats | python3 -m json.tool
Linux-Prompt# ceph fs perf stats | jq
{
    "version": 2,
    "global_counters": [
        "cap_hit",
        "read_latency",
        "write_latency",
        "metadata_latency",
        "dentry_lease",
        "opened_files",
        "pinned_icaps",
        "opened_inodes",
        "read_io_sizes",
        "write_io_sizes",
        "avg_read_latency",
        "stdev_read_latency",
        "avg_write_latency",
        "stdev_write_latency",
        "avg_metadata_latency",
        "stdev_metadata_latency"
    ],
    "counters": [],
    "client_metadata": {
        "cephfs": {
            "client.455589863": {
                "hostname": "node.mydomian.fr",
                "root": "/xxx/infra/pod",
                "mount_point": "N/A",
                "IP": "v1:10.xx.yy.197"
            }
        }
    },
    "global_metrics": {
        "cephfs": {
            "client.455589863": [
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ],
                [
                    0,
                    0
                ]
            ]
        }
    },
    "metrics": {
        "delayed_ranks": [],
        "mds.0": {
            "client.250909022": [],
            "client.250909220": [],
            "Most Clients Removed": [],
            "client.455595930": [],
            "client.455598213": []
        },
        "mds.1": {
            "client.250909022": [],
            "client.250909220": [],
            "Most Clients Removed": [],
            "client.455595930": [],
            "client.455598213": []
        }
    }
}

Environment

  • Red Hat Ceph Storage (RHCS) 5.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content