[Ceph] custom grafana certificate not being used after RHCS 7 upgrade

Resolution

The storage format and config key of the certificate has changed. RHCS7 & later versions use different keys: mgr/cephadm/cert_store.cert.grafana_cert and mgr/cephadm/cert_store.key.grafana_key which are now stored in a json format. Certificates which are stored in mgr/cephadm/grafana_crt and mgr/cephadm/grafana_key are not used anymore.

The new json format looks similar to the following examples (we assume a host name of 'grafana-hostname').

• Certificate:

  {
    "grafana-hostname": {
      "cert": "-----BEGIN CERTIFICATE-----\nMIIDFTCCAf0CFHlMUaJ3V886wJAD5NbQr8d8pufAMA0GCSqGSIb3DQEBCwUAMEcx\nCzAJBgNVBAYTAkFUMQ8wDQYDVQQIDAZTdHlyaWExEjAQBgNVBAoMCUFDTUUgQ29y\ncDETMBEGA1UEAwwKTW9uaXRvcmluZzAeFw0yNTAyMjgxMDM5MjFaFw0yNzExMjYx\nMDM5MjFaMEcxCzAJBgNVBAYTAkFUMQ8wDQYDVQQIDAZTdHlyaWExEjAQBgNVBAoM\nCUFDTUUgQ29ycDETMBEGA1UEAwwKTW9uaXRvcmluZzCCASIwDQYJKoZIhvcNAQEB\nBQADggEPADCCAQoCggEBAK1jxxqrDD2bIYP2JW7fGGGwaBS7QjZ1cjuQ4jZdvbk0\n322R4ov0ixe4j1tqWnqle8YkRND828ebMr5cR+WHtJa0Rl2NySBQ79esIY4t0Y/V\naI3w5792wpNFFSEO+SCegiomWUcmXvuLUsVTEsaf/njQy6iiQywpcaUGS245RvNv\nXheKcO3y8DBo7rbOFu27+hbiRPdzCw09QZo4ALWKdNCehQnIOyoakDmUsYiaqXIt\nyqD5A3CHuZiswA7CEOAS7extxf5JDJtFF0iBnmaGMe2OpIfJZ6U7z/dTa40tNo8A\nKIOx6QsP4NWf13tT4JZ+ZK+WqOuZ6sSecTFHHXCy/SkCAwEAATANBgkqhkiG9w0B\nAQsFAAOCAQEAlBrLlmX7+dYfXzprHPE2ZUdoVJ6wj2a3PJmu4uadDB3AUDTGZq6j\ndaWyDWYx6XM0ZJRZ2Bq9vlY3DN63cIJ8mGTJ7MULc+cGFkiD1t5wZkVuJSR5qQuM\nlLoFtMhUKpf15VNkgJUH83nu9Dj8ePGMHJkoHk7n/9aYqy/eZ2nCWuBOoWFm9o6M\niosAS2Wfb67idrZLHEiw49KZE9owqX7PAY8J8o6/vAJtMiHYyl786LbL572vO/4G\nStIsRyKz7IhFedBUbpItANElbA87b7mNifyY3XUxC6DggkfY4ZEo5C8nMV8/S2X2\nqAYGKKu8MFAili5Z0ZO/JtBSS6DiCn5bbQ==\n-----END CERTIFICATE-----\n",
      "user_made": true
    }
  }
  

• Key:

  {
    "grafana-hostname": {
      "key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCtY8caqww9myGD\n9iVu3xhhsGgUu0I2dXI7kOI2Xb25NN9tkeKL9IsXuI9balp6pXvGJETQ/NvHmzK+\nXEflh7SWtEZdjckgUO/XrCGOLdGP1WiN8Oe/dsKTRRUhDvkgnoIqJllHJl77i1LF\nUxLGn/540MuookMsKXGlBktuOUbzb14XinDt8vAwaO62zhbtu/oW4kT3cwsNPUGa\nOAC1inTQnoUJyDsqGpA5lLGImqlyLcqg+QNwh7mYrMAOwhDgEu3sbcX+SQybRRdI\ngZ5mhjHtjqSHyWelO8/3U2uNLTaPACiDsekLD+DVn9d7U+CWfmSvlqjrmerEnnEx\nRx1wsv0pAgMBAAECggEAEBi825HQ0njQAxhTOBBjyvDlrrQGNPhRBSzhhU7NTc/1\n5I4IMDpA/fAjNedBFFT/0UCdd2HcFPMoYjH+tjUzm5CLnkSLDRIafAUVveVsyKu1\nF8VLPHP94w5lJGMnl2LDvn4+KEvsgjF4WwuH6RLiHMz0dTiwH/3OeDWmA8rzNmbj\nUoVcuP2EhBRkXhuB/ldoRSKkHUaOSW6esZ+6HYwD9PwEiHuClkLkFZzI0+O7ou2C\nXGz7Bbm+cHSqn2K84JJKuc/7P0r0chdyVqG+kK40bv0uICYGpN/B7guRlVlKBy8K\nSwyaspVUGMT9gc9O8Nx6CvHnJ6GwVULl+0WOPDfzAQKBgQDuvqzGTdESWXQ+loUo\nuHJFcDIW/gRxocfyEDHGwT2LtkBan3WfoRHqdoM7ApeMrRhuDNsu1FI5Dwd/7ftk\nMowJ2rcFQeMY8kQghV5bNnGSN0gT/TnCHysOL9ruJQRYkwDZJQ2qZ0eV2F0lRzOZ\noF75dHszO/4UMOttE8OiEJ53AQKBgQC56+J6s3o7j7v8kFHDKi229qtUi+ImXIps\nMuxwZosxx67SHLB3nzWKEfJQHat8jP0yi/MlV2sLBP/dWyMYcb/0mH93gXIN+ins\nXFny2cp4KYo+1eHxC4rNr0bQGgNEKIWpdnNnfl6tlNlIRKNw2M8qtaRnmsbeTi3G\nczDVG6/uKQKBgQCZJNiBT/NXuOf3Keh6yW4Lmr+WvhStq2AtpzO20cgf8aVwPC9g\nWRq78o+Vb8E07Ofp1/dVH78qeLiP2GFU0ZPA0npqNext2SHYVpTDH7ZUW3+xytKc\n8g9VR19BOZ+DDbESmVEdyKhDt4lwWy7/Ub76dzx8DfSakRLDYlUwhUaRAQKBgGKa\nFFnot6nA9/zheUtdIjOHrKPBMCYSLCNn+nc+GxJCiZfLOVenqxhXnkAZDCqZB2t4\nR3KFYzLL6vq1a+553UjWW7vAfgGlq/g7nkSqdrO1rCjY++P/sBSXoaGaOiM6rnCK\nbnfz0DiU2Yt3Q4sEoRJQlSmRL2kU4zf9PWe4MFzJAoGAUYVoSZoyfgwanCtB2PCG\n0Cz9NyCcXTfjoUtIYxwL5jl7GKGGCBfIBxz4r6F7DA2U7GvyA5ZIZynjmAqV51LF\nwSOlOAI6DX7nhdm/dsAeAI0ZTRjI+mDxNqPcGAfeb9xHGU6SxVJPI0FBZ8nKkMDt\nlYL6G3f8qPFWSkhk7o6AI/Y=\n-----END PRIVATE KEY-----\n",
      "user_made": true
    }
  }
  

To help generate the correct json certificate store format you can use the attached script (which can be downloaded from either the bottom of this KCS or from the upstream github link from the artifacts table). If you have more than one certificate or key you can execute the script multiple times and the certificates or keys will be added to the store.

Update Grafana certificate(s)

1. Create a file which only contains the certificate content, for example:

   -----BEGIN CERTIFICATE-----
   MIIFCDCCAvCgAwIBAgIUcttQRwsQ3MialkNhMsB61uegdC8wDQYJKoZIhvcNAQEL
   [...]
   13dN3wVQEw9y/8jTphPhnvjphNSOlcS49/78n23X2ayjESoZucSsj1mYHcg=
   -----END CERTIFICATE-----
   

2. Run the script as following:

   ⓘ If you need multiple certificates and hosts to be added run the script as often as needed, it will update the /path/to/grafana_certs.json file accordingly.

   # python3 ./update_grafana_certs.py cert <host-name> </path/to/cert-file> </path/to/grafana_certs.json>
   

3. Update the cert_store.cert.grafana_cert setting to add the new certificates:

   # ceph config-key set mgr/cephadm/cert_store.cert.grafana_cert -i /path/to/grafana_certs.json
   

4. You can review the setting to make sure it has the correct certificate set:

   # ceph config-key get mgr/cephadm/cert_store.cert.grafana_cert
   

Update Grafana key(s)

1. Create a file which only contains the key content, for example:

   -----BEGIN RSA PRIVATE KEY-----
   MIIJKQIBAAKCAgEA0j41QbO0emj3QLfJI4Pl8cc1YcJurs4tuYVDGnBKchDjaVOC
   [...]
   13dN3wVQEw9y/8jTphPhnvjphNSOlcS49/78n23X2ayjESoZucSsj1mYHcg=
   -----END RSA PRIVATE KEY-----
   

2. Run the script as following:

   ⓘ If you need multiple keys and hosts to be added run the script as often as needed, it will update the /path/to/grafana_keys.json file accordingly.

   # python3 ./update_grafana_certs.py key <your-host-name> </path/to/cert-file> </path/to/grafana_keys.json>
   

3. Update the cert_store.key.grafana_key setting to add the new keys:

   # ceph config-key set mgr/cephadm/cert_store.key.grafana_key -i /path/to/grafana_keys.json
   

4. To review the setting execute:

   # ceph config-key get mgr/cephadm/cert_store.key.grafana_key
   

Apply the new certificates and keys

1. Force the MGR cephadm module to recognize the new cert-store content:

   # ceph mgr fail
   

2. Reconfigure Grafana daemon after the step above has been run:

   ⓘ To find the correct grafana daemon name use ceph orch ps.

   # ceph orch daemon reconfig <grafana-daemon-name>
   

   Alternativly you can execute the following command to reconfigure all grafana daemons:

   # ceph orch reconfig grafana