Ceph upgrade failed at TASK [ceph-dashboard : inject grafana dashboard layouts]

Issue

• Ceph upgrade failed at TASK [ceph-dashboard : inject grafana dashboard layouts] with error "Could not connect to Grafana server" :

TASK [ceph-dashboard : inject grafana dashboard layouts] **
Thursday 12 May 2022  22:07:03 +0800 (0:00:00.749)       0:23:13.771 **
fatal: [cephnode1 -> cephnode1]: FAILED! => changed=false
  cmd:
  - docker
  - exec
  - ceph-mon-cephnode1
  - ceph
  - --cluster
  - ceph
  - dashboard
  - grafana
  - dashboards
  - update
  delta: '0:00:15.526389'
  end: '2022-05-12 22:07:19.642968'
  msg: non-zero return code
  rc: 22
  start: '2022-05-12 22:07:04.116579'
  stderr: |-
    Error EINVAL: Traceback (most recent call last):
      File "/lib/python3.6/site-packages/urllib3/connection.py", line 162, in _new_conn
        (self._dns_host, self.port), self.timeout, **extra_kw)
      File "/lib/python3.6/site-packages/urllib3/util/connection.py", line 57, in create_connection
        for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
      File "/lib64/python3.6/socket.py", line 745, in getaddrinfo
        for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
    socket.gaierror: [Errno -2] Name or service not known

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
        chunked=chunked)
      File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 343, in _make_request
        self._validate_conn(conn)
      File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 839, in _validate_conn
        conn.connect()
      File "/lib/python3.6/site-packages/urllib3/connection.py", line 315, in connect
        conn = self._new_conn()
      File "/lib/python3.6/site-packages/urllib3/connection.py", line 171, in _new_conn
        self, "Failed to establish a new connection: %s" % e)
    urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f89eb92f5f8>: Failed to establish a new connection: [Errno -2] Name or service not known

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
        timeout=timeout
      File "/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
        _stacktrace=sys.exc_info()[2])
      File "/lib/python3.6/site-packages/urllib3/util/retry.py", line 399, in increment
        raise MaxRetryError(_pool, url, error or ResponseError(cause))
    urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ansibleundefined', port=3000): Max retries exceeded with url: /api/dashboards/db (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f89eb92f5f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/share/ceph/mgr/dashboard/grafana.py", line 51, in push_dashboard
        verify=Settings.GRAFANA_API_SSL_VERIFY
      File "/lib/python3.6/site-packages/requests/api.py", line 116, in post
        return request('post', url, data=data, json=json, **kwargs)
      File "/lib/python3.6/site-packages/requests/api.py", line 60, in request
        return session.request(method=method, url=url, **kwargs)
      File "/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
        resp = self.send(prep, **send_kwargs)
      File "/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
        r = adapter.send(request, **kwargs)
      File "/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
        raise ConnectionError(e, request=request)
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ansibleundefined', port=3000): Max retries exceeded with url: /api/dashboards/db (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f89eb92f5f8>: Failed to establish a new connection: [Errno -2] Name or service not known',))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/share/ceph/mgr/mgr_module.py", line 1022, in _handle_command
        return self.handle_command(inbuf, cmd)
      File "/usr/share/ceph/mgr/dashboard/module.py", line 387, in handle_command
        push_local_dashboards()
      File "/usr/share/ceph/mgr/dashboard/grafana.py", line 133, in push_local_dashboards
        retry()
      File "/usr/share/ceph/mgr/dashboard/grafana.py", line 90, in __call__
        result = self.func(self.args, *self.kwargs)
      File "/usr/share/ceph/mgr/dashboard/grafana.py", line 128, in push
        grafana.push_dashboard(body)
      File "/usr/share/ceph/mgr/dashboard/grafana.py", line 54, in push_dashboard
        raise GrafanaError("Could not connect to Grafana server")
    dashboard.exceptions.GrafanaError: Could not connect to Grafana server
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>

• Run command "ceph dashboard grafana dashboards update" directly failed with the same error