Elasticsearch logs intermittently drop out in Kibana - connection is successful

Solution Verified - Updated -

Issue

Kibana is showing logs part of the time, but there are gaps that can last for a few hours at a time. These logs never appear. Restarting Elasticsearch and Kibana does not fix the issue, and neither does recreating the Kibana index or deploying a new build.

Kibana logs show this error:

{"type":"log","@timestamp":"2019-05-31T14:23:34Z","tags":["error","elasticsearch","admin"],"pid":225,"message":"Request error, retrying\nPOST https://logging-es:9200/.kibana/_search => socket hang up"}

There are no obvious connection or cert issues with Fluentd:

--Connectivity between logging-fluentd-bk4bl and elasticsearch
  with ca
* About to connect() to logging-es port 9200 (#0)
*   Trying 10.0.0.1...
* Connected to logging-es (10.0.0.1.) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/fluent/keys/ca
  CApath: none
* NSS: client certificate from file
*   subject: CN=system.logging.fluentd,OU=OpenShift,O=Logging
*   start date: Apr 18 01:03:53 2019 GMT
*   expire date: Apr 17 01:03:53 2021 GMT
*   common name: system.logging.fluentd
*   issuer: CN=logging-signer-test
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=logging-es,OU=OpenShift,O=Logging
*   start date: Apr 18 01:04:18 2019 GMT
*   expire date: Apr 17 01:04:18 2021 GMT
*   common name: logging-es
*   issuer: CN=logging-signer-test
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: logging-es:9200
> Accept: */*
>

Environment

  • RedHat OpenShift Container Platform
    • 3.11
    • 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content