Elasticsearch logs intermittently drop out in Kibana - connection is successful

Solution Verified - Updated -

Issue

Kibana is showing logs part of the time, but there are gaps that can last for a few hours at a time. These logs never appear. Restarting Elasticsearch and Kibana does not fix the issue, and neither does recreating the Kibana index or deploying a new build.

Kibana logs show this error:

{"type":"log","@timestamp":"2019-05-31T14:23:34Z","tags":["error","elasticsearch","admin"],"pid":225,"message":"Request error, retrying\nPOST https://logging-es:9200/.kibana/_search => socket hang up"}

There are no obvious connection or cert issues with Fluentd:

--Connectivity between logging-fluentd-bk4bl and elasticsearch
  with ca
* About to connect() to logging-es port 9200 (#0)
*   Trying 10.0.0.1...
* Connected to logging-es (10.0.0.1.) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/fluent/keys/ca
  CApath: none
* NSS: client certificate from file
*   subject: CN=system.logging.fluentd,OU=OpenShift,O=Logging
*   start date: Apr 18 01:03:53 2019 GMT
*   expire date: Apr 17 01:03:53 2021 GMT
*   common name: system.logging.fluentd
*   issuer: CN=logging-signer-test
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: CN=logging-es,OU=OpenShift,O=Logging
*   start date: Apr 18 01:04:18 2019 GMT
*   expire date: Apr 17 01:04:18 2021 GMT
*   common name: logging-es
*   issuer: CN=logging-signer-test
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: logging-es:9200
> Accept: */*
>

Environment

OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In