Elasticsearch logs intermittently drop out in Kibana - connection is successful
Issue
Kibana is showing logs part of the time, but there are gaps that can last for a few hours at a time. These logs never appear. Restarting Elasticsearch and Kibana does not fix the issue, and neither does recreating the Kibana index or deploying a new build.
Kibana logs show this error:
{"type":"log","@timestamp":"2019-05-31T14:23:34Z","tags":["error","elasticsearch","admin"],"pid":225,"message":"Request error, retrying\nPOST https://logging-es:9200/.kibana/_search => socket hang up"}
There are no obvious connection or cert issues with Fluentd:
--Connectivity between logging-fluentd-bk4bl and elasticsearch
with ca
* About to connect() to logging-es port 9200 (#0)
* Trying 10.0.0.1...
* Connected to logging-es (10.0.0.1.) port 9200 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/fluent/keys/ca
CApath: none
* NSS: client certificate from file
* subject: CN=system.logging.fluentd,OU=OpenShift,O=Logging
* start date: Apr 18 01:03:53 2019 GMT
* expire date: Apr 17 01:03:53 2021 GMT
* common name: system.logging.fluentd
* issuer: CN=logging-signer-test
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: CN=logging-es,OU=OpenShift,O=Logging
* start date: Apr 18 01:04:18 2019 GMT
* expire date: Apr 17 01:04:18 2021 GMT
* common name: logging-es
* issuer: CN=logging-signer-test
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: logging-es:9200
> Accept: */*
>
Environment
- RedHat OpenShift Container Platform
- 3.11
- 4.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.