Fluentd Pods Report Constant "Could not push log to Elasticsearch"
Issue
- Scaling up the number of fluentd pods, at somewhere between 100-150 fluentd pods, the fluentd logs start filling with frequent "Could not push to Elasticsearch" messages
2017-07-13 16:12:00 -0400 [error]: Could not push log to Elasticsearch: {"took"=>3, "errors"=>true, "items"=>[{"create"=>{"_index"=>".operations.2017.07.13", "_type"=>"com.redhat.viaq.common", "_id"=>"AV09lEt-yH
UkC3cmRhe-", "status"=>429, "error"=>{"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$4@1a34d37a on EsThreadPoolExecutor[bulk, queue capac
ity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@312a2162[Running, pool size = 32, active threads = 32, queued tasks = 50, completed tasks = 327053]]"}}}]}
2017-07-13 16:12:00 -0400 [error]: Could not push log to Elasticsearch: {"took"=>1, "errors"=>true, "items"=>[{"create"=>{"_index"=>".operations.2017.07.13", "_type"=>"com.redhat.viaq.common", "_id"=>"AV09lLjtBr
GY7t4hWyhu", "status"=>429, "error"=>{"type"=>"es_rejected_execution_exception", "reason"=>"rejected execution of org.elasticsearch.transport.TransportService$4@390823ea on EsThreadPoolExecutor[bulk, queue capac
ity = 50,
Environment
- Red Hat OpenShift Container Platform
- 3.5
- 3.6
- 3.7
- 3.9
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.