Forwarding logs to Elasticsearch fails with unknown parameter type error in RHOCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Azure RedHat OpenShift 4.x
  • Red Hat OpenShift Logging (RHOL)
    • 5
  • Elasticsearch 8
  • Fluentd
  • Vector

Issue

  • Is supported to forward the logs to Elasticsearch 8?

  • Fluentd fails to forward the logs to Elasticsearch with error:

    2022-04-25 06:24:05 +0000 [warn]: [elasticsearch-secure] failed to flush the buffer. retry_time=36 next_retry_seconds=2022-04-25 06:25:02 +0000 chunk="5dd7459d6d36c1e0875d29c091e3947f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.example.com\", :port=>9200, :scheme=>\"https\", :user=>\"user\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}"
  • Vector fails log forwarding to Elasticsearch with unknown parameter [_type]

Resolution

Red Hat reviewed the feature request in RFE OBSDA-161 and delivered a fix in:

Root Cause

In Elasticsearch 8 was removed the mapping types. Then, it's not supported to indicate the type elasticsearch as it was in previous versions.

This mapping type is used in the fluentd/vector configuration when forwarding the logs to one Elasticsearch output, then, the current configuration done in RHOL for the fluentd is not supported more when the output is Elasticsearch 8.

Also, it's indicated in the RHOCP documentation the Supported log data output types in OpenShift Logging 5 where it's not included Elasticsearch 8.

Diagnostic Steps

Check in the fluentd pod logs that is visible the error Action/metadata line [1] contains an unknown parameter [_type].

In newer versions of RHOL:

$ oc logs <collector pod> -c collector|grep "contains an unknown parameter" 
2022-04-25 06:24:05 +0000 [warn]: [elasticsearch-secure] failed to flush the buffer. retry_time=36 next_retry_seconds=2022-04-25 06:25:02 +0000 chunk="5dd7459d6d36c1e0875d29c091e3947f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.example.com\", :port=>9200, :scheme=>\"https\", :user=>\"user\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}"

In older versions of RHOL:

$ oc logs <fluentd pod> -c fluentd|grep "contains an unknown parameter" 
2022-04-25 06:24:05 +0000 [warn]: [elasticsearch-secure] failed to flush the buffer. retry_time=36 next_retry_seconds=2022-04-25 06:25:02 +0000 chunk="5dd7459d6d36c1e0875d29c091e3947f" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.example.com\", :port=>9200, :scheme=>\"https\", :user=>\"user\", :password=>\"obfuscated\"}): [400] {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments