Fluentd JSON parsing fails when there is numeric output

Issue

• In OpenShift Logging, when enabling JSON logging, some structured JSON logs are not parsed correctly and are not shown in in Grafana.

• The collector Pod is showing the following "Rejected by Elasticsearch" error for a JSON log message ({"test": 1234}):

  $ oc logs collector-tfvc7 -c collector 
  [..]
  2022-05-31 07:27:28 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.containers.fedora-5b4f76ffff-pg4bg_example-project_fedora-5953dc2b505e7424f1b803ddfdf2720c935918cd61e0620ccedd9dc42668bea9.log" time=2022-05-31 07:27:27.385839853 +0000 record={"docker"=>{"container_id"=>"5953dc2b505e7424f1b803ddfdf2720c935918cd61e0620ccedd9dc42668bea9"}, "kubernetes"=>{"container_name"=>"fedora", "namespace_name"=>"example-project", "pod_name"=>"fedora-5b4f76ffff-pg4bg", "container_image"=>"registry.fedoraproject.org/fedora@sha256:63fa666e22333d85eaa6684fba51efe4b48dc5f5ff7b194cdc1fa1f1290b5c32", "container_image_id"=>"registry.fedoraproject.org/fedora@sha256:63fa666e22333d85eaa6684fba51efe4b48dc5f5ff7b194cdc1fa1f1290b5c32", "pod_id"=>"737b0828-97a9-4ade-91b1-fc449e3924b0", "pod_ip"=>"10.128.2.123", "host"=>"ip-10-0-144-2.eu-central-1.compute.internal", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"8747ef26-f3ef-42a5-b2d2-4601807d0ae7", "namespace_labels"=>{"owner"=>"example-user", "kubernetes_io/metadata_name"=>"example-project"}, "flat_labels"=>["deployment=fedora", "pod-template-hash=5b4f76ffff"]}, "message"=>"{\"test\":1234}", "level"=>"unknown", "hostname"=>"ip-10-0-144-2.eu-central-1.compute.internal", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.0.144.2", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-05-31T07:27:27.385307+00:00", "version"=>"1.7.4 1.6.0"}}, "openshift"=>{"sequence"=>513}, "@timestamp"=>"2022-05-31T07:27:26.276153+00:00", "viaq_index_name"=>"app-nologformat-write", "viaq_msg_id"=>"YmQyYzNjYjItYWNlOS00YTM2LTljNDUtYzgzODI2ODEzOWM1", "log_type"=>"application", "structured"=>{"test"=>1234}}