Configuration failed because of ClusterBlockException blocked by FORBIDDEN index read-only

Issue

• Elasticsearch is green, but new logs are not being sent

• Fluentd failing with failed to flush the buffer:

  2021-02-25 17:58:26 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=9 next_retry_seconds=2021-02-25 18:03:04 +0000 chunk="5bb77d12307578cb980fdb14a5b712c7" error_class=Fluent::Plugin::ElasticsearchOutput::RetryStreamEmitFailure error="buffer is full."
  2021-02-25 17:58:26 +0000 [warn]: suppressed same stacktrace
  2021-02-25 17:58:29 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=10 next_retry_seconds=2021-02-25 18:03:46 +0000 chunk="5bb77d0220422cb4260d78c120cdf9ba" error_class=Fluent::Plugin::ElasticsearchOutput::RetryStreamEmitFailure error="buffer is full."
  

• Elasticsearch logs are getting forbidden messages:

  2021-02-25T22:16:58.462928658Z Populate config from /opt/app-root/src/sgconfig/
  2021-02-25T22:16:58.462928658Z Will update 'security/config' with /opt/app-root/src/sgconfig/config.yml 
  2021-02-25T22:16:58.521640896Z    FAIL: Configuration for 'config' failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (API)];]
  2021-02-25T22:16:58.521640896Z Will update 'security/roles' with /opt/app-root/src/sgconfig/roles.yml 
  2021-02-25T22:16:58.538709754Z    FAIL: Configuration for 'roles' failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (API)];]
  2021-02-25T22:16:58.538709754Z Will update 'security/rolesmapping' with /opt/app-root/src/sgconfig/roles_mapping.yml 
  2021-02-25T22:16:58.542602629Z    FAIL: Configuration for 'rolesmapping' failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (API)];]
  2021-02-25T22:16:58.542602629Z Will update 'security/internalusers' with /opt/app-root/src/sgconfig/internal_users.yml 
  2021-02-25T22:16:58.548608581Z    FAIL: Configuration for 'internalusers' failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (API)];]
  2021-02-25T22:16:58.548608581Z Will update 'security/actiongroups' with /opt/app-root/src/sgconfig/action_groups.yml         2021-02-25T22:16:58.54864276Z 
  2021-02-25T22:16:58.555262054Z    FAIL: Configuration for 'actiongroups' failed because of ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (API)];]
  2021-02-25T22:16:58.575692979Z Done with failures
  

• How to manually rollover Elasticsearch index?