Why does starting td-agent fails with an error "error=#<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/log/td-agent/tmp/fluentd-docker.pos>" ?
Issue
-
When combining https://access.redhat.com/documentation/en/openshift-enterprise/version-3.0/openshift-enterprise-30-administrator-guide/chapter-5-aggregating-container-logs with http://www.fluentd.org/guides/recipes/graylog2 to forward OpenShift log messages into a graylog instance the td-agent fails.
The servers don't have internet access to download rubygems so I downloaded the current version from https://rubygems.org manually. The versions seem to be incomaptible -
Error shown in the td-agent.log-
2015-08-24 13:39:51 +0200 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2015-08-24 13:39:51 +0200 [error]: unexpected error error_class=Errno::ENOENT error=#<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/log/td-agent/tmp/fluentd-docker.pos>
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:78:in `initialize'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:78:in `open'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:78:in `start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/root_agent.rb:111:in `block in start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/root_agent.rb:110:in `each'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/root_agent.rb:110:in `start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/engine.rb:201:in `start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/engine.rb:151:in `run'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:481:in `run_engine'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:140:in `block in start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:266:in `call'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:266:in `main_process'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:241:in `block in supervise'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:240:in `fork'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:240:in `supervise'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/supervisor.rb:134:in `start'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/command/fluentd.rb:167:in `<top (required)>'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/bin/fluentd:6:in `<top (required)>'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `load'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
2015-08-24 13:39:51 +0200 [error]: /usr/sbin/td-agent:7:in `load'
2015-08-24 13:39:51 +0200 [error]: /usr/sbin/td-agent:7:in `<main>'
2015-08-24 13:39:51 +0200 [info]: shutting down fluentd
2015-08-24 13:39:51 +0200 [warn]: unexpected error while shutting down input plugin plugin=Fluent::ForwardInput plugin_id="object:3feb10f69088" error_class=RuntimeError error=#<RuntimeError: loop not running>
2015-08-24 13:39:51 +0200 [warn]: unexpected error while shutting down input plugin plugin=Fluent::HttpInput plugin_id="object:3feb10eb158c" error_class=RuntimeError error=#<RuntimeError: loop not running>
2015-08-24 13:39:51 +0200 [error]: unexpected error error="no watchers for this loop"
2015-08-24 13:39:51 +0200 [error]: unexpected error error=#<RuntimeError: no watchers for this loop> error_class=RuntimeError
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:95:in `stop'
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:95:in `stop'
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:107:in `shutdown'
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/root_agent.rb:121:in `block (2 levels) in shutdown'
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:67:in `shutdown'
2015-08-24 13:39:51 +0200 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/root_agent.rb:121:in `block (2 levels) in shutdown'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:84:in `run'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:84:in `run'
2015-08-24 13:39:51 +0200 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:91:in `run'
2015-08-24 13:39:51 +0200 [info]: process finished code=0
2015-08-24 13:39:51 +0200 [warn]: process died within 1 second. exit.
- Steps executed:
yum -y localinstall td-agent-2.2.0-0.x86_64.rpm
/opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes-0.3.1.gem docker-api-1.22.2.gem excon-0.45.4.gem gelf-1.4.0.gem
mkdir -p /etc/td-agent/config.d
chown td-agent:td-agent /etc/td-agent/config.d
echo 'DAEMON_ARGS=' >> /etc/sysconfig/td-agent
echo 'TD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config"' >> /etc/sysconfig/td-agent
echo "@include config.d/*.conf" >> /etc/td-agent/td-agent.conf
cp out_gelf.rb /etc/td-agent/plugin/
systemctl enable td-agent
systemctl start td-agent
Environment
- Openshift 3.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.