Feb 12 03:34:01 kcsandnp01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2198" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 12 03:34:03 kcsandnp01 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Feb 12 03:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 03:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 03:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 03:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 03:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 03:34:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3192: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 03:34:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3192 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 03:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 03:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 03:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 03:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 03:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 03:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 03:49:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3193: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 03:49:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3193 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 03:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 04:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 04:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 04:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 04:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 04:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 04:04:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3194: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 04:04:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3194 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 04:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 04:19:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 04:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 04:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 04:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 04:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 04:19:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3195: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 04:19:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3195 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 04:19:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 04:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 04:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 04:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 04:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 04:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 04:34:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3196: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 04:34:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3196 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 04:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 04:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 04:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 04:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 04:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 04:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 04:49:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3197: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 04:49:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3197 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 04:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 05:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 05:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 05:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 05:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 05:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 05:04:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3198: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 05:04:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3198 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 05:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 05:19:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 05:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 05:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 05:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 05:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 05:19:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3199: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 05:19:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3199 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 05:19:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 05:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 05:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 05:34:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 05:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 05:34:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 05:34:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3200: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 05:34:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3200 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 05:34:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 05:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 05:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 05:49:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 05:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 05:49:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 05:49:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3201: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 05:49:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3201 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 05:49:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 06:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 06:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 06:04:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 06:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 06:04:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 06:04:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3202: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 06:04:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3202 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 06:04:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 06:19:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 06:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 06:19:44 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 06:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 06:19:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 06:19:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3203: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 06:19:44 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3203 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 06:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 06:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 06:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 06:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 06:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 06:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 06:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3204: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 06:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3204 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 06:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 06:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 06:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 06:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 06:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 06:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 06:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3205: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 06:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3205 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 06:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 07:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 07:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 07:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 07:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 07:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 07:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3206: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 07:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3206 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 07:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 07:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 07:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 07:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 07:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 07:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 07:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3207: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 07:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3207 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 07:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 07:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 07:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 07:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 07:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 07:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 07:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3208: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 07:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3208 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 07:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 07:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 07:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 07:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 07:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 07:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 07:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3209 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 07:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 07:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3209: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 08:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 08:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 08:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 08:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 08:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 08:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3210: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 08:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3210 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 08:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 08:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 08:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 08:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 08:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 08:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 08:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3211: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 08:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3211 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 08:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 08:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 08:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 08:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 08:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 08:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 08:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3212: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 08:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3212 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 08:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 08:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 08:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 08:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 08:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 08:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 08:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3213: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 08:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3213 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 08:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 09:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 09:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 09:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 09:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 09:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 09:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3214: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 09:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3214 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 09:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 09:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 09:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 09:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 09:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 09:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 09:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3215: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 09:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3215 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 09:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 09:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 09:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 09:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 09:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 09:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 09:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3216: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 09:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3216 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 09:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 09:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 09:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 09:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 09:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 09:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 09:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3217: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 09:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3217 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 09:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 10:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 10:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 10:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 10:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 10:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 10:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3218 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 10:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 10:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3218: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 10:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 10:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 10:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 10:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 10:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 10:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3219: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 10:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3219 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 10:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 10:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 10:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 10:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 10:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 10:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 10:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3220: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 10:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3220 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 10:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 10:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 10:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 10:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 10:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 10:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 10:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3221: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 10:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3221 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 10:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 11:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 11:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 11:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 11:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 11:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 11:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3222: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 11:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3222 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 11:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 11:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 11:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 11:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 11:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 11:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 11:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3223: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 11:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3223 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 11:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 11:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 11:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 11:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 11:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 11:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 11:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3224: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 11:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3224 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 11:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 11:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 11:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 11:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 11:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 11:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 11:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3225: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 11:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3225 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 11:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 12:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 12:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 12:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 12:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 12:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 12:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3226: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 12:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3226 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 12:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 12:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 12:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 12:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 12:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 12:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 12:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3227: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 12:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3227 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 12:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 12:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 12:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 12:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 12:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 12:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 12:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3228: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 12:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3228 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 12:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 12:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 12:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 12:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 12:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 12:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 12:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3229: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 12:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3229 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 12:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 13:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 13:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 13:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 13:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 13:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 13:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3230: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 13:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3230 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 13:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 13:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 13:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 13:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 13:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 13:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 13:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3231: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 13:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3231 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 13:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 13:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 13:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 13:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 13:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 13:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 13:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3232: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 13:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3232 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 13:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 13:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 13:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 13:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 13:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 13:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 13:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3233 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 13:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 13:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3233: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 14:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 14:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 14:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 14:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 14:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 14:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3234: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 14:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3234 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 14:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 14:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 14:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 14:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 14:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 14:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 14:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3235: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 14:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3235 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 14:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 14:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 14:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 14:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 14:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 14:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 14:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3236: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 14:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3236 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 14:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 14:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 14:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 14:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 14:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 14:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 14:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3237 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 14:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 14:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3237: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 15:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 15:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 15:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 15:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 15:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 15:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3238: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 15:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3238 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 15:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 15:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 15:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 15:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 15:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 15:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 15:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3239: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 15:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3239 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 15:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 15:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 15:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 15:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 15:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 15:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 15:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3240: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 15:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3240 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 15:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 15:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 15:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 15:49:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 15:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 15:49:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 15:49:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3241: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 15:49:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3241 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 15:49:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 16:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:04:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:04:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:04:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3242: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 16:04:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3242 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 16:04:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 16:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:19:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:19:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:19:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3243: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 16:19:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3243 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 16:19:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 16:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:34:45 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:34:45 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:34:45 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3244: /var/lib/pacemaker/pengine/pe-input-59.bz2
Feb 12 16:34:45 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3244 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Complete
Feb 12 16:34:45 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 47 00 00 08 00
Feb 12 16:38:39 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 71
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:1:0: reservation conflict
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:1:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 12 16:38:39 kcsandnp01 kernel: sd 1:0:1:0: [sdc] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Feb 12 16:38:39 kcsandnp01 kernel: end_request: critical nexus error, dev sdc, sector 0
Feb 12 16:38:39 kcsandnp01 LVM(ha_lvm)[64905]: ERROR: WARNING: vg_share is active without the cluster tag, "pacemaker"
Feb 12 16:38:39 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:64905:stderr [   /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error ]
Feb 12 16:38:39 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:64905:stderr [   /dev/sdc: read failed after 0 of 4096 at 0: Input/output error ]
Feb 12 16:38:39 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:64905:stderr [   Volume group "vg_share" not found ]
Feb 12 16:38:39 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:64905:stderr [   Cannot process volume group vg_share ]
Feb 12 16:38:39 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: unknown error (node=kcsandnp01, call=533, rc=1, cib-update=3828, confirmed=false)
Feb 12 16:38:39 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-ha_lvm_monitor_10000:533 [   /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error\n  /dev/sdc: read failed after 0 of 4096 at 0: Input/output error\n  Volume group "vg_share" not found\n  Cannot process volume group vg_share\n ]
Feb 12 16:38:39 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for ha_lvm on kcsandnp01 after failed monitor: rc=1 (update=value++, time=1486892319)
Feb 12 16:38:39 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 12 16:38:39 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-ha_lvm (1)
Feb 12 16:38:39 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 240: fail-count-ha_lvm=1
Feb 12 16:38:39 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-ha_lvm (1486892319)
Feb 12 16:38:39 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 242: last-failure-ha_lvm=1486892319
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Recover ha_lvm#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart mqha_fs#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart vip#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart wmq#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3245: /var/lib/pacemaker/pengine/pe-input-60.bz2
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Recover ha_lvm#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart mqha_fs#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart vip#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart wmq#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 12 16:38:39 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3246: /var/lib/pacemaker/pengine/pe-input-61.bz2
Feb 12 16:38:39 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: stop iib_stop_0 on kcsandnp01 (local)
Feb 12 16:38:40 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5932] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 12 16:38:40 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5932] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 12 16:38:50 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: ok (node=kcsandnp01, call=533, rc=0, cib-update=3832, confirmed=false)
Feb 12 16:38:50 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by ha_lvm_monitor_10000 'modify' on kcsandnp01: Old event (magic=0:0;9:1332:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.27, source=process_graph_event:555, 0)
Feb 12 16:38:53 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10601' is being 'stopped'. 
Feb 12 16:38:53 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10606' is being 'stopped'. 
Feb 12 16:38:54 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10603' is being 'stopped'. 
Feb 12 16:38:55 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10607' is being 'stopped'. 
Feb 12 16:38:55 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10602' is being 'stopped'. 
Feb 12 16:38:56 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10605' is being 'stopped'. 
Feb 12 16:38:58 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10604' is being 'stopped'. 
Feb 12 16:38:58 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10600' is being 'stopped'. 
Feb 12 16:39:02 kcsandnp01 IIB[5770]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5770] (Msg 1/1) BIP2204I: Integration server using process '5770' thread '5770' stopped. 
Feb 12 16:39:09 kcsandnp01 IIB[5575]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5728] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 12 16:39:10 kcsandnp01 IIB[5570]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5570] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 12 16:39:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=569, rc=0, cib-update=3833, confirmed=true)
Feb 12 16:39:10 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3246 (Complete=2, Pending=0, Fired=0, Skipped=18, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-61.bz2): Stopped
Feb 12 16:39:10 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 12 16:39:10 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 12 16:39:10 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3247: /var/lib/pacemaker/pengine/pe-input-62.bz2
Feb 12 16:39:11 kcsandnp01 IIB[1332]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1332] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 12 16:39:11 kcsandnp01 IIB[1374]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1374] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 1379. 
Feb 12 16:39:12 kcsandnp01 IIB[1379]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1379] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 12 16:39:13 kcsandnp01 IIB[1379]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1379] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 12 16:39:13 kcsandnp01 IIB[1379]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1441] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 12 16:39:14 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1481] (Msg 1/1) BIP2208I: Integration server (64) started: process '1481'; thread '1481'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 12 16:39:20 kcsandnp01 IIB[1379]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1482] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 12 16:39:25 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1722] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 12 16:39:25 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1722] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 12 16:39:27 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1722] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 12 16:39:33 kcsandnp01 IIB[1442]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 1489] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 12 16:39:35 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1718] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 12 16:39:35 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1718] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 12 16:39:36 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1720] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 12 16:39:40 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=570, rc=0, cib-update=3835, confirmed=true)
Feb 12 16:39:40 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 12 16:39:41 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=571, rc=0, cib-update=3836, confirmed=false)
Feb 12 16:39:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3247 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-62.bz2): Complete
Feb 12 16:39:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:53:17 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 12 16:53:17 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 12 16:53:17 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 0e 68 6f 00 00 08 00
Feb 12 16:53:17 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 944239
Feb 12 16:53:17 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:17 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:17 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4899.0.FDC
Feb 12 16:53:17 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4899.0.FDC
Feb 12 16:53:17 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4899.0.FDC
Feb 12 16:53:17 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1819] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 12 16:53:17 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1819] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:17 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1693] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '1693' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 12 16:53:17 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1693] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2162'. 
Feb 12 16:53:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4828.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4828.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4828.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4828.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4861.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4844.0.FDC
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1875] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1875] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1803] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1803] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1781] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1781] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1817] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1817] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1774] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1774] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1810] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1810] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1813] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1813] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1784] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1784] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1777] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1777] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1809] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1809] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1780] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1780] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1787] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1787] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1802] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1771] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1771] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1799] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1773] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1799] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1773] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1871] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1871] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1868] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1868] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1873] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 12 16:53:29 kcsandnp01 IIB[1481]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1873] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 12 16:53:39 kcsandnp01 IIB[1379]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1441] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 12 16:53:40 kcsandnp01 IIB[1374]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1374] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 12 16:53:40 kcsandnp01 IIB[1374]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1374] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 19187. 
Feb 12 16:53:40 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 12 16:53:41 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 12 16:53:51 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: not running (node=kcsandnp01, call=571, rc=7, cib-update=3837, confirmed=false)
Feb 12 16:53:51 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:571 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 12 16:53:51 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486893231)
Feb 12 16:53:51 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (6)
Feb 12 16:53:51 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 12 16:53:51 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 244: fail-count-iib=6
Feb 12 16:53:51 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1486893231)
Feb 12 16:53:51 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 246: last-failure-iib=1486893231
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3248: /var/lib/pacemaker/pengine/pe-input-63.bz2
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 12 16:53:51 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3249: /var/lib/pacemaker/pengine/pe-input-64.bz2
Feb 12 16:53:51 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: stop iib_stop_0 on kcsandnp01 (local)
Feb 12 16:53:51 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:571 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 12 16:53:56 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 12 16:53:56 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 12 16:53:57 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 12 16:53:57 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.19187', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 12 16:53:57 kcsandnp01 IIB[19187]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 19187] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 12 16:53:58 kcsandnp01 IIB[1374]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1374] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 12 16:54:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=565, rc=7, cib-update=3841, confirmed=false)
Feb 12 16:54:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:565 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 12 16:54:17 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_60000 'modify' on kcsandnp01: Old event (magic=0:7;3:2837:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.34, source=process_graph_event:555, 0)
Feb 12 16:54:17 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486893257)
Feb 12 16:54:17 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (3)
Feb 12 16:54:17 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 248: fail-count-wmq=3
Feb 12 16:54:17 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1486893257)
Feb 12 16:54:17 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 250: last-failure-wmq=1486893257
Feb 12 16:54:40 kcsandnp01 abrt[21041]: Saved core dump of pid 1481 (/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-12-16:54:26-1481 (1627213824 bytes)
Feb 12 16:54:40 kcsandnp01 abrtd: Directory 'ccpp-2017-02-12-16:54:26-1481' creation detected
Feb 12 16:54:40 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 12 16:54:40 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-12-16:54:26-1481' exited with 1
Feb 12 16:54:40 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-12-16:54:26-1481'
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=573, rc=0, cib-update=3842, confirmed=true)
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3249 (Complete=2, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-64.bz2): Stopped
Feb 12 16:54:40 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 12 16:54:40 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3250: /var/lib/pacemaker/pengine/pe-input-65.bz2
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:565 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=575, rc=0, cib-update=3844, confirmed=true)
Feb 12 16:54:40 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: start wmq_start_0 on kcsandnp01 (local)
Feb 12 16:54:41 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=576, rc=0, cib-update=3845, confirmed=true)
Feb 12 16:54:41 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 12 16:54:41 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 12 16:54:41 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=577, rc=0, cib-update=3846, confirmed=false)
Feb 12 16:54:42 kcsandnp01 IIB[21935]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21935] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 12 16:54:42 kcsandnp01 IIB[22000]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22000] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 22005. 
Feb 12 16:54:42 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22005] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 12 16:54:44 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22005] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 12 16:54:44 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22083] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 12 16:54:44 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22129] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 1481, has shutdown. 
Feb 12 16:54:45 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22133] (Msg 1/1) BIP2208I: Integration server (64) started: process '22133'; thread '22133'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 12 16:54:46 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 33.130001
Feb 12 16:54:50 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22134] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 12 16:54:55 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22353] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 12 16:54:55 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22353] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 12 16:54:57 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22353] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 12 16:55:03 kcsandnp01 IIB[22084]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 22141] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 12 16:55:06 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22349] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 12 16:55:07 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22349] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 12 16:55:07 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22351] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 12 16:55:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=578, rc=0, cib-update=3847, confirmed=true)
Feb 12 16:55:10 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 12 16:55:12 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=579, rc=0, cib-update=3848, confirmed=false)
Feb 12 16:55:12 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3250 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-65.bz2): Complete
Feb 12 16:55:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 16:55:16 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 21.260000
Feb 12 16:55:46 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 12.880000
Feb 12 16:56:16 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 7.810000
Feb 12 16:56:46 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 4.730000
Feb 12 17:10:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 17:10:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 17:10:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 17:10:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 17:10:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 17:10:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 17:10:12 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3251: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 17:10:12 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3251 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 17:10:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 17:25:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 17:25:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 17:25:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 17:25:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 17:25:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 17:25:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 17:25:12 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3252: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 17:25:12 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3252 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 17:25:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 17:40:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 17:40:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 17:40:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 17:40:12 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 17:40:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 17:40:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 17:40:12 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3253: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 17:40:12 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3253 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 17:40:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 17:55:12 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 17:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 17:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 17:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 17:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 17:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 17:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3254: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 17:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3254 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 17:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 18:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 18:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 18:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 18:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 18:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 18:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 18:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3255: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 18:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3255 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 18:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 18:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 18:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 18:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 18:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 18:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 18:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 18:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3256 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 18:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 18:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3256: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 18:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 18:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 18:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 18:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 18:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 18:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 18:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3257: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 18:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3257 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 18:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 18:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 18:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 18:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 18:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 18:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 18:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 18:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3258: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 18:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3258 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 18:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 19:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 19:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 19:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 19:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 19:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 19:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 19:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3259: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 19:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3259 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 19:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 19:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 19:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 19:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 19:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 19:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 19:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 19:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3260: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 19:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3260 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 19:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 19:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 19:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 19:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 19:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 19:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 19:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 19:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3261: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 19:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3261 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 19:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 19:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 19:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 19:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 19:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 19:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 19:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 19:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3262 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 19:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 19:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3262: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 20:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 20:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 20:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 20:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 20:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 20:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 20:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3263: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 20:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3263 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 20:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 20:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 20:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 20:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 20:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 20:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 20:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 20:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3264: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 20:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3264 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 20:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 20:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 20:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 20:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 20:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 20:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 20:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 20:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3265: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 20:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3265 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 20:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 20:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 20:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 20:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 20:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 20:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 20:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 20:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3266: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 20:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3266 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 20:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 21:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 21:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 21:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 21:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 21:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 21:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 21:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3267: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 21:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3267 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 21:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 21:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 21:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 21:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 21:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 21:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 21:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 21:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3268: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 21:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3268 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 21:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 21:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 21:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 21:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 21:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 21:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 21:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 21:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3269: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 21:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3269 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 21:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 21:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 21:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 21:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 21:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 21:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 21:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 21:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3270: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 21:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3270 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 21:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 22:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 22:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 22:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 22:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 22:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 22:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 22:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3271: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 22:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3271 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 22:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 22:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 22:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 22:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 22:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 22:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 22:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 22:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3272: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 22:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3272 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 22:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 22:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 22:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 22:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 22:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 22:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 22:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 22:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3273: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 22:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3273 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 22:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 22:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 22:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 22:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 22:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 22:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 22:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 22:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3274: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 22:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3274 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 22:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 23:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 23:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 23:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 23:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 23:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 23:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 23:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3275: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 23:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3275 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 23:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 23:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 23:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 23:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 23:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 23:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 23:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 23:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3276: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 23:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3276 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 23:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 23:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 23:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 23:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 23:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 23:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 23:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 23:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3277: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 23:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3277 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 23:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 12 23:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 12 23:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 12 23:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 12 23:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 12 23:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 12 23:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 12 23:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3278: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 12 23:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3278 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 12 23:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 00:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 00:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 00:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 00:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 00:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 00:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 00:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3279: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 00:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3279 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 00:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 00:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 00:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 00:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 00:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 00:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 00:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 00:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3280: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 00:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3280 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 00:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 00:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 00:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 00:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 00:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 00:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 00:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 00:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3281: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 00:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3281 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 00:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 00:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 00:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 00:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 00:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 00:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 00:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 00:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3282: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 00:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3282 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 00:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 01:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 01:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 01:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 01:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 01:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 01:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 01:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3283: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 01:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3283 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 01:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 01:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 01:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 01:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 01:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 01:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 01:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 01:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3284: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 01:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3284 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 01:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 01:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 01:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 01:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 01:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 01:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 01:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 01:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3285: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 01:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3285 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 01:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 01:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 01:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 01:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 01:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 01:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 01:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 01:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3286 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 01:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 01:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3286: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 02:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 02:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 02:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 02:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 02:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 02:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 02:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3287: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 02:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3287 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 02:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 02:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 02:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 02:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 02:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 02:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 02:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 02:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3288: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 02:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3288 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 02:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 02:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 02:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 02:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 02:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 02:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 02:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 02:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3289: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 02:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3289 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 02:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 02:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 02:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 02:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 02:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 02:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 02:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 02:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3290: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 02:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3290 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 02:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 03:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 03:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 03:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 03:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 03:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 03:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 03:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3291: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 03:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3291 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 03:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 03:23:02 kcsandnp01 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Feb 13 03:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 03:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 03:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 03:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 03:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 03:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 03:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3292: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 03:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3292 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 03:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 03:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 03:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 03:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 03:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 03:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 03:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 03:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3293: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 03:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3293 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 03:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 03:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 03:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 03:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 03:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 03:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 03:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 03:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3294: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 03:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3294 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 03:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 04:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 04:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 04:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 04:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 04:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 04:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 04:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3295: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 04:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3295 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 04:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 04:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 04:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 04:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 04:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 04:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 04:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 04:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3296: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 04:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3296 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 04:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 04:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 04:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 04:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 04:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 04:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 04:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 04:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3297: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 04:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3297 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 04:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 04:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 04:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 04:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 04:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 04:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 04:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 04:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3298: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 04:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3298 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 04:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 05:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 05:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 05:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 05:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 05:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 05:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 05:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3299: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 05:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3299 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 05:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 05:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 05:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 05:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 05:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 05:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 05:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 05:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3300: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 05:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3300 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 05:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 05:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 05:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 05:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 05:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 05:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 05:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 05:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3301: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 05:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3301 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 05:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 05:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 05:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 05:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 05:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 05:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 05:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 05:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3302: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 05:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3302 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 05:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 06:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 06:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 06:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 06:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 06:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 06:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 06:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3303: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 06:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3303 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 06:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 06:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 06:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 06:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 06:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 06:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 06:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 06:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3304: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 06:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3304 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 06:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 06:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 06:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 06:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 06:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 06:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 06:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 06:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3305: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 06:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3305 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 06:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 06:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 06:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 06:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 06:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 06:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 06:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 06:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3306: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 06:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3306 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 06:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 07:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 07:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 07:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 07:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 07:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 07:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 07:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3307: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 07:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3307 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 07:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 07:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 07:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 07:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 07:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 07:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 07:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 07:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3308: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 07:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3308 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 07:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 07:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 07:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 07:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 07:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 07:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 07:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 07:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3309: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 07:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3309 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 07:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 07:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 07:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 07:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 07:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 07:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 07:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 07:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3310: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 07:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3310 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 07:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 08:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 08:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 08:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 08:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 08:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 08:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 08:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3311: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 08:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3311 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 08:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 08:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 08:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 08:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 08:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 08:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 08:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 08:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3312: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 08:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3312 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 08:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 08:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 08:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 08:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 08:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 08:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 08:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 08:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3313: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 08:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3313 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 08:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 08:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 08:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 08:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 08:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 08:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 08:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 08:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3314: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 08:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3314 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 08:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 09:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 09:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 09:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 09:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 09:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 09:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 09:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3315: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 09:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3315 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 09:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 09:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 09:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 09:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 09:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 09:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 09:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 09:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3316: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 09:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3316 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 09:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 09:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 09:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 09:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 09:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 09:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 09:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 09:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3317: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 09:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3317 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 09:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 09:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 09:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 09:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 09:55:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 09:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 09:55:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 09:55:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3318: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 09:55:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3318 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 09:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 10:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 10:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 10:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 10:10:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 10:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 10:10:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 10:10:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3319: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 10:10:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3319 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 10:10:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 10:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 10:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 10:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 10:25:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 10:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 10:25:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 10:25:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3320: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 10:25:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3320 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 10:25:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 10:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 10:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 10:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 10:40:13 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 10:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 10:40:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 10:40:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3321: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 10:40:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3321 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 10:40:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 10:55:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 10:55:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 10:55:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 10:55:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 10:55:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 10:55:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 10:55:14 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3322: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 10:55:14 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3322 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 10:55:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 11:10:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 11:10:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:10:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:10:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:10:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:10:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:10:14 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3323: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 11:10:14 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3323 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 11:10:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 11:25:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 11:25:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:25:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:25:14 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:25:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:25:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:25:14 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3324: /var/lib/pacemaker/pengine/pe-input-66.bz2
Feb 13 11:25:14 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3324 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-66.bz2): Complete
Feb 13 11:25:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 11:31:30 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 11:31:30 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 11:31:30 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 18 2a 47 00 00 08 00
Feb 13 11:31:30 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 1583687
Feb 13 11:31:30 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:30 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:30 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:30 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:30 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21291.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22361] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22361] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21204.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21204.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21204.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22390] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22390] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22407] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22371] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22364] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22387] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22397] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22368] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22368] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21251.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22360] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22838] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 11:31:40 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22838] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:40 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22391] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22391] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22405] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22405] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22398] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22398] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22481] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22481] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22401] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22401] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22471] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22471] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22374] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22367] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:31:41 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22367] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:41 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ21217.0.FDC
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQ' transport. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQTT' transport. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22254] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.22133', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22374] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22374] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22359] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22359] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22005]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22083] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22407] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22407] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22371] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22371] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22364] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22364] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22387] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22387] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22397] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22397] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22357] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22357] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22356] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22356] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22355] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22355] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22360] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 13 11:31:51 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22360] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:52 kcsandnp01 IIB[22000]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22000] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 13 11:31:52 kcsandnp01 IIB[22000]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22000] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 64134. 
Feb 13 11:31:52 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 11:31:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: not running (node=kcsandnp01, call=579, rc=7, cib-update=3923, confirmed=false)
Feb 13 11:31:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:579 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 13 11:31:53 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486960313)
Feb 13 11:31:53 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (7)
Feb 13 11:31:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 13 11:31:53 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 252: fail-count-iib=7
Feb 13 11:31:53 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1486960313)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3325: /var/lib/pacemaker/pengine/pe-input-67.bz2
Feb 13 11:31:53 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 254: last-failure-iib=1486960313
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3326: /var/lib/pacemaker/pengine/pe-input-68.bz2
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 13 11:31:53 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: stop iib_stop_0 on kcsandnp01 (local)
Feb 13 11:31:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:579 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 13 11:31:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3327: /var/lib/pacemaker/pengine/pe-input-69.bz2
Feb 13 11:31:54 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 13 11:31:58 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22404] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.TCPIP Client In Rcv ISO'. 
Feb 13 11:31:58 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22404] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 13 11:31:58 kcsandnp01 IIB[22133]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 22404] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:31:59 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 13 11:31:59 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:32:00 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 13 11:32:00 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.64134', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 11:32:00 kcsandnp01 IIB[64134]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 64134] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 13 11:32:01 kcsandnp01 IIB[22000]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 22000] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 13 11:32:27 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=577, rc=7, cib-update=3927, confirmed=false)
Feb 13 11:32:27 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:577 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 11:32:27 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_60000 'modify' on kcsandnp01: Old event (magic=0:7;3:3250:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.46, source=process_graph_event:555, 0)
Feb 13 11:32:27 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486960347)
Feb 13 11:32:27 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (4)
Feb 13 11:32:27 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 256: fail-count-wmq=4
Feb 13 11:32:27 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1486960347)
Feb 13 11:32:27 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 258: last-failure-wmq=1486960347
Feb 13 11:32:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 13.760000
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=581, rc=0, cib-update=3928, confirmed=true)
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3327 (Complete=2, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-69.bz2): Stopped
Feb 13 11:32:56 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 11:32:56 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:577 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 11:32:56 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3328: /var/lib/pacemaker/pengine/pe-input-70.bz2
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=583, rc=0, cib-update=3930, confirmed=true)
Feb 13 11:32:56 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: start wmq_start_0 on kcsandnp01 (local)
Feb 13 11:32:57 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=584, rc=0, cib-update=3931, confirmed=true)
Feb 13 11:32:57 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 13 11:32:57 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=585, rc=0, cib-update=3932, confirmed=false)
Feb 13 11:32:57 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 13 11:32:58 kcsandnp01 abrt[598]: Saved core dump of pid 22133 (/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-13-11:32:43-22133 (1621266432 bytes)
Feb 13 11:32:58 kcsandnp01 abrtd: Directory 'ccpp-2017-02-13-11:32:43-22133' creation detected
Feb 13 11:32:58 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 13 11:32:58 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-13-11:32:43-22133' exited with 1
Feb 13 11:32:58 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-13-11:32:43-22133'
Feb 13 11:32:58 kcsandnp01 IIB[1575]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1575] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 13 11:32:59 kcsandnp01 IIB[1603]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1603] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 1608. 
Feb 13 11:32:59 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1608] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 11:33:00 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1608] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 13 11:33:01 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1670] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 13 11:33:01 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1708] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 22133, has shutdown. 
Feb 13 11:33:01 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1712] (Msg 1/1) BIP2208I: Integration server (64) started: process '1712'; thread '1712'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 13 11:33:07 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1711] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 13 11:33:13 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1933] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 11:33:13 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1933] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 13 11:33:15 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1933] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 13 11:33:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 27.490000
Feb 13 11:33:22 kcsandnp01 IIB[1671]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 1719] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 13 11:33:22 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 11:33:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 11:33:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 18 2b 97 00 00 08 00
Feb 13 11:33:22 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 1584023
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2087] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2087] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2162. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1990] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1990] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1990] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.MQ Out: ANYID_REQ_FROM_ITMX' when writing message to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1990] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1990] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 13 11:33:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1985] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO1.TCPIP Client Input'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1985] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO1.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 11:33:23 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1985] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Outbound_Res_PO1.MQ Out: ANYID_RES_FROM_ITMX' when writing message to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 11:33:25 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1929] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 11:33:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:26 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1929] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=586, rc=0, cib-update=3933, confirmed=true)
Feb 13 11:33:27 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 13 11:33:27 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1931] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ850.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ881.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ813.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ813.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ813.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ833.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ1712.0.FDC
Feb 13 11:33:33 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2184] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:33:33 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2184] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2101. 
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ826.0.FDC
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1935] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1935] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1936] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1936] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1938] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 11:33:35 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1938] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1975] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1984] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1975] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1984] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1943] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1943] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1959] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1959] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1952] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:33:41 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1952] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1962] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO4.TCPIP Client Input'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1962] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO4.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1962] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1923] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '1923' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1923] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1991] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1991] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQ' transport. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQTT' transport. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1924] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.1712', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2183] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2183] (Msg 2/2) BIP2679E: Failed to setup a MQ callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1993] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1993] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1953] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1953] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1944] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1944] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1987] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1987] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1956] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1956] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1947] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1947] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1972] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1972] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2078] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2078] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1983] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1983] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1942] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1942] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2076] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2076] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1977] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1977] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2075] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2075] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1982] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1982] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1986] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1986] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2081] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 2081] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 1/3) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'ESB_LoadCache.ANYID_CACHE_CMD.OUT'. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 3/3) BIP2666E: Unable to open queue 'ANYID_CACHE_CMD.OUT' on WebSphere MQ queue manager 'ESBXPRD01': completion code 2; reason code 2009. 
Feb 13 11:33:43 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 1/1) BIP2682W: Unable to rollback a WebSphere MQ transaction due to a broken connection to the WebSphere MQ queue manager ('ESBXPRD01'). MQRC=2009. 
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: not running (node=kcsandnp01, call=587, rc=7, cib-update=3934, confirmed=false)
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:587 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 13 11:33:43 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 20 (iib_monitor_60000) on kcsandnp01 failed (target: 0 vs. rc: 7): Error
Feb 13 11:33:43 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486960423)
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by iib_monitor_60000 'modify' on kcsandnp01: Event failed (magic=0:7;20:3328:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.54, source=match_graph_event:344, 0)
Feb 13 11:33:43 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486960423)
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (8)
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3328 (Complete=9, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-70.bz2): Stopped
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 260: fail-count-iib=8
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1486960423)
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 262: last-failure-iib=1486960423
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (9)
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 264: fail-count-iib=9
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1486960423)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:33:43 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 266: last-failure-iib=1486960423
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3329: /var/lib/pacemaker/pengine/pe-input-71.bz2
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: stop iib_stop_0 on kcsandnp01 (local)
Feb 13 11:33:43 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3330: /var/lib/pacemaker/pengine/pe-input-72.bz2
Feb 13 11:33:43 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:587 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 13 11:33:44 kcsandnp01 IIB[1608]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1670] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1976] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:44 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1965] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 13 11:33:45 kcsandnp01 IIB[1603]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1603] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 13 11:33:45 kcsandnp01 IIB[1603]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1603] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 3861. 
Feb 13 11:33:45 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 11:33:46 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 13 11:33:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 18.010000
Feb 13 11:33:49 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1979] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 13 11:33:49 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1979] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 13 11:33:49 kcsandnp01 IIB[1712]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 1979] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:51 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 13 11:33:51 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 11:33:52 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 13 11:33:52 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.3861', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 11:33:52 kcsandnp01 IIB[3861]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 3861] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 13 11:33:53 kcsandnp01 IIB[1603]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 1603] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 13 11:33:57 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=585, rc=7, cib-update=3939, confirmed=false)
Feb 13 11:33:57 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:585 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 11:33:57 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_60000 'modify' on kcsandnp01: Old event (magic=0:7;3:3328:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.58, source=process_graph_event:555, 0)
Feb 13 11:33:57 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486960437)
Feb 13 11:33:57 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (5)
Feb 13 11:33:57 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 268: fail-count-wmq=5
Feb 13 11:33:57 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1486960437)
Feb 13 11:33:57 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 270: last-failure-wmq=1486960437
Feb 13 11:34:07 kcsandnp01 abrt[4021]: Saved core dump of pid 1712 (/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-13-11:33:56-1712 (1643319296 bytes)
Feb 13 11:34:07 kcsandnp01 abrtd: Directory 'ccpp-2017-02-13-11:33:56-1712' creation detected
Feb 13 11:34:07 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 13 11:34:07 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-13-11:33:56-1712' exited with 1
Feb 13 11:34:07 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-13-11:33:56-1712'
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=589, rc=0, cib-update=3940, confirmed=true)
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3330 (Complete=2, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-72.bz2): Stopped
Feb 13 11:34:07 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 13 11:34:07 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3331: /var/lib/pacemaker/pengine/pe-input-73.bz2
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:585 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=591, rc=0, cib-update=3942, confirmed=true)
Feb 13 11:34:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: start wmq_start_0 on kcsandnp01 (local)
Feb 13 11:34:08 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=592, rc=0, cib-update=3943, confirmed=true)
Feb 13 11:34:08 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 13 11:34:08 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=593, rc=0, cib-update=3944, confirmed=false)
Feb 13 11:34:08 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 13 11:34:09 kcsandnp01 IIB[5020]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5020] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 13 11:34:09 kcsandnp01 IIB[5045]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5045] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 5050. 
Feb 13 11:34:09 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5050] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 11:34:11 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5050] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 13 11:34:11 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5112] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 13 11:34:11 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5145] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 1712, has shutdown. 
Feb 13 11:34:12 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5149] (Msg 1/1) BIP2208I: Integration server (64) started: process '5149'; thread '5149'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 13 11:34:17 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5148] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 13 11:34:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 36.380001
Feb 13 11:34:22 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5321] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 11:34:22 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5321] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 13 11:34:24 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5321] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 13 11:34:24 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5329] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 13 11:34:24 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5336] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:34:24 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5336] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:34:24 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5336] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:34:25 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5358] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 11:34:25 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5360] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 11:34:30 kcsandnp01 IIB[5113]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 5156] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 13 11:34:34 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5317] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 11:34:34 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5317] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 11:34:35 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5319] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 13 11:34:37 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=594, rc=0, cib-update=3945, confirmed=true)
Feb 13 11:34:37 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 13 11:34:41 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=595, rc=0, cib-update=3946, confirmed=false)
Feb 13 11:34:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3331 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-73.bz2): Complete
Feb 13 11:34:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 11:34:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 22.950001
Feb 13 11:35:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 13.960000
Feb 13 11:35:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 8.460000
Feb 13 11:36:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 5.190000
Feb 13 11:49:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 11:49:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 11:49:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 11:49:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 11:49:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 11:49:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 11:49:41 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3332: /var/lib/pacemaker/pengine/pe-input-74.bz2
Feb 13 11:49:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3332 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-74.bz2): Complete
Feb 13 11:49:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 12:04:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 12:04:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:04:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:04:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:04:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:04:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:04:41 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3333: /var/lib/pacemaker/pengine/pe-input-74.bz2
Feb 13 12:04:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3333 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-74.bz2): Complete
Feb 13 12:04:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 12:19:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 12:19:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:19:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:19:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:19:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:19:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:19:41 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3334: /var/lib/pacemaker/pengine/pe-input-74.bz2
Feb 13 12:19:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3334 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-74.bz2): Complete
Feb 13 12:19:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 12:34:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 12:34:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:34:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:34:41 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:34:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:34:41 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:34:41 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3335: /var/lib/pacemaker/pengine/pe-input-74.bz2
Feb 13 12:34:41 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3335 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-74.bz2): Complete
Feb 13 12:34:41 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 12:46:32 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 12:46:32 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 12:46:32 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 18 84 77 00 00 10 00
Feb 13 12:46:32 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 1606775
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:32 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5362] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO3.TCPIP Client Input'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5362] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO3.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5362] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Outbound_Res_PO3.MQ Out: ANYID_RES_FROM_ITMX' when writing message to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5346] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO4.TCPIP Client Input'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5346] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO4.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5346] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Outbound_Res_PO4.MQ Out: ANYID_RES_FROM_ITMX' when writing message to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5363] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO2.TCPIP Client Input'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5363] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO2.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5363] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Outbound_Res_PO2.MQ Out: ANYID_RES_FROM_ITMX' when writing message to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 12:46:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4330.0.FDC
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5453] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5453] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2446. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5368] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5368] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2446. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5365] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5365] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 6016] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 6016] (Msg 2/2) BIP2679E: Failed to setup a MQ callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2101. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5367] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO1.TCPIP Client Input'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5367] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO1.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5367] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2162. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5349] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5349] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 13 12:46:33 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5349] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.MQ Out: ANYID_REQ_FROM_ITMX' when writing message to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4259.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4259.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4259.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4259.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4287.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:38 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:39 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ4272.0.FDC
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5267] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '5267' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5267] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5374] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI3.TCPIP Client In Rcv ISO'. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5374] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI3.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5374] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5361] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5361] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 13 12:46:43 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5361] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:46:44 kcsandnp01 IIB[5050]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5112] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 13 12:46:45 kcsandnp01 IIB[5045]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5045] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 13 12:46:45 kcsandnp01 IIB[5045]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5045] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 27614. 
Feb 13 12:46:45 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 12:46:46 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5359] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5359] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5375] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5375] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5337] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5337] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5364] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5419] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5419] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5333] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5333] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5447] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5447] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5358] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5358] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5364] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5414] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5414] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5355] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5340] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5369] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5340] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5355] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5369] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5330] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5330] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5325] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5325] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5323] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5323] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5328] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5328] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5326] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 12:46:53 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5326] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5336] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5336] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5366] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5366] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5343] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5343] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5373] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 13 12:47:03 kcsandnp01 IIB[5149]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 5373] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 12:47:15 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=593, rc=7, cib-update=3951, confirmed=false)
Feb 13 12:47:15 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:593 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 12:47:15 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486964835)
Feb 13 12:47:15 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 13 12:47:15 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (6)
Feb 13 12:47:15 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 272: fail-count-wmq=6
Feb 13 12:47:15 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1486964835)
Feb 13 12:47:15 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 274: last-failure-wmq=1486964835
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3336: /var/lib/pacemaker/pengine/pe-input-75.bz2
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3337: /var/lib/pacemaker/pengine/pe-input-76.bz2
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 13 12:47:15 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: stop iib_stop_0 on kcsandnp01 (local)
Feb 13 12:47:15 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3338: /var/lib/pacemaker/pengine/pe-input-77.bz2
Feb 13 12:47:17 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 13 12:47:17 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 12:47:18 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 13 12:47:18 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.27614', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 12:47:18 kcsandnp01 IIB[27614]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 27614] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 13 12:47:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 25.790001
Feb 13 12:47:19 kcsandnp01 IIB[5045]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 5045] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 13 12:47:23 kcsandnp01 abrt[27907]: Saved core dump of pid 5149 (/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-13-12:47:09-5149 (1655689216 bytes)
Feb 13 12:47:23 kcsandnp01 abrtd: Directory 'ccpp-2017-02-13-12:47:09-5149' creation detected
Feb 13 12:47:23 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 13 12:47:23 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-13-12:47:09-5149' exited with 1
Feb 13 12:47:23 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-13-12:47:09-5149'
Feb 13 12:47:23 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=597, rc=0, cib-update=3955, confirmed=true)
Feb 13 12:47:23 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 13 12:47:23 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:593 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=599, rc=0, cib-update=3956, confirmed=true)
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start wmq_start_0 on kcsandnp01 (local)
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=600, rc=0, cib-update=3957, confirmed=true)
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=601, rc=0, cib-update=3958, confirmed=false)
Feb 13 12:47:24 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 21: start iib_start_0 on kcsandnp01 (local)
Feb 13 12:47:25 kcsandnp01 IIB[29425]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29425] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 13 12:47:26 kcsandnp01 IIB[29449]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29449] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 29454. 
Feb 13 12:47:26 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29454] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 12:47:27 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29454] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 13 12:47:28 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29516] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 13 12:47:28 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29549] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 5149, has shutdown. 
Feb 13 12:47:28 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29553] (Msg 1/1) BIP2208I: Integration server (64) started: process '29553'; thread '29553'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 13 12:47:35 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29552] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 13 12:47:40 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29768] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 12:47:40 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29768] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 13 12:47:41 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29768] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 13 12:47:41 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29818] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 13 12:47:47 kcsandnp01 IIB[29517]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 29560] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 13 12:47:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 26.559999
Feb 13 12:47:51 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29762] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 12:47:52 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29762] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 12:47:52 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29764] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 13 12:47:54 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=602, rc=0, cib-update=3959, confirmed=true)
Feb 13 12:47:54 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 13 12:47:58 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=603, rc=0, cib-update=3960, confirmed=false)
Feb 13 12:47:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3338 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-77.bz2): Complete
Feb 13 12:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 12:48:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 16.650000
Feb 13 12:48:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 10.090000
Feb 13 12:49:18 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 6.110000
Feb 13 12:49:48 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 3.700000
Feb 13 13:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 13:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 13:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 13:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 13:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 13:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 13:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3339: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 13:02:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3339 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 13:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 13:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 13:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 13:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 13:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 13:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 13:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 13:17:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3340: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 13:17:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3340 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 13:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 13:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 13:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 13:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 13:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 13:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 13:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 13:32:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3341: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 13:32:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3341 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 13:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 13:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 13:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 13:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 13:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 13:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 13:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 13:47:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3342: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 13:47:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3342 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 13:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 14:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 14:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 14:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 14:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 14:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 14:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 14:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3343: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 14:02:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3343 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 14:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 14:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 14:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 14:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 14:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 14:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 14:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 14:17:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3344: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 14:17:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3344 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 14:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 14:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 14:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 14:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 14:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 14:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 14:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 14:32:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3345: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 14:32:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3345 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 14:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 14:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 14:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 14:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 14:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 14:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 14:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 14:47:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3346: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 14:47:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3346 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 14:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 15:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 15:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3347: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 15:02:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3347 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 15:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 15:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 15:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 15:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 15:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 15:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 15:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 15:17:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3348: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 15:17:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3348 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 15:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 15:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 15:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 15:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 15:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 15:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 15:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 15:32:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3349: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 15:32:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3349 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 15:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 15:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 15:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 15:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 15:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 15:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 15:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 15:47:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3350: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 15:47:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3350 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 15:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 16:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 16:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 16:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 16:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 16:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 16:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 16:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3351: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 16:02:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3351 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 16:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 16:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 16:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 16:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 16:17:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 16:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 16:17:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 16:17:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3352: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 16:17:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3352 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 16:17:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 16:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 16:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 16:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 16:32:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 16:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 16:32:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 16:32:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3353: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 16:32:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3353 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 16:32:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 08 3f 00 00 08 00
Feb 13 16:34:51 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 2111
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 06 3f e7 bf 00 00 08 00
Feb 13 16:34:51 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 104851391
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 06 3f e8 2f 00 00 08 00
Feb 13 16:34:51 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 104851503
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 16:34:51 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 08 3f 00 00 08 00
Feb 13 16:34:51 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 2111
Feb 13 16:34:51 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:43078:stderr [   /dev/vg_share/mqhalv: read failed after 0 of 4096 at 0: Input/output error ]
Feb 13 16:34:51 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:43078:stderr [   /dev/vg_share/mqhalv: read failed after 0 of 4096 at 53682831360: Input/output error ]
Feb 13 16:34:51 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:43078:stderr [   /dev/vg_share/mqhalv: read failed after 0 of 4096 at 53682888704: Input/output error ]
Feb 13 16:35:43 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 16:35:43 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 16:35:43 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 04 c6 be 2f 00 00 10 00
Feb 13 16:35:43 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 80133679
Feb 13 16:35:43 kcsandnp01 kernel: Buffer I/O error on device dm-4, logical block 10016446
Feb 13 16:35:43 kcsandnp01 kernel: lost page write due to I/O error on dm-4
Feb 13 16:35:43 kcsandnp01 kernel: Buffer I/O error on device dm-4, logical block 10016447
Feb 13 16:35:43 kcsandnp01 kernel: lost page write due to I/O error on dm-4
Feb 13 16:35:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:37:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:38:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:38:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:39:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:40:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:40:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:41:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:41:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:42:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:43:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:43:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:43:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:44:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:44:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:44:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:44:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:45:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:45:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:45:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:45:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:46:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:46:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:46:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:47:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:47:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:47:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:47:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 16:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 16:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 16:47:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 16:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 16:47:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 16:47:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3354: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 16:47:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3354 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 16:47:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 16:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:49:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:49:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:49:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:49:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:50:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:51:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:52:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:52:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:53:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:53:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:53:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:53:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:54:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:54:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:54:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:55:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:55:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:56:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:57:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:58:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:58:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:58:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 16:59:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:00:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:00:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:00:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:01:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:01:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:01:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:01:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:02:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:02:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 17:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:02:58 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3355 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-78.bz2): Complete
Feb 13 17:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 17:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3355: /var/lib/pacemaker/pengine/pe-input-78.bz2
Feb 13 17:03:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:03:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:03:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:04:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:04:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:04:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:05:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:05:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:05:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:05:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:05:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:06:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:06:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:06:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:07:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:08:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:08:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:08:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:08:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:09:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:09:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:10:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:10:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:10:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:11:16 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 13 17:11:16 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 13 17:11:16 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 04 a8 a7 00 00 10 00
Feb 13 17:11:16 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 305319
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29816] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO2.TCPIP Client Input'. 
Feb 13 17:11:16 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29816] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO2.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 13 17:11:16 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29816] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2162. 
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:16 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28776.0.FDC
Feb 13 17:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:11:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28708.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28708.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28708.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28708.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28736.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:26 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ28721.0.FDC
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29833] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29833] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29787] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29787] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29778] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29778] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29826] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29826] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29790] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29810] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29790] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29810] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 30889] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 30889] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29781] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29781] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29806] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29806] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29715] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '29715' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29715] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29818] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29818] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29824] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29824] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29774] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29774] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29786] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29786] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29777] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29777] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29973] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29973] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29817] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29817] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29793] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29793] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29819] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 13 17:11:26 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29819] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:27 kcsandnp01 IIB[29454]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29516] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 13 17:11:27 kcsandnp01 IIB[29449]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29449] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 13 17:11:27 kcsandnp01 IIB[29449]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29449] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 21635. 
Feb 13 17:11:27 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 17:11:29 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29771] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29771] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29770] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29770] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29772] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 13 17:11:30 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29772] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 17:11:37 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29809] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 13 17:11:37 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29831] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 13 17:11:37 kcsandnp01 IIB[29553]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 29831] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 13 17:11:50 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=601, rc=7, cib-update=3978, confirmed=false)
Feb 13 17:11:50 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:601 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 17:11:50 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1486980710)
Feb 13 17:11:50 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (7)
Feb 13 17:11:50 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 13 17:11:50 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 276: fail-count-wmq=7
Feb 13 17:11:50 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1486980710)
Feb 13 17:11:50 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 278: last-failure-wmq=1486980710
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3356: /var/lib/pacemaker/pengine/pe-input-79.bz2
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 13 17:11:50 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3357: /var/lib/pacemaker/pengine/pe-input-80.bz2
Feb 13 17:11:50 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: stop iib_stop_0 on kcsandnp01 (local)
Feb 13 17:11:54 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 13 17:11:54 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 13 17:11:54 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 13 17:11:54 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.21635', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 13 17:11:55 kcsandnp01 IIB[21635]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 21635] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 13 17:11:55 kcsandnp01 IIB[29449]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 29449] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=605, rc=0, cib-update=3982, confirmed=true)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:601 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=607, rc=0, cib-update=3983, confirmed=true)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start wmq_start_0 on kcsandnp01 (local)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=608, rc=0, cib-update=3984, confirmed=true)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 21: start iib_start_0 on kcsandnp01 (local)
Feb 13 17:12:53 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=609, rc=0, cib-update=3985, confirmed=false)
Feb 13 17:12:54 kcsandnp01 IIB[24146]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24146] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 13 17:12:54 kcsandnp01 IIB[24170]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24170] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 24175. 
Feb 13 17:12:55 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24175] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 13 17:12:56 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24175] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 13 17:12:56 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24279] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 13 17:12:56 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24305] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 29553, has shutdown. 
Feb 13 17:12:57 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24314] (Msg 1/1) BIP2208I: Integration server (64) started: process '24314'; thread '24314'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 13 17:13:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:02 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24311] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 13 17:13:07 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24538] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 13 17:13:07 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24538] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 13 17:13:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:09 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24538] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 13 17:13:15 kcsandnp01 IIB[24278]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 24316] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 13 17:13:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24534] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 17:13:19 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24534] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 13 17:13:19 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24536] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 13 17:13:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:22 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=610, rc=0, cib-update=3986, confirmed=true)
Feb 13 17:13:22 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 13 17:13:24 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=611, rc=0, cib-update=3987, confirmed=false)
Feb 13 17:13:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3357 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-80.bz2): Complete
Feb 13 17:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 17:13:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:13:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:14:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:15:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:16:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:16:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:17:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:17:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:17:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:18:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:19:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:19:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:19:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:20:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:20:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:20:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:21:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:21:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:21:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:22:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:22:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:23:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:23:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:23:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:23:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:24:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:24:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:25:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:25:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:25:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:26:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:27:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:27:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 17:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:28:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3358: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 17:28:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3358 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 17:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 17:28:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:30:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:30:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:30:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:30:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:31:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:31:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:32:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:32:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:32:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:33:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:33:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:33:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:34:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:34:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:34:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:35:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:35:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:35:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:36:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:36:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:36:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:37:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:37:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:37:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:38:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:38:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:39:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:39:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:40:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:41:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:41:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:41:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:41:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:42:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:43:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 17:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:43:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3359: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 17:43:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3359 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 17:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 17:43:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:43:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:44:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:45:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:45:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:45:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:46:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:47:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:47:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:47:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:47:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:48:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:49:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:49:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:49:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:49:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:49:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:50:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:50:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:50:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:51:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:51:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:52:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:52:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:52:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:52:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:53:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:53:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:54:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:54:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:54:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:54:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:55:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:55:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:55:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:56:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:57:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:58:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:58:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 17:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 17:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 17:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 17:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 17:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 17:58:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3360: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 17:58:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3360 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 17:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 17:58:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:59:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:59:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 17:59:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:00:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:00:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:00:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:00:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:00:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:01:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:01:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:01:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:01:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:01:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:02:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:03:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:03:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:03:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:03:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:03:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:04:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:04:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:04:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:06:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:07:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:07:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:07:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:07:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:08:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:08:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:08:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:10:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:11:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:12:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:12:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:12:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:12:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:13:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 18:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 18:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 18:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 18:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 18:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 18:13:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3361: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 18:13:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3361 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 18:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 18:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:14:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:14:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:14:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:14:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:14:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:15:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:16:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:16:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:18:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:19:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:19:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:19:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:20:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:21:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:21:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:21:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:21:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:22:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:23:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:23:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:23:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:24:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:24:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:25:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:25:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:25:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:26:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:26:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:26:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:26:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:27:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:27:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:27:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:27:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 18:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 18:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 18:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 18:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 18:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 18:28:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3362: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 18:28:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3362 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 18:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 18:28:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:28:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:29:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:29:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:29:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:29:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:30:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:30:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:30:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:30:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:31:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:31:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:31:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:31:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:32:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:32:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:32:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:32:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:32:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:33:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:33:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:33:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:34:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:36:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:36:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:37:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:37:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:37:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:38:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:38:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:38:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:39:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:40:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:41:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:43:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 18:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 18:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 18:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 18:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 18:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 18:43:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3363: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 18:43:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3363 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 18:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 18:43:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:43:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:44:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:44:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:44:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:44:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:45:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:45:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:45:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:46:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:47:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:47:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:47:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:47:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:48:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:48:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:48:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:48:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:49:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:50:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:50:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:51:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:51:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:51:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:52:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:52:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:52:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:52:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:53:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:54:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:54:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:55:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:56:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:57:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:57:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:57:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:58:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:58:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 18:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 18:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 18:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 18:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 18:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 18:58:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3364: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 18:58:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3364 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 18:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 18:58:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:58:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 18:59:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:00:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:00:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:01:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:01:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:02:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:04:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:04:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:04:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:04:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:05:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:06:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:06:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:06:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:07:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:07:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:07:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:08:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:09:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:09:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:09:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:09:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:09:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:10:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:11:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:11:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:11:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:11:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:12:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 19:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 19:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 19:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 19:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 19:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 19:13:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3365: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 19:13:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3365 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 19:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 19:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:13:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:14:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:14:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:15:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:15:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:15:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:15:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:16:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:16:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:17:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:17:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:17:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:18:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:18:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:18:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:18:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:19:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:19:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:19:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:19:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:20:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:20:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:20:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:21:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:22:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:22:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:23:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:24:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:25:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:25:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:25:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:26:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:27:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:27:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:27:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 19:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 19:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 19:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 19:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 19:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 19:28:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3366: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 19:28:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3366 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 19:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 19:28:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:28:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:29:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:29:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:29:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:29:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:29:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:30:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:31:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:32:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:32:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:33:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:33:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:33:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:33:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:34:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:34:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:34:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:35:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:36:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:36:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:36:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:37:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:37:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:37:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:37:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:38:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:39:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:39:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:39:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:39:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:39:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:40:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:41:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:41:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:41:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:42:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:42:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:42:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:43:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:43:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:43:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 19:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 19:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 19:43:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 19:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 19:43:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 19:43:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3367: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 19:43:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3367 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 19:43:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 19:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:43:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:44:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:44:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:44:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:45:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:45:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:45:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:46:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:47:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:47:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:47:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:48:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:48:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:48:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:48:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:48:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:49:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:49:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:49:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:49:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:50:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:50:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:50:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:51:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:52:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:52:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:52:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:53:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:53:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:53:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:53:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:54:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:54:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:54:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:55:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:56:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:56:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:57:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:57:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:57:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:57:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 19:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 19:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 19:58:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 19:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 19:58:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 19:58:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3368: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 19:58:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3368 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 19:58:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 19:58:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:58:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:59:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:59:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:59:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 19:59:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:00:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:00:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:00:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:00:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:02:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:02:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:03:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:03:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:03:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:04:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:04:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:04:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:04:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:05:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:05:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:05:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:06:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:06:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:06:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:07:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:07:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:08:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:08:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:08:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:09:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:09:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:09:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:09:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:09:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:10:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:11:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:12:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:12:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:13:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:13:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 20:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 20:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 20:13:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 20:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 20:13:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 20:13:24 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3369: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 20:13:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3369 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 20:13:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 20:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:14:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:14:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:14:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:16:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:16:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:16:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:16:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:17:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:18:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:18:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:18:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:18:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:19:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:19:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:20:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:21:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:21:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:21:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:21:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:22:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:22:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:22:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:23:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:23:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:23:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:23:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:23:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:24:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:24:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:24:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:24:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:26:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:27:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:28:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:28:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:28:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 20:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 20:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 20:28:24 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 20:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 20:28:24 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 20:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3370: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 20:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3370 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 20:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 20:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:29:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:30:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:30:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:30:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:30:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:30:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:31:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:31:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:32:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:32:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:32:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:32:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:33:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:33:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:34:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:35:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:35:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:35:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:35:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:36:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:36:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:36:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:36:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:37:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:37:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:37:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:37:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:38:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:38:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:38:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:39:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:39:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:40:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:40:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:40:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:40:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:41:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:41:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:42:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:43:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 20:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 20:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 20:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 20:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 20:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 20:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3371 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 20:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 20:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3371: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 20:43:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:44:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:44:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:44:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:45:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:45:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:45:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:45:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:46:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:46:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:46:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:46:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:46:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:47:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:48:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:49:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:50:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:50:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:52:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:52:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:52:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:52:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:53:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:53:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:53:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:53:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:54:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:56:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:57:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:57:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:57:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 20:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 20:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 20:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 20:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 20:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 20:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3372: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 20:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3372 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 20:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 20:58:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:58:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:59:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:59:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 20:59:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:00:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:00:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:00:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:02:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:02:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:03:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:03:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:04:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:04:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:05:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:06:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:06:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:06:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:07:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:07:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:08:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:08:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:09:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:10:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:10:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:11:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:11:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:11:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:12:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:12:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:12:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:12:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:13:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 21:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 21:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 21:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 21:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 21:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 21:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3373: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 21:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3373 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 21:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 21:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:13:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:14:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:14:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:14:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:14:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:14:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:15:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:15:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:17:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:17:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:18:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:18:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:18:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:19:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:20:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:20:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:21:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:21:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:21:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:21:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:21:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:22:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:23:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:24:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:24:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:25:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:26:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:26:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:26:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:26:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:27:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:27:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 21:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 21:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 21:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 21:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 21:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 21:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3374: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 21:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3374 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 21:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 21:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:29:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:29:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:29:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:30:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:30:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:31:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:31:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:31:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:32:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:32:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:32:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:33:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:33:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:33:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:34:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:35:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:35:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:35:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:36:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:36:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:37:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:37:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:37:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:37:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:37:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:38:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:38:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:38:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:39:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:39:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:39:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:40:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:40:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:40:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:40:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:41:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:41:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:41:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:42:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:42:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:42:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:43:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 21:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 21:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 21:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 21:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 21:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 21:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3375: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 21:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3375 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 21:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 21:43:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:44:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:44:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:44:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:45:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:45:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:45:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:45:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:45:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:46:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:47:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:47:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:47:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:47:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:48:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:48:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:48:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:48:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:49:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:50:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:50:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:51:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:51:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:51:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:51:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:52:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:52:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:52:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:53:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:53:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:53:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:53:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:53:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:54:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:54:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:54:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:54:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:54:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:55:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:55:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:55:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:55:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:56:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:56:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:57:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:57:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 21:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 21:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 21:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 21:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 21:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 21:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3376: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 21:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3376 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 21:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 21:58:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:58:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:59:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 21:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:00:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:01:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:01:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:01:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:01:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:02:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:04:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:04:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:04:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:04:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:04:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:06:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:06:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:06:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:06:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:08:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:08:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:09:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:09:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:09:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:09:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:10:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:11:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:12:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:12:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:12:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:12:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 22:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 22:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 22:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 22:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 22:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 22:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3377: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 22:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3377 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 22:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 22:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:13:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:13:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:14:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:14:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:15:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:15:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:15:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:16:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:16:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:16:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:17:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:17:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:17:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:18:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:18:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:18:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:18:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:18:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:19:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:19:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:20:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:21:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:21:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:22:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:22:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:22:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:22:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:23:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:23:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:24:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:25:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:25:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:26:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:26:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:26:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:27:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:28:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 22:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 22:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 22:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 22:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 22:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 22:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3378: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 22:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3378 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 22:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 22:28:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:29:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:29:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:30:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:30:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:30:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:31:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:31:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:32:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:32:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:32:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:33:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:33:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:33:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:34:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:35:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:35:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:35:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:36:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:36:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:36:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:36:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:37:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:37:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:37:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:37:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:37:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:38:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:39:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:40:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:41:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:41:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:42:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:43:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 22:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 22:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 22:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 22:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 22:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 22:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3379: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 22:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3379 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 22:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 22:43:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:43:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:44:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:44:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:45:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:45:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:45:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:45:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:46:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:46:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:46:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:46:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:47:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:48:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:48:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:49:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:49:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:49:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:50:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:50:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:50:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:50:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:51:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:51:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:52:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:52:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:53:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:53:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:53:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:55:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:55:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:55:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:55:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:55:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:56:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:56:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:56:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:57:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:57:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:57:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 22:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 22:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 22:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 22:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 22:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 22:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3380: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 22:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3380 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 22:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 22:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 22:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:00:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:01:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:02:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:03:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:04:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:04:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:04:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:05:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:06:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:06:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:06:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:07:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:07:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:08:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:10:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:11:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:12:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:12:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:13:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 23:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 23:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 23:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 23:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 23:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 23:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3381: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 23:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3381 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 23:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 23:13:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:13:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:13:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:14:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:16:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:17:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:17:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:18:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:18:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:20:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:21:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:21:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:21:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:22:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:22:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:23:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:23:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:24:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:24:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:24:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:25:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:25:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:26:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:26:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:27:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:27:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 23:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 23:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 23:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 23:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 23:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 23:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3382: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 23:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3382 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 23:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 23:28:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:29:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:31:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:32:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:32:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:33:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:33:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:33:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:33:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:34:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:35:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:36:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:37:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:37:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:37:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:38:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:38:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:39:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:39:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:39:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:39:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:40:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:40:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:41:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:41:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:41:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:41:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:42:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:42:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 23:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 23:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 23:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 23:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 23:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 23:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3383: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 23:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3383 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 23:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 23:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:46:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:47:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:48:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:48:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:50:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:51:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:51:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:51:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:52:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:52:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:52:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:53:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:54:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 13 23:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 13 23:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 13 23:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 13 23:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 13 23:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 13 23:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 13 23:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3384: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 13 23:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3384 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 13 23:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 13 23:58:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:00:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:00:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:00:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:01:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:02:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:02:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:04:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:04:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:05:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:05:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:06:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:07:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:07:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:08:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:09:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:12:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 00:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 00:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 00:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 00:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 00:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 00:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3385: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 00:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3385 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 00:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 00:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:14:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:14:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:14:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:14:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:15:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:15:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:17:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:17:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:23:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:25:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:25:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:26:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:26:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:27:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:27:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 00:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 00:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 00:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 00:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 00:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 00:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3386: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 00:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3386 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 00:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 00:30:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:31:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:32:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:32:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:36:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:36:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:36:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:38:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:38:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:38:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:39:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:39:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:41:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:42:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 00:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 00:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 00:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 00:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 00:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 00:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3387: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 00:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3387 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 00:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 00:43:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:44:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:53:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:53:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:54:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:54:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:58:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 00:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 00:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 00:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 00:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 00:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 00:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 00:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3388: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 00:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3388 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 00:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 00:58:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:07:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:09:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:10:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:12:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 01:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 01:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 01:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 01:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 01:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 01:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3389: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 01:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3389 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 01:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 01:13:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:13:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:13:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:14:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:17:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:21:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:22:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:22:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:23:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:28:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 01:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 01:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 01:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 01:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 01:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 01:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3390: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 01:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3390 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 01:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 01:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:29:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:29:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:31:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:32:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:34:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:34:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:34:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:35:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:37:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:39:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 01:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 01:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 01:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 01:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 01:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 01:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3391: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 01:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3391 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 01:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 01:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:47:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 01:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 01:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 01:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 01:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 01:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 01:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3392: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 01:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3392 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 01:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 01:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 01:59:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:01:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:02:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:03:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:03:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 02:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 02:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 02:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 02:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 02:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 02:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3393 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 02:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 02:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3393: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 02:20:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:20:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:23:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:23:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 02:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 02:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 02:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 02:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 02:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 02:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3394: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 02:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3394 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 02:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 02:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:31:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:32:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:32:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:33:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:38:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:39:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:42:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:43:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 02:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 02:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 02:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 02:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 02:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 02:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3395: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 02:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3395 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 02:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 02:44:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:44:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:56:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:57:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:57:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:58:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 02:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 02:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 02:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 02:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 02:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 02:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3396: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 02:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3396 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 02:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 02:58:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 02:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:06:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:10:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:10:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 03:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 03:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 03:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 03:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 03:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 03:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3397: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 03:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3397 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 03:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 03:20:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:20:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:22:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:24:03 kcsandnp01 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Feb 14 03:26:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:26:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 03:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 03:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 03:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 03:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 03:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 03:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3398: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 03:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3398 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 03:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 03:30:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:31:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:36:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:37:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:38:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:39:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:41:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 03:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 03:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 03:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 03:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 03:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 03:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3399 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 03:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 03:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3399: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 03:46:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:54:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:57:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 03:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 03:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 03:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 03:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 03:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 03:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3400: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 03:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3400 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 03:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 03:58:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 03:59:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:00:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:11:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 04:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 04:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 04:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 04:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 04:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 04:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3401: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 04:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3401 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 04:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 04:19:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:24:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:24:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:25:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:26:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 04:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 04:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 04:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 04:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 04:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 04:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3402: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 04:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3402 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 04:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 04:28:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:29:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:32:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:38:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:39:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:40:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:40:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:40:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:41:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:41:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:42:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:42:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 04:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 04:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 04:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 04:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 04:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 04:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3403: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 04:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3403 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 04:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 04:49:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:49:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:50:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:50:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:56:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 04:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 04:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 04:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 04:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 04:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 04:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3404: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 04:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3404 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 04:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 04:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:59:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:59:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 04:59:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:01:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:01:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:10:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 05:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 05:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 05:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 05:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 05:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 05:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3405: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 05:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3405 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 05:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 05:19:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:20:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:20:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:26:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 05:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 05:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 05:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 05:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 05:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 05:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3406: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 05:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3406 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 05:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 05:29:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:32:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:34:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:34:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:34:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:34:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:35:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:37:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:38:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:39:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 05:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 05:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 05:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 05:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 05:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 05:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3407: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 05:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3407 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 05:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 05:49:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:49:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:50:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:50:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:53:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:53:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:53:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:54:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 05:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 05:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 05:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 05:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 05:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 05:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 05:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3408: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 05:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3408 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 05:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 06:00:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:00:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:01:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:02:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:05:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:05:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:06:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:07:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:08:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:08:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:09:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:09:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:13:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 06:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 06:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 06:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 06:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 06:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 06:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3409: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 06:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3409 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 06:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 06:13:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:16:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:16:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:16:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:20:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:21:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:21:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:21:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:22:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:22:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:23:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:23:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:24:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:27:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:27:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 06:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 06:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 06:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 06:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 06:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 06:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3410: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 06:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3410 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 06:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 06:30:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:31:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:31:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:32:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:32:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:34:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:35:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:35:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:35:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:36:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:37:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:37:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:40:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:41:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:41:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 06:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 06:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 06:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 06:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 06:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 06:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3411 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 06:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 06:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3411: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 06:44:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:46:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:46:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:47:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:48:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:49:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:50:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:51:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:51:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:51:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:52:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:52:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:53:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:53:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:53:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:55:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:55:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:56:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:56:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:57:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:57:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:57:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 06:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 06:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 06:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 06:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 06:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 06:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 06:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3412: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 06:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3412 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 06:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 06:59:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:00:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:01:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:02:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:03:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:03:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:03:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:03:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:04:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:05:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:05:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:05:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:06:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:07:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:07:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:07:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:07:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:08:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:08:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:08:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:11:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:11:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:12:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:12:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 07:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 07:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 07:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 07:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 07:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 07:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3413: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 07:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3413 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 07:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 07:13:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:13:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:14:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:14:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:14:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:14:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:14:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:15:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:15:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:15:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:16:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:16:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:16:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:18:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:18:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:19:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:19:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:20:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:20:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:20:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:20:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:21:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:22:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:22:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:23:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:23:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:24:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:24:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:25:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:25:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:25:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:26:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:27:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:27:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:28:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 07:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 07:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 07:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 07:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 07:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 07:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3414: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 07:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3414 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 07:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 07:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:28:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:28:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:28:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:29:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:30:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:32:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:33:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:33:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:34:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:34:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:35:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:35:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:35:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:36:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:37:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:37:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:37:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:38:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:38:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:38:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:39:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:39:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:39:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:40:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:41:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:42:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:42:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:42:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:43:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 07:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 07:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 07:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 07:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 07:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 07:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3415: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 07:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3415 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 07:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 07:43:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:43:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:43:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:44:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:44:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:45:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:45:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:45:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:45:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:46:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:46:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:47:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:47:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:47:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:47:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:47:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:49:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:49:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:50:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:50:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:50:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:51:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:51:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:51:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:51:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:51:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:52:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:53:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:54:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:54:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:54:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:55:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:55:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:55:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:55:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:55:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:56:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:57:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 07:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 07:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 07:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 07:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 07:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 07:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3416: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 07:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3416 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 07:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 07:58:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 07:59:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:00:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:00:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:00:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:00:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:02:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:02:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:02:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:03:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:04:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:04:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:04:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:05:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:06:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:07:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:07:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:07:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:09:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:10:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:10:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:11:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:11:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:11:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:11:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:13:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 08:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 08:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 08:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 08:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 08:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 08:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3417: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 08:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3417 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 08:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 08:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:15:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:15:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:17:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:18:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:18:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:18:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:18:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:19:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:19:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:20:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:21:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:21:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:21:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:22:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:22:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:23:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:24:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:24:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:25:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:25:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:25:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:26:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:26:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:26:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:28:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 08:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 08:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 08:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 08:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 08:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 08:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3418: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 08:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3418 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 08:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 08:28:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:28:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:30:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:30:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:30:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:31:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:31:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:31:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:32:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:32:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:32:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:33:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:33:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:33:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:33:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:34:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:34:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:35:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:35:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:36:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:36:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:36:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:37:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:38:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:39:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:39:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:39:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:40:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:40:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:41:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:41:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:41:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:41:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:42:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:42:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:42:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:42:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:43:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:43:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:43:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 08:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 08:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 08:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 08:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 08:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 08:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3419 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 08:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 08:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3419: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 08:43:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:44:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:45:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:45:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:45:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:45:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:45:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:46:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:46:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:47:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:48:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:48:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:49:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:49:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:49:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:49:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:50:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:51:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:51:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:52:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:52:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:53:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:53:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:53:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:53:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:55:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:56:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:56:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:56:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:56:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:57:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:57:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 08:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 08:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 08:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 08:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 08:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 08:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3420: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 08:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3420 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 08:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 08:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:59:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:59:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:59:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:59:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 08:59:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:00:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:00:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:00:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:01:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:02:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:02:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:02:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:02:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:02:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:03:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:03:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:04:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:04:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:04:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:04:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:05:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:05:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:05:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:06:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:07:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:08:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:08:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:09:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:10:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:10:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:10:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:10:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:11:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:11:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:11:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:11:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:11:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:12:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:13:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:13:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 09:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 09:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 09:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 09:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 09:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 09:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3421: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 09:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3421 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 09:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 09:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:14:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:14:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:15:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:15:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:15:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:15:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:16:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:17:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:18:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:18:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:18:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:18:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:19:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:19:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:19:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:19:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:19:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:20:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:20:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:21:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:22:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:22:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:23:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:23:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:24:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:24:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:25:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:25:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:25:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:26:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:26:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:26:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:27:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 09:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 09:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 09:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 09:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 09:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 09:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3422: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 09:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3422 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 09:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 09:28:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:28:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:29:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:29:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:30:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:30:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:30:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:31:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:32:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:32:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:32:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:32:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:33:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:33:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:33:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:34:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:34:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:34:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:35:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:35:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:35:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:35:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:36:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:37:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:37:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:37:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:38:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:39:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:40:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:40:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:40:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:40:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:41:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:41:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:41:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:41:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:42:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 09:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 09:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 09:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 09:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 09:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 09:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3423 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 09:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 09:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3423: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 09:43:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:44:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:44:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:45:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:45:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:45:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:45:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:46:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:46:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:46:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:47:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:47:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:48:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:48:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:48:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:49:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:49:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:49:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:50:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:50:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:50:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:50:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:51:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:51:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:51:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:51:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:51:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:52:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:52:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:52:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:52:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:53:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:54:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:54:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:54:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:54:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:54:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:55:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:56:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:57:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:57:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 09:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 09:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 09:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 09:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 09:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 09:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3424: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 09:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3424 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 09:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 09:58:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:58:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:59:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:59:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 09:59:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:00:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:00:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:00:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:00:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:00:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:01:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:01:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:01:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:02:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:02:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:03:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:03:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:03:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:03:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:04:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:04:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:04:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:04:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:05:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:05:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:05:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:05:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:06:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:06:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:06:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:06:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:06:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:07:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:07:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:08:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:09:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:09:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:09:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:09:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:10:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:10:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:10:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 10:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 10:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 10:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 10:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 10:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 10:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3425: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 10:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3425 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 10:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 10:13:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:13:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:14:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:15:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:16:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:16:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:16:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:18:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:18:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:18:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:18:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:18:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:19:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:19:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:19:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:20:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:20:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:21:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:22:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:23:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:23:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:24:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:25:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:25:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:25:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:26:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:26:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:26:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:26:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:27:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 10:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 10:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 10:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 10:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 10:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 10:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3426: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 10:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3426 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 10:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 10:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:29:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:29:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:29:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:29:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:30:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:30:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:30:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:31:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:31:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:32:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:33:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:33:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:33:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:33:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:34:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:35:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:35:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:36:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:38:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:39:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:40:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:40:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:40:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:40:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:41:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:42:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:43:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 10:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 10:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 10:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 10:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 10:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 10:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3427: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 10:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3427 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 10:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 10:43:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:43:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:43:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:43:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:44:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:44:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:44:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:44:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:44:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:45:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:46:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:46:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:46:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:47:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:47:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:48:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:48:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:48:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:49:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:49:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:49:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:49:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:50:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:50:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:50:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:50:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:51:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:52:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:52:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:52:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:52:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:53:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:53:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:53:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:54:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:54:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:54:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:54:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:55:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:56:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:56:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:56:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:56:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:57:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 10:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 10:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 10:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 10:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 10:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 10:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3428: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 10:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3428 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 10:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 10:58:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 10:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:00:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:00:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:01:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:01:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:01:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:01:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:01:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:02:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:03:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:04:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:04:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:05:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:05:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:05:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:06:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:06:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:06:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:06:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:06:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:07:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:08:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:08:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:09:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:09:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:09:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:10:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:11:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:11:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:11:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:11:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:12:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:12:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:12:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:12:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:12:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 11:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 11:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 11:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 11:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 11:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 11:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3429 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 11:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 11:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3429: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 11:13:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:13:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:14:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:15:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:15:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:15:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:16:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:16:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:16:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:18:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:18:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:18:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:18:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:18:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:20:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:21:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:21:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:21:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:22:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:22:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:22:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:23:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:24:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:25:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:25:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:25:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:26:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:26:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:26:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:26:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:26:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:27:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:27:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:27:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:28:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:28:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 11:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 11:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 11:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 11:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 11:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 11:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3430: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 11:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3430 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 11:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 11:28:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:29:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:29:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:29:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:29:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:30:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:30:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:31:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:32:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:32:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:32:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:33:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:33:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:33:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:35:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:36:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:36:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:36:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:37:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:39:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:40:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:41:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:41:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:41:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:41:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:42:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:43:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 11:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 11:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 11:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 11:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 11:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 11:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3431: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 11:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3431 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 11:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 11:43:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:43:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:43:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:44:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:44:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:44:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:44:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:44:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:45:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:45:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:45:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:46:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:47:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:48:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:50:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:50:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:50:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:51:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:51:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:51:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:51:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:51:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:52:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:52:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:52:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:53:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:53:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:53:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:54:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:55:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:56:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:56:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:56:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:57:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:57:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:57:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:57:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:58:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:58:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 11:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 11:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 11:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 11:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 11:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 11:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3432: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 11:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3432 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 11:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 11:58:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 11:59:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:00:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:00:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:00:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:00:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:01:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:02:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:02:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:02:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:02:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:03:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:03:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:04:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:04:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:04:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:04:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:05:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:05:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:05:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:05:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:05:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:06:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:07:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:07:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:07:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:07:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:08:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:08:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:08:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:08:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:09:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:09:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:09:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:09:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:10:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:10:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:10:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:11:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:12:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:12:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:12:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:12:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:13:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 12:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 12:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 12:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 12:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 12:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 12:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3433 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 12:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 12:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3433: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 12:13:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:14:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:14:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:14:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:15:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:15:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:15:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:18:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:18:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:18:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:18:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:18:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:19:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:19:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:20:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:20:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:20:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:20:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:22:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:23:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:23:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:23:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:23:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:24:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:25:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:26:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:28:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:28:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 12:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 12:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 12:28:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 12:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 12:28:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 12:28:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3434: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 12:28:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3434 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 12:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 12:28:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:28:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:29:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:30:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:30:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:30:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:30:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:30:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:31:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:32:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:32:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:32:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:32:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:32:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:33:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:33:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:33:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:34:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:35:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:35:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:35:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:36:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:38:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:39:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:39:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:39:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:40:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:41:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 12:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 12:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 12:43:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 12:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 12:43:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 12:43:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3435: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 12:43:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3435 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 12:43:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 12:43:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:43:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:44:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:45:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:45:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:46:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:47:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:47:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:47:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:47:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:49:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:49:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:49:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:49:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:49:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:50:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:50:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:51:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:53:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:53:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:53:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:54:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:54:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:54:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:54:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:55:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:56:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:57:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:58:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:58:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 12:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 12:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 12:58:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 12:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 12:58:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 12:58:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3436: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 12:58:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3436 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 12:58:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 12:58:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:58:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:58:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:59:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:59:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 12:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:00:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:00:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:00:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:00:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:01:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:01:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:01:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:04:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:04:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:04:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:04:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:05:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:05:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:05:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:05:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:06:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:07:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:08:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:09:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:09:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:09:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:09:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:10:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:10:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:11:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:11:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:11:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:12:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 13:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 13:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 13:13:25 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 13:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 13:13:25 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 13:13:25 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3437: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 13:13:25 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3437 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 13:13:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 13:13:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:13:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:13:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:13:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:14:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:14:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:14:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:14:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:14:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:15:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:15:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:15:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:15:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:17:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:17:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:17:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:17:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:18:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:18:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:18:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:18:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:19:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:19:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:19:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:20:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:21:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:21:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:22:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:22:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:22:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:22:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:22:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:23:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:23:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:23:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:23:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:24:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:25:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:26:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:27:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:27:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:27:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:27:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:27:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:28:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:28:25 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 13:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 13:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 13:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 13:28:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 13:28:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 13:28:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3438: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 13:28:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3438 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 13:28:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 13:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:29:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:29:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:29:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:29:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:30:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:30:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:30:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:30:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:31:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:33:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:33:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:34:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:34:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:34:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:34:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:34:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:35:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:37:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:37:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:37:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:39:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:39:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:39:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:39:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:40:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:40:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:40:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:40:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:41:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:41:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:41:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:41:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:42:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:42:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:42:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:42:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:43:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 13:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 13:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 13:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 13:43:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 13:43:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 13:43:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3439 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 13:43:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 13:43:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3439: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 13:43:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:44:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:44:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:44:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:45:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:45:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:45:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:46:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:46:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:46:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:46:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:46:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:47:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:47:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:47:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:47:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:48:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:49:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:49:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:49:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:49:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:50:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:50:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:50:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:50:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:51:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:52:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:52:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:52:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:52:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:52:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:53:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:54:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:54:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:54:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:54:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:55:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:55:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:55:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:55:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:56:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:57:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:58:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:58:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:58:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 13:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 13:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 13:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 13:58:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 13:58:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 13:58:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3440: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 13:58:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3440 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 13:58:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 13:58:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 13:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:00:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:01:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:02:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:03:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:03:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:04:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:04:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:04:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:05:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:05:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:06:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:06:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:06:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:06:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:06:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:07:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:07:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:07:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:07:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:08:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:08:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:08:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:09:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:09:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:10:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 14:13:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 14:13:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 14:13:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 14:13:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 14:13:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 14:13:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3441: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 14:13:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3441 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 14:13:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 14:13:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:13:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:14:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:14:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:14:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:14:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:14:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:15:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:16:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:16:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:17:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:17:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:17:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:17:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:18:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:18:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:18:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:18:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:18:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:19:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:20:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:20:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:20:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:20:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:20:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:21:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:22:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:22:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:22:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:22:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:23:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:24:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:24:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:24:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:24:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:25:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:26:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:26:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:27:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:27:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:27:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 14:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 14:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 14:28:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 14:28:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 14:28:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 14:28:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3442: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 14:28:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3442 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 14:28:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 14:28:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:28:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:30:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:30:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:30:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:31:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:31:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:32:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:32:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:34:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:34:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:35:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:36:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:36:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:36:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:37:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:37:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:37:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:38:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:38:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:38:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:39:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:39:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:40:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:41:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:42:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:42:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:42:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:42:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:43:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 14:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 14:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 14:43:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 14:43:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 14:43:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 14:43:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3443: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 14:43:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3443 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 14:43:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 14:43:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:43:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:44:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:44:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:45:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:46:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:46:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:46:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:47:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:48:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:48:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:48:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:48:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:49:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:49:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:50:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:50:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:50:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:50:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:51:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:52:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:52:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:52:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:52:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:53:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:54:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:54:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:54:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:55:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:55:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:56:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:56:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:56:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:57:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:58:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:58:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 14:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 14:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 14:58:26 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 14:58:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 14:58:26 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 14:58:26 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3444: /var/lib/pacemaker/pengine/pe-input-81.bz2
Feb 14 14:58:26 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3444 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-81.bz2): Complete
Feb 14 14:58:26 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 14:58:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:58:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:59:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:59:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:59:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 14:59:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:00:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:01:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:02:18 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 14 15:02:18 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 14 15:02:18 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 07 74 27 00 00 08 00
Feb 14 15:02:18 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 488487
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 1/2) BIP4623E: Exception condition detected by input node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2162. 
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24581] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24581] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24679] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24679] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24589] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24589] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24589] (Msg 3/3) BIP2667E: An error occurred in 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.MQ Out: ANYID_REQ_FROM_ITMX' when writing message to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01'. State = '-1' 'MQW102' '2101' '' 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24589] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI1.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:18 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24589] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23471.0.FDC
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:23 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:27 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23502.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23434.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23434.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23434.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23462.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:28 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:29 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ23447.0.FDC
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24487] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '24487' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24487] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 1/2) BIP4623E: Exception condition detected by input node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI2.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:38 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24578] (Msg 2/2) BIP3910I: Input node 'TCPIP Client In Rcv ISO' has retried a message the maximum number of times. The message is now propagated to the failure terminal. 
Feb 14 15:02:39 kcsandnp01 IIB[24175]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24279] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24541] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24540] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24541] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24540] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24542] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24542] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 1/2) BIP4623E: Exception condition detected by input node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification'. 
Feb 14 15:02:39 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:40 kcsandnp01 IIB[24170]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24170] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 14 15:02:40 kcsandnp01 IIB[24170]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24170] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 38317. 
Feb 14 15:02:40 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 14 15:02:40 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 1/2) BIP4623E: Exception condition detected by input node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification'. 
Feb 14 15:02:40 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24543] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:41 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24559] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24582] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24582] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24572] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24572] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24545] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24545] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24583] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:02:42 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24583] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24575] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 25039] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24575] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 25039] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24556] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24556] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24549] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24549] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 59534] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 59534] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24553] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24553] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24678] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24678] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24552] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24592] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24592] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24546] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24546] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24586] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24586] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24591] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI3.TCPIP Client In Rcv ISO'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24591] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI3.MQ Out: ANYID_REQ_FROM_ITMX'. 
Feb 14 15:02:52 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24591] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:02:58 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=609, rc=7, cib-update=4075, confirmed=false)
Feb 14 15:02:58 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:609 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 14 15:02:58 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1487059378)
Feb 14 15:02:58 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 14 15:02:58 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (8)
Feb 14 15:02:58 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 280: fail-count-wmq=8
Feb 14 15:02:58 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1487059378)
Feb 14 15:02:58 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 282: last-failure-wmq=1487059378
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3445: /var/lib/pacemaker/pengine/pe-input-82.bz2
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: LogActions: Restart iib#011(Started kcsandnp01)
Feb 14 15:02:58 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3446: /var/lib/pacemaker/pengine/pe-input-83.bz2
Feb 14 15:02:58 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: stop iib_stop_0 on kcsandnp01 (local)
Feb 14 15:03:01 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 14 15:03:01 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:03:02 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 14 15:03:02 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.38317', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 14 15:03:02 kcsandnp01 IIB[38317]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 38317] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 14 15:03:02 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24590] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 14 15:03:02 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24590] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:03:02 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24576] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 14 15:03:02 kcsandnp01 IIB[24314]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 24576] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:03:03 kcsandnp01 IIB[24170]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 24170] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 14 15:03:22 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 26.680000
Feb 14 15:03:32 kcsandnp01 abrt[39308]: Saved core dump of pid 24314 (/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-14-15:03:14-24314 (1687736320 bytes)
Feb 14 15:03:32 kcsandnp01 abrtd: Directory 'ccpp-2017-02-14-15:03:14-24314' creation detected
Feb 14 15:03:32 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.6/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 14 15:03:32 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-14-15:03:14-24314' exited with 1
Feb 14 15:03:32 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-14-15:03:14-24314'
Feb 14 15:03:32 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=613, rc=0, cib-update=4079, confirmed=true)
Feb 14 15:03:32 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 14 15:03:32 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:609 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=615, rc=0, cib-update=4080, confirmed=true)
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start wmq_start_0 on kcsandnp01 (local)
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=616, rc=0, cib-update=4081, confirmed=true)
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=617, rc=0, cib-update=4082, confirmed=false)
Feb 14 15:03:33 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 21: start iib_start_0 on kcsandnp01 (local)
Feb 14 15:03:35 kcsandnp01 IIB[40382]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40382] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 14 15:03:35 kcsandnp01 IIB[40409]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40409] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 40414. 
Feb 14 15:03:35 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40414] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 14 15:03:37 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40414] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 14 15:03:37 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40503] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 14 15:03:37 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40541] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 24314, has shutdown. 
Feb 14 15:03:38 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40545] (Msg 1/1) BIP2208I: Integration server (64) started: process '40545'; thread '40545'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 14 15:03:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:03:44 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40544] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 14 15:03:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40769] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 14 15:03:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40769] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 14 15:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:03:52 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40769] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 14 15:03:52 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 36.369999
Feb 14 15:03:52 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40812] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 14 15:03:58 kcsandnp01 IIB[40504]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 40547] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 14 15:04:03 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40765] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 14 15:04:04 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=618, rc=0, cib-update=4083, confirmed=true)
Feb 14 15:04:04 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 14 15:04:04 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40765] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 14 15:04:04 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40767] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 14 15:04:06 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=619, rc=0, cib-update=4084, confirmed=false)
Feb 14 15:04:06 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3446 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-83.bz2): Complete
Feb 14 15:04:06 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 15:04:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:04:22 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 22.750000
Feb 14 15:04:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:04:52 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 14.390000
Feb 14 15:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:05:22 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 8.780000
Feb 14 15:05:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:05:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:05:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:05:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:05:52 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 5.390000
Feb 14 15:05:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:06:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:06:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:06:22 kcsandnp01 crmd[2998]:   notice: throttle_handle_load: High CPU load detected: 3.330000
Feb 14 15:06:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:08:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:09:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:10:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:11:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:11:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:11:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:11:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:12:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:13:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:13:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:13:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:14:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:14:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:14:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:15:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:16:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:16:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:16:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:16:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:17:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:17:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:17:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:18:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:19:06 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 15:19:06 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:19:06 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:19:06 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:19:06 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:19:06 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:19:06 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3447: /var/lib/pacemaker/pengine/pe-input-84.bz2
Feb 14 15:19:06 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3447 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-84.bz2): Complete
Feb 14 15:19:06 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 15:19:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:19:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:19:49 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 14 15:19:49 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 14 15:19:49 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 07 8d 4f 00 00 10 00
Feb 14 15:19:49 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 494927
Feb 14 15:19:49 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:19:49 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:19:49 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39697.0.FDC
Feb 14 15:19:49 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39697.0.FDC
Feb 14 15:19:50 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39697.0.FDC
Feb 14 15:19:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40806] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:19:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40806] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 14 15:19:50 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 14 15:19:50 kcsandnp01 IIB[40414]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40503] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 14 15:19:52 kcsandnp01 IIB[40409]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40409] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 14 15:19:52 kcsandnp01 IIB[40409]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40409] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 60308. 
Feb 14 15:19:52 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 14 15:19:53 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40790] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40790] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40814] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40814] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40807] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40807] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'ESB_LoadCache.ANYID_CACHE_CMD.IN' to queue 'ANYID_CACHE_CMD.IN' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40783] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40783] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40897] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40897] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40887] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40887] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40817] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40817] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40823] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40823] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40784] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40784] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40777] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40777] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40787] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40787] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40780] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40780] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40803] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40803] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40813] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 14 15:19:59 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40813] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39629.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39629.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39629.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39629.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:00 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39657.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:01 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ39642.0.FDC
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40729] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '40729' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40729] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40772] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40772] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40771] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40771] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40773] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40773] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40775] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40775] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQ' transport. 
Feb 14 15:20:10 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQTT' transport. 
Feb 14 15:20:10 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40730] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.40545', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 14 15:20:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: not running (node=kcsandnp01, call=619, rc=7, cib-update=4086, confirmed=false)
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:619 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 14 15:20:19 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1487060419)
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 14 15:20:19 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (10)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 284: fail-count-iib=10
Feb 14 15:20:19 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1487060419)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 14 15:20:19 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 286: last-failure-iib=1487060419
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3448: /var/lib/pacemaker/pengine/pe-input-85.bz2
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by status-kcsandnp01-fail-count-iib, fail-count-iib=10: Transient attribute change (modify cib=0.483.95, source=te_update_diff:391, path=/cib/status/node_state[@id='kcsandnp01']/transient_attributes[@id='kcsandnp01']/instance_attributes[@id='status-kcsandnp01']/nvpair[@id='status-kcsandnp01-fail-count-iib'], 0)
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3448 (Complete=0, Pending=0, Fired=0, Skipped=8, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-85.bz2): Stopped
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3449: /var/lib/pacemaker/pengine/pe-input-86.bz2
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 14 15:20:19 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3450: /var/lib/pacemaker/pengine/pe-input-87.bz2
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: stop iib_stop_0 on kcsandnp01 (local)
Feb 14 15:20:19 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:619 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 14 15:20:20 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 14 15:20:20 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:21 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 14 15:20:21 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.60308', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 14 15:20:21 kcsandnp01 IIB[60308]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 60308] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 14 15:20:21 kcsandnp01 IIB[40409]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 40409] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40776] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40776] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40891] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40891] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40821] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40821] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD' to queue 'ANYID_ITMX_MGMT_CMD' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40810] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Res_PO3.TCPIP Client Input'. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40810] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Outbound_Res_PO3.MQ Out: ANYID_RES_FROM_ITMX'. 
Feb 14 15:20:29 kcsandnp01 IIB[40545]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 40810] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 14 15:20:35 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=617, rc=7, cib-update=4090, confirmed=false)
Feb 14 15:20:35 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:617 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 14 15:20:35 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_60000 'modify' on kcsandnp01: Old event (magic=0:7;3:3446:0:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.483.97, source=process_graph_event:555, 0)
Feb 14 15:20:35 kcsandnp01 crmd[2998]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1487060435)
Feb 14 15:20:35 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (9)
Feb 14 15:20:35 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 288: fail-count-wmq=9
Feb 14 15:20:35 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1487060435)
Feb 14 15:20:35 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 290: last-failure-wmq=1487060435
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=621, rc=0, cib-update=4091, confirmed=true)
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3450 (Complete=2, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-87.bz2): Stopped
Feb 14 15:21:20 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 14 15:21:20 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: stop wmq_stop_0 on kcsandnp01 (local)
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:617 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 14 15:21:20 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3451: /var/lib/pacemaker/pengine/pe-input-88.bz2
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=623, rc=0, cib-update=4093, confirmed=true)
Feb 14 15:21:20 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: start wmq_start_0 on kcsandnp01 (local)
Feb 14 15:21:21 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=624, rc=0, cib-update=4094, confirmed=true)
Feb 14 15:21:21 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 14 15:21:21 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 14 15:21:21 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=625, rc=0, cib-update=4095, confirmed=false)
Feb 14 15:21:22 kcsandnp01 IIB[63448]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63448] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 14 15:21:22 kcsandnp01 IIB[63472]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63472] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10006; process ID 63477. 
Feb 14 15:21:22 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63477] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 14 15:21:22 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 14 15:21:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 14 15:21:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 04 c6 43 1f 00 00 10 00
Feb 14 15:21:22 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 80102175
Feb 14 15:21:22 kcsandnp01 kernel: Buffer I/O error on device dm-4, logical block 10012508
Feb 14 15:21:22 kcsandnp01 kernel: lost page write due to I/O error on dm-4
Feb 14 15:21:22 kcsandnp01 kernel: Buffer I/O error on device dm-4, logical block 10012509
Feb 14 15:21:22 kcsandnp01 kernel: lost page write due to I/O error on dm-4
Feb 14 15:21:24 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63477] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 14 15:21:25 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63580] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 14 15:21:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:21:25 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63618] (Msg 1/1) BIP2060W: The integration node has detected that the integration server anyid_interface_itmx, process ID 40545, has shutdown. 
Feb 14 15:21:25 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63623] (Msg 1/1) BIP2208I: Integration server (64) started: process '63623'; thread '63623'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.6/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 14 15:21:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:21:34 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63622] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 14 15:21:40 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63830] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 14 15:21:40 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63830] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 14 15:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:21:42 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63830] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 14 15:21:42 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63885] (Msg 1/1) BIP2648E: Message backed out to a queue; message flow node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 14 15:21:49 kcsandnp01 IIB[63581]: IBM Integration Bus v10006 (ESBXPRD01BK.HTTPListener) [Thread 63625] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 14 15:21:51 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=626, rc=0, cib-update=4096, confirmed=true)
Feb 14 15:21:51 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 14 15:21:52 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=627, rc=0, cib-update=4097, confirmed=false)
Feb 14 15:21:52 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3451 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-88.bz2): Complete
Feb 14 15:21:52 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 15:21:53 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63826] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 14 15:21:54 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63826] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 14 15:21:54 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63828] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 14 15:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:22:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:22:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:22:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:24:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:24:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:25:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:26:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:26:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:26:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:26:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:26:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:28:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:28:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:29:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:29:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:29:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:29:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:29:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:30:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:31:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:31:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:32:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:32:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:32:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:32:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:32:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:33:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:34:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:35:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:35:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:35:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:35:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:36:52 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 15:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3452: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 15:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3452 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 15:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 15:37:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:37:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:37:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:37:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:37:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:37:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:39:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:40:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:40:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:41:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:41:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:42:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:42:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:43:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:44:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:45:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:46:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:47:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:47:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:48:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:48:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:48:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:49:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:49:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:49:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:49:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:50:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:50:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:50:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:50:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 15:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 15:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 15:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 15:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 15:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 15:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3453: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 15:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3453 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 15:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 15:52:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:52:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:52:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:52:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:53:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:53:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:54:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:55:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:55:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:55:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:55:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:56:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:56:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:56:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:57:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:58:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:58:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:59:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 15:59:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:00:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:00:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:00:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:01:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:02:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:02:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:02:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:02:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:02:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:03:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:03:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:03:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:03:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:03:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:04:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:04:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:04:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:04:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:06:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:06:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 16:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 16:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 16:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 16:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 16:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 16:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3454: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 16:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3454 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 16:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 16:07:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:07:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:08:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:09:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:10:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:11:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:11:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:11:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:12:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:12:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:12:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:13:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:13:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:14:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:14:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:14:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:14:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:14:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:15:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:16:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:16:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:16:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:16:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:16:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:17:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:18:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:18:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:19:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:19:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:20:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:21:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:21:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:21:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 16:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 16:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 16:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 16:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 16:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 16:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3455: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 16:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3455 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 16:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 16:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:22:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:22:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:22:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:22:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:23:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:23:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:23:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:24:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:25:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:25:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:25:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:25:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:26:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:27:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:28:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:29:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:30:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:30:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:30:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:30:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:31:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:31:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:32:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:32:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:32:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:32:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:33:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:34:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:34:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:34:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:35:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 16:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 16:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 16:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 16:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 16:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 16:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3456: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 16:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3456 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 16:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 16:37:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:37:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:37:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:37:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:37:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:38:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:39:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:39:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:39:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:39:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:40:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:41:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:41:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:41:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:42:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:42:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:42:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:42:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:42:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:43:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:43:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:43:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:43:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:44:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:44:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:44:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:44:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:44:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:46:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:46:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:46:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:46:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:47:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:47:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:48:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:48:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:48:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:49:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:49:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:49:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:49:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:49:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:50:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:50:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:50:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 16:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 16:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 16:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 16:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 16:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 16:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3457: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 16:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3457 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 16:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 16:52:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:52:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:52:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:53:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:53:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:53:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:54:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:55:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:55:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:56:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:56:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:56:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:56:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:57:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:58:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 16:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:00:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:00:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:02:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:02:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:02:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:02:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:02:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:03:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:04:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:04:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:05:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:05:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:05:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:05:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:05:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:06:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 17:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 17:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 17:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 17:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 17:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 17:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3458: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 17:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3458 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 17:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 17:07:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:07:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:08:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:09:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:10:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:11:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:12:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:12:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:12:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:12:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:13:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:13:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:13:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:14:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:14:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:15:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:15:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:15:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:15:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:16:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:17:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:17:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:17:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:18:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:18:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:18:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:18:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:19:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:19:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:19:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:19:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:20:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 17:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 17:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 17:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 17:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 17:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 17:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3459: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 17:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3459 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 17:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 17:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:23:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:25:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:25:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:25:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:25:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:26:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:27:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:27:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:28:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:29:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:31:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:32:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:32:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:33:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:33:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:33:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:34:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:34:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:34:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 17:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 17:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 17:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 17:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 17:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 17:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3460: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 17:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3460 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 17:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 17:37:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:38:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:38:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:38:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:38:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:38:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:39:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:39:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:39:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:40:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:40:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:40:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:42:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:42:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:43:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:43:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:43:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:43:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:45:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:46:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:46:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:46:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:46:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:47:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:47:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:47:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:48:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:48:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:48:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:48:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:48:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:49:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:49:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:49:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:49:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:49:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:50:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:50:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:50:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:50:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 17:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 17:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 17:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 17:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 17:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 17:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3461: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 17:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3461 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 17:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 17:52:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:52:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:52:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:52:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:53:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:53:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:53:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:53:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:54:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:54:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:55:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:55:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:55:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:55:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:55:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:56:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:56:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:56:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:57:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:58:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 17:59:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:00:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:00:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:02:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:03:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:03:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:03:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:03:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:05:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:05:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:06:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:06:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:06:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:06:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 18:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 18:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 18:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 18:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 18:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 18:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3462: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 18:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3462 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 18:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 18:07:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:07:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:07:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:08:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:08:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:08:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:08:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:09:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:09:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:09:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:09:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:09:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:10:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:10:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:10:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:11:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:11:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:11:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:12:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:12:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:12:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:13:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:13:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:13:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:14:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:14:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:14:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:14:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:15:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:15:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:15:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:15:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:16:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:16:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:16:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:16:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:17:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:17:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:18:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:18:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:19:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:19:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:20:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:21:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:21:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:21:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:21:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 18:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 18:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 18:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 18:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 18:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 18:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3463: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 18:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3463 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 18:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 18:22:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:22:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:22:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:22:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:22:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:22:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:23:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:24:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:24:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:24:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:24:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:25:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:25:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:26:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:26:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:26:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:27:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:27:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:29:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:29:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:29:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:29:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:29:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:30:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:30:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:30:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:30:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:31:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:32:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:32:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:32:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:32:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:33:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:33:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:34:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:35:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 18:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 18:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 18:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 18:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 18:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 18:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3464: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 18:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3464 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 18:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 18:37:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:37:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:37:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:38:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:38:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:39:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:39:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:39:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:39:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:39:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:40:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:41:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:41:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:41:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:42:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:42:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:43:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:43:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:43:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:44:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:45:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:46:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:46:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:46:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:46:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:47:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:47:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:47:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:48:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:48:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:49:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:50:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 18:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 18:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 18:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 18:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 18:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 18:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3465: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 18:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3465 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 18:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 18:52:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:53:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:53:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:53:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:53:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:54:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:54:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:55:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:55:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:55:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:56:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:57:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:58:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:58:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:58:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:58:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 18:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:00:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:00:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:00:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:01:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:03:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:03:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:03:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:03:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:04:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:05:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:05:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:05:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:05:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 19:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 19:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 19:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 19:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 19:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 19:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3466: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 19:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3466 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 19:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 19:07:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:07:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:07:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:07:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:08:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:08:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:09:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:09:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:09:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:09:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:11:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:11:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:11:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:11:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:12:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:12:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:12:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:14:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:15:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:16:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:17:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:18:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:18:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:18:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:19:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:19:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:19:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:20:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 19:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 19:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 19:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 19:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 19:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 19:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3467: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 19:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3467 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 19:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 19:21:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:23:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:23:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:24:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:24:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:25:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:25:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:25:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:26:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:28:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:28:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:28:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:29:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:29:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:29:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:29:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:30:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:30:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:30:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:30:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:31:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:34:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:35:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:35:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:35:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:35:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 19:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 19:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 19:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 19:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 19:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 19:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3468: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 19:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3468 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 19:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 19:37:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:37:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:38:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:38:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:38:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:38:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:39:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:39:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:39:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:39:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:40:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:40:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:40:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:41:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:41:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:41:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:41:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:42:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:42:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:42:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:42:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:43:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:44:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:45:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:45:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:46:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:46:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:47:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:47:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:47:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:48:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:48:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:48:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:48:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:49:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:50:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:50:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:50:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:50:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:51:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:51:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:51:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 19:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 19:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 19:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 19:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 19:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 19:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3469: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 19:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3469 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 19:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 19:51:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:53:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:54:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:54:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:54:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:55:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:55:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:55:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:56:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:57:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:57:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:58:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:58:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:58:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:58:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:58:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 19:59:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:00:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:00:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:00:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:01:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:01:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:03:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:03:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:03:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:03:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:04:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:04:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:04:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:05:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:05:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:06:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:06:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 20:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 20:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 20:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 20:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 20:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 20:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3470: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 20:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3470 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 20:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 20:07:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:07:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:07:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:07:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:09:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:09:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:10:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:11:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:11:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:12:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:12:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:14:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:14:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:15:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:15:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:15:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:16:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:18:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:18:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:18:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:18:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:18:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:19:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:20:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:20:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 20:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 20:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 20:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 20:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 20:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 20:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3471: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 20:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3471 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 20:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 20:22:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:23:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:23:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:24:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:24:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:24:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:26:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:27:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:28:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:28:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:28:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:28:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:30:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:30:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:30:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:31:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:31:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:31:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:31:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:32:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:32:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:32:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:32:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:32:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:33:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:33:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:33:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:33:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:34:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:34:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:34:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:34:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:35:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:35:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:35:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:36:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 20:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 20:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 20:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 20:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 20:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 20:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3472: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 20:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3472 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 20:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 20:37:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:37:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:37:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:38:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:39:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:39:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:39:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:39:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:40:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:40:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:41:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:41:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:42:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:43:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:43:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:43:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:44:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:46:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:46:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:46:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:46:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:47:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:48:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:49:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:50:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 20:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 20:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 20:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 20:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 20:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 20:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3473 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 20:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 20:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3473: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 20:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:52:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:52:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:53:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:54:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:54:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:54:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:55:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:56:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:56:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:56:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:57:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 20:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:00:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:00:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:00:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:01:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:01:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:01:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:02:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:03:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:03:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:03:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:03:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:04:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:04:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:04:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:04:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:04:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:05:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:05:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:06:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 21:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 21:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 21:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 21:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 21:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 21:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3474: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 21:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3474 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 21:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 21:06:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:07:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:07:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:07:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:07:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:08:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:09:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:09:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:10:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:10:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:10:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:11:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:12:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:12:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:13:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:13:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:13:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:14:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:15:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:15:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:15:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:15:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:16:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:16:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:16:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:17:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:17:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:17:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:19:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:20:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:20:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:21:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:21:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:21:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 21:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 21:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 21:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 21:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 21:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 21:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3475: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 21:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3475 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 21:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 21:22:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:22:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:22:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:23:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:24:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:24:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:25:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:26:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:27:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:27:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:27:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:28:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:28:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:28:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:28:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:29:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:30:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:30:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:31:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:32:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:33:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:33:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:33:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:33:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:34:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:35:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:35:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:36:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:36:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 21:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 21:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 21:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 21:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 21:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 21:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3476: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 21:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3476 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 21:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 21:37:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:37:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:37:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:37:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:38:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:38:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:38:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:38:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:39:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:40:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:40:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:40:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:41:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:41:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:42:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:42:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:43:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:44:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:44:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:45:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:46:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:46:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:47:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:47:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:47:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:48:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:48:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:48:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:49:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:49:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:49:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:50:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:50:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 21:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 21:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 21:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 21:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 21:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 21:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3477: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 21:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3477 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 21:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 21:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:52:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:53:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:54:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:54:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:54:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:55:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:55:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:56:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:57:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:57:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:57:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:58:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:58:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:58:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:59:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 21:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:00:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:00:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:01:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:02:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:03:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:04:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:05:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:05:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:05:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:06:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:06:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 22:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 22:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 22:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 22:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 22:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 22:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3478: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 22:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3478 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 22:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 22:07:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:07:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:07:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:07:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:08:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:08:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:09:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:09:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:09:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:10:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:10:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:10:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:11:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:11:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:11:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:12:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:13:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:13:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:13:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:14:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:15:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:15:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:16:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:16:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:16:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:17:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:17:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:17:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:18:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:18:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:19:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:19:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:21:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:21:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:21:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 22:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 22:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 22:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 22:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 22:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 22:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3479: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 22:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3479 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 22:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 22:22:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:22:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:23:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:23:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:24:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:25:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:25:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:25:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:26:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:27:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:27:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:27:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:27:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:28:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:28:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:28:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:30:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:30:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:31:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:31:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:31:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:32:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:33:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:33:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:34:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:34:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:34:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:35:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:35:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:35:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:36:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:36:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:36:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 22:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 22:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 22:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 22:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 22:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 22:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3480: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 22:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3480 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 22:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 22:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:37:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:37:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:37:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:37:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:38:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:38:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:38:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:38:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:39:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:39:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:39:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:39:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:40:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:40:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:40:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:40:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:41:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:41:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:41:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:42:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:42:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:42:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:42:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:43:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:44:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:45:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:45:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:46:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:46:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:47:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:47:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:47:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:47:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:48:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:49:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:49:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:49:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:50:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:50:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:50:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:51:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 22:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 22:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 22:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 22:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 22:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 22:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3481: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 22:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3481 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 22:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 22:52:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:53:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:54:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:54:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:54:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:54:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:54:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:55:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:55:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:55:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:55:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:56:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:56:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:56:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:56:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:57:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:58:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:58:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:58:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:59:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 22:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:00:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:00:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:00:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:00:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:01:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:01:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:01:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:02:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:02:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:02:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:03:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:04:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:06:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:06:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 23:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 23:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 23:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 23:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 23:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 23:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3482: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 23:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3482 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 23:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 23:06:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:07:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:07:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:08:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:09:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:09:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:09:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:10:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:10:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:11:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:11:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:13:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:14:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:14:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:14:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:15:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:15:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:15:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:15:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:15:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:16:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:17:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:18:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:18:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:19:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:20:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:20:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 23:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 23:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 23:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 23:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 23:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 23:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3483: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 23:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3483 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 23:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 23:22:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:22:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:22:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:22:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:23:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:23:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:24:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:25:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:25:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:25:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:26:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:27:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:27:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:28:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:28:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:28:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:29:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:30:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:30:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:31:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:31:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:34:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:34:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:34:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:34:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:34:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:35:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:35:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 23:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 23:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 23:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 23:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 23:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 23:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3484 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 23:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 23:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3484: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 23:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:38:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:38:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:39:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:39:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:40:03 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 14 23:40:03 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 14 23:40:03 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Read(10): 28 00 00 00 00 47 00 00 08 00
Feb 14 23:40:03 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 71
Feb 14 23:40:03 kcsandnp01 lrmd[2995]:   notice: operation_finished: ha_lvm_monitor_10000:12288:stderr [   /dev/sdb1: read failed after 0 of 4096 at 4096: Input/output error ]
Feb 14 23:40:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:40:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:41:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:42:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:43:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:44:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:47:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:48:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:51:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 14 23:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 14 23:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 14 23:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 14 23:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 14 23:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 14 23:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3485: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 14 23:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3485 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 14 23:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 14 23:52:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:52:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:52:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:53:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:54:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:54:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:55:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:55:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 14 23:57:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:00:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:00:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:00:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:02:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:04:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:05:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:05:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:06:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:06:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 00:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 00:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 00:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 00:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 00:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 00:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3486: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 00:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3486 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 00:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 00:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:07:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:08:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:09:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:10:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:11:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:11:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:11:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:12:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:12:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:13:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:13:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:15:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:18:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 00:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 00:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 00:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 00:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 00:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 00:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3487: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 00:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3487 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 00:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 00:22:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:22:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:22:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:23:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:24:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:25:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:25:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:25:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:26:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:26:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:26:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:27:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:27:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:28:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:29:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:29:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:29:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:30:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:30:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:35:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:36:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 00:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 00:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 00:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 00:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 00:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 00:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3488: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 00:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3488 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 00:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 00:37:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:39:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:43:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:45:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:45:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:46:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:47:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:49:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:49:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:50:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 00:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 00:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 00:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 00:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 00:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 00:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3489: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 00:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3489 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 00:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 00:57:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:57:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:57:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:58:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 00:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:00:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:00:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:00:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:01:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:02:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:04:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:05:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 01:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 01:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 01:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 01:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 01:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 01:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3490: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 01:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3490 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 01:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 01:07:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:10:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:11:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:12:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:14:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:15:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:16:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:16:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:18:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:19:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 01:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 01:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 01:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 01:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 01:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 01:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3491: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 01:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3491 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 01:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 01:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:25:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:25:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:26:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:26:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:26:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:27:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:34:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:36:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 01:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 01:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 01:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 01:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 01:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 01:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3492: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 01:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3492 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 01:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 01:37:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:37:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:41:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:42:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:42:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:43:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:44:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:44:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:44:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:44:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:46:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:46:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:48:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:49:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 01:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 01:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 01:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 01:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 01:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 01:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3493 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 01:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 01:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3493: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 01:53:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:54:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 01:55:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:00:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:04:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:04:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 02:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 02:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 02:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 02:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 02:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 02:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3494: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 02:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3494 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 02:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 02:15:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:16:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:16:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:18:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 02:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 02:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 02:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 02:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 02:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 02:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3495: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 02:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3495 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 02:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 02:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:24:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:33:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:33:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:36:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:36:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 02:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 02:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 02:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 02:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 02:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 02:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3496: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 02:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3496 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 02:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 02:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:50:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 02:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 02:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 02:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 02:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 02:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 02:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 02:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3497: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 02:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3497 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 02:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 03:03:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:05:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 03:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 03:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 03:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 03:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 03:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 03:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3498: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 03:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3498 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 03:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 03:07:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:08:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:10:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:11:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:13:07 kcsandnp01 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Feb 15 03:16:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:16:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:17:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:18:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:18:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:19:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 03:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 03:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 03:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 03:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 03:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 03:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3499: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 03:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3499 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 03:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 03:23:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 03:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 03:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 03:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 03:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 03:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 03:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3500: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 03:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3500 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 03:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 03:46:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 03:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 03:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 03:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 03:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 03:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 03:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3501: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 03:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3501 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 03:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 03:55:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:55:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 03:55:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 04:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 04:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 04:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 04:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 04:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 04:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3502: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 04:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3502 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 04:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 04:06:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:12:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:14:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:16:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 04:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 04:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 04:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 04:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 04:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 04:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3503: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 04:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3503 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 04:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 04:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:26:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:27:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:28:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:29:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 04:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 04:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 04:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 04:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 04:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 04:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3504: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 04:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3504 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 04:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 04:40:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 04:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 04:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 04:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 04:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 04:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 04:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3505: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 04:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3505 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 04:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 04:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 04:58:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:01:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:01:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:04:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:04:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:05:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 05:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 05:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 05:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 05:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 05:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 05:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3506: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 05:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3506 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 05:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 05:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:11:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 05:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 05:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 05:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 05:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 05:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 05:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3507 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 05:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 05:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3507: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 05:35:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:35:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:35:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:36:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 05:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 05:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 05:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 05:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 05:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 05:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3508: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 05:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3508 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 05:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 05:38:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:38:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:43:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:43:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:47:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:48:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:48:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 05:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 05:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 05:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 05:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 05:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 05:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3509: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 05:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3509 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 05:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 05:52:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:53:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:53:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:59:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:59:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:59:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 05:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:02:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:04:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:05:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:05:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:06:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 06:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 06:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 06:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 06:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 06:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 06:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3510 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 06:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 06:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3510: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 06:06:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:07:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:07:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:08:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:09:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:10:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:12:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:13:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:13:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:13:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:14:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:14:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:17:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:17:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:17:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:17:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:17:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:18:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:18:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:21:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:21:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:21:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 06:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 06:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 06:21:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 06:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 06:21:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 06:21:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3511 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 06:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 06:21:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3511: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 06:22:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:22:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:24:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:25:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:26:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:27:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:27:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:27:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:29:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:29:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:29:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:30:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:30:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:31:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:31:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:32:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:32:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:32:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:32:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:33:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:33:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:36:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:36:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:36:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 06:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 06:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 06:36:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 06:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 06:36:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 06:36:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3512: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 06:36:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3512 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 06:36:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 06:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:38:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:38:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:38:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:39:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:40:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:41:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:42:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:43:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:44:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:48:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:48:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:49:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 06:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 06:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 06:51:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 06:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 06:51:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 06:51:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3513: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 06:51:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3513 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 06:51:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 06:52:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:52:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:53:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:53:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:53:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:54:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:55:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:55:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:56:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:56:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:56:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:56:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:57:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:57:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:58:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:58:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:59:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:59:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 06:59:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:00:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:00:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:00:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:01:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:01:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:02:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:02:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:03:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:03:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:06:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:06:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 07:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 07:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 07:06:53 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 07:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 07:06:53 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 07:06:53 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3514: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 07:06:53 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3514 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 07:06:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 07:07:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:07:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:10:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:11:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:11:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:11:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:11:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:12:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:12:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:14:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:14:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:14:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:15:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:16:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:16:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:17:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:18:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:19:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:19:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:20:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:21:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:21:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:21:53 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 07:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 07:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 07:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 07:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 07:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 07:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3515: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 07:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3515 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 07:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 07:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:23:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:23:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:23:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:23:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:24:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:24:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:24:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:25:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:26:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:26:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:26:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:27:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:27:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:28:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:28:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:28:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:29:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:29:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:29:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:30:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:30:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:31:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:31:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:31:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:32:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:33:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:34:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:34:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:34:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:36:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 07:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 07:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 07:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 07:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 07:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 07:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3516: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 07:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3516 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 07:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 07:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:37:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:37:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:37:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:38:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:39:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:39:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:41:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:41:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:42:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:43:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:43:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:43:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:43:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:44:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:44:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:44:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:44:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:45:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:45:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:45:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:46:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:46:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:46:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:47:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:47:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:48:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:48:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:48:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:49:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:50:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:50:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 07:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 07:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 07:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 07:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 07:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 07:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3517: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 07:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3517 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 07:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 07:52:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:52:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:53:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:53:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:54:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:54:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:54:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:56:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:56:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:56:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:57:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:57:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:57:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:57:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:59:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:59:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:59:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:59:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 07:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:00:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:00:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:00:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:00:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:01:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:02:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:02:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:03:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:03:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:03:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:03:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:03:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:04:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:04:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:04:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:05:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:05:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:05:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 08:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 08:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 08:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 08:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 08:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 08:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3518: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 08:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3518 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 08:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 08:07:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:07:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:07:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:09:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:09:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:10:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:10:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:10:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:10:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:11:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:11:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:11:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:11:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:12:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:13:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:13:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:13:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:14:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:14:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:14:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:14:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:15:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:16:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:16:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:17:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:17:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:17:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:18:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:18:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:18:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:19:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:19:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:20:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:20:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:21:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:21:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 08:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 08:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 08:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 08:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 08:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 08:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3519: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 08:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3519 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 08:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 08:22:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:23:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:24:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:24:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:24:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:24:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:25:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:25:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:25:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:26:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:26:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:27:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:27:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:27:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:28:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:28:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:29:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:30:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:30:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:30:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:31:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:32:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:32:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:32:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:32:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:33:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:33:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:33:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:33:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:35:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:35:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:35:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:35:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 08:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 08:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 08:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 08:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 08:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 08:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3520: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 08:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3520 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 08:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 08:37:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:37:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:38:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:38:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:39:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:39:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:39:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:40:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:40:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:42:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:42:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:42:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:42:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:43:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:44:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:45:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:46:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:46:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:46:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:46:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:47:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:48:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:48:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:48:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:49:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:49:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:50:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:50:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:50:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:50:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:50:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:51:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:51:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:51:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:51:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 08:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 08:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 08:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 08:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 08:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 08:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3521: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 08:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3521 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 08:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 08:52:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:52:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:52:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:52:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:53:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:54:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:54:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:54:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:54:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:55:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:55:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:55:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:55:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:56:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:56:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:57:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:57:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:58:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:58:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:59:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:59:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:59:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:59:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 08:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:00:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:00:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:00:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:01:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:02:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:02:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:02:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:03:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:04:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:04:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:04:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:04:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:05:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:05:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:06:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:06:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 09:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 09:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 09:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 09:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 09:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 09:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3522: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 09:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3522 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 09:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 09:07:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:07:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:07:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:07:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:08:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:09:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:10:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:10:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:10:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:10:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:10:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:12:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:12:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:12:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:12:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:14:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:14:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:14:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:14:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:15:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:16:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:16:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:16:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:16:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:17:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:18:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:19:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:20:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:21:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:21:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:21:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 09:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 09:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 09:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 09:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 09:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 09:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3523: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 09:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3523 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 09:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 09:21:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:22:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:22:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:22:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:23:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:23:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:23:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:23:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:24:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:24:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:24:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:25:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:26:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:27:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:27:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:27:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:28:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:28:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:28:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:29:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:31:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:31:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:32:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:32:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:33:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:34:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:34:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:35:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:35:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 09:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 09:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 09:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 09:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 09:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 09:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3524: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 09:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3524 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 09:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 09:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:37:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:37:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:37:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:37:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:37:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:38:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:39:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:39:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:39:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:39:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:40:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:40:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:40:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:42:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:42:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:42:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:42:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:43:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:44:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:44:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:45:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:46:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:47:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:48:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:48:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:48:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:49:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:49:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:49:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 09:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 09:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 09:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 09:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 09:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 09:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3525: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 09:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3525 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 09:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 09:52:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:52:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:53:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:53:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:54:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:54:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:55:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:55:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:55:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:56:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:56:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:56:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:57:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:59:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:59:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:59:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 09:59:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:00:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:00:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:00:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:00:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:00:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:01:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:01:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:01:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:01:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:01:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:02:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:02:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:02:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:03:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:03:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:03:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:03:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:04:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:04:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:05:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:05:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:05:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:06:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:06:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:06:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 10:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 10:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 10:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 10:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 10:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 10:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3526: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 10:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3526 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 10:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 10:06:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:07:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:07:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:07:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:07:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:08:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:09:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:09:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:09:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:09:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:09:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:10:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:10:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:12:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:12:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:12:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:13:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:13:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:13:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:14:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:14:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:14:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:14:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:15:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:16:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:17:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:17:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:18:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:18:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:18:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:18:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:19:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:19:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:19:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:19:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:20:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:20:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:20:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 10:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 10:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 10:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 10:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 10:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 10:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3527: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 10:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3527 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 10:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 10:22:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:22:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:22:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:22:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:23:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:24:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:24:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:24:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:24:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:25:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:26:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:26:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:28:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:28:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:29:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:29:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:29:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:33:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:33:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:33:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:34:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:35:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:35:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 10:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 10:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 10:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 10:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 10:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 10:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3528: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 10:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3528 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 10:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 10:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:37:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:38:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:39:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:40:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:41:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:42:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:42:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:43:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:44:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:44:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:44:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:46:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:46:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:46:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:46:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:46:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:48:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:49:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:49:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:49:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:49:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:49:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:50:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:50:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:50:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:50:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:50:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 10:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 10:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 10:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 10:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 10:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 10:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3529 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 10:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 10:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3529: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 10:52:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:52:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:53:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:54:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:55:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:56:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:56:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:56:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:57:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:57:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:57:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:58:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:58:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 10:59:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:00:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:00:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:00:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:00:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:00:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:01:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:03:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:03:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:03:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:04:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:04:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:04:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:04:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:05:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:05:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:05:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 11:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 11:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 11:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 11:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 11:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 11:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3530: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 11:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3530 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 11:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 11:06:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:08:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:08:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:08:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:08:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:09:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:10:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:10:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:10:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:11:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:11:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:11:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:11:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:11:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:12:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:12:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:12:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:12:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:12:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:13:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:13:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:13:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:14:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:14:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:14:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:15:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:16:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:16:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:16:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:16:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:16:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:17:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:17:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:17:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:19:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:20:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:20:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:20:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:20:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:20:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 11:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 11:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 11:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 11:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 11:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 11:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3531: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 11:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3531 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 11:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 11:21:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:22:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:23:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:23:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:23:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:23:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:24:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:25:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:25:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:25:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:25:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:25:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:26:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:27:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:27:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:27:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:28:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:29:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:30:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:30:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:30:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:31:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:32:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:32:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:32:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:32:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:33:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:33:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:33:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:33:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:34:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:34:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:34:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:34:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:35:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:36:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:36:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:36:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:36:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 11:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 11:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 11:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 11:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 11:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 11:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3532: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 11:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3532 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 11:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 11:36:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:38:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:38:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:39:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:40:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:40:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:41:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:42:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:43:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:44:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:44:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:44:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:46:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:46:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:48:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:48:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:48:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:49:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:49:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:49:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:49:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:50:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 11:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 11:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 11:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 11:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 11:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 11:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3533: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 11:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3533 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 11:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 11:51:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:52:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:52:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:52:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:52:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:53:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:53:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:53:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:53:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:53:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:55:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:55:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:55:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:55:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:56:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:57:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:57:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:57:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:57:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:59:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:59:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:59:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:59:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 11:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:00:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:00:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:00:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:00:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:01:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:01:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:01:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:02:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:03:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:03:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:03:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:03:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:04:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:04:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:05:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:05:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:05:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:06:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:06:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:06:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:06:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 12:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 12:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 12:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 12:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 12:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 12:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3534: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 12:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3534 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 12:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 12:07:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:07:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:07:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:07:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:07:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:08:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:09:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:10:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:10:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:11:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:11:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:11:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:11:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:12:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:13:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:13:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:13:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:13:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:14:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:14:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:14:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:15:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:15:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:15:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:16:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:16:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:16:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:16:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:17:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:18:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:18:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:18:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:18:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:18:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:19:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:20:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:21:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:21:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:21:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 12:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 12:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 12:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 12:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 12:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 12:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3535: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 12:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3535 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 12:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 12:22:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:22:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:22:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:22:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:22:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:24:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:25:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:25:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:25:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:25:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:26:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:26:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:26:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:26:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:27:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:27:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:27:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:27:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:27:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:28:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:28:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:29:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:29:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:29:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:29:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:30:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:30:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:30:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:30:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:30:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:31:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:31:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:31:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:31:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:31:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:32:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:34:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:34:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:34:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:34:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:34:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:35:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:36:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 12:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 12:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 12:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 12:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 12:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 12:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3536: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 12:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3536 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 12:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 12:37:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:37:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:37:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:38:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:38:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:38:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:39:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:39:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:39:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:41:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:41:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:41:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:41:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:42:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:43:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:44:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:44:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:44:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:44:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:45:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:45:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:45:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:45:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:45:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:46:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:47:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:48:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:48:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:48:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:48:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:49:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:49:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:49:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:49:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:49:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:50:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:50:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:50:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 12:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 12:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 12:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 12:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 12:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 12:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3537: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 12:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3537 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 12:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 12:51:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:52:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:52:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:52:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:52:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:53:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:54:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:54:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:54:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:55:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:55:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:55:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:56:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:56:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:56:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:56:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:59:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:59:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:59:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:59:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 12:59:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:00:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:01:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:02:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:02:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:02:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:03:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:03:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:03:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:04:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:04:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:04:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:04:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:05:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:06:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:06:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 13:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 13:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 13:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 13:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 13:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 13:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3538: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 13:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3538 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 13:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 13:06:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:07:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:07:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:08:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:08:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:08:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:08:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:09:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:09:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:09:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:09:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:10:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:11:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:12:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:13:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:13:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:14:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:15:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:16:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:16:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:16:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:16:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:17:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:17:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:17:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:18:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:19:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:20:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:21:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:21:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 13:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 13:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 13:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 13:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 13:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 13:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3539: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 13:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3539 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 13:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 13:21:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:22:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:23:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:23:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:23:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:23:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:23:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:25:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:25:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:25:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:26:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:26:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:26:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:27:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:27:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:28:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:28:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:28:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:28:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:29:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:29:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:29:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:29:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:30:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:32:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:33:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:34:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:35:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:35:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:35:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:36:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 13:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 13:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 13:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 13:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 13:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 13:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3540: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 13:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3540 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 13:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 13:37:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:37:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:37:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:37:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:38:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:38:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:38:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:39:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:40:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:40:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:41:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:42:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:43:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:44:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:44:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:44:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:45:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:45:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:47:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:49:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:50:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:50:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:50:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:50:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:51:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:51:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 13:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 13:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 13:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 13:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 13:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 13:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3541: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 13:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3541 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 13:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 13:52:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:52:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:52:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:52:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:53:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:53:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:53:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:53:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:53:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:54:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:54:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:54:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:54:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:54:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:55:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:55:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:55:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:56:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:57:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:57:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:57:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:57:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:58:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:58:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:58:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:59:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:59:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:59:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 13:59:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:00:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:00:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:00:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:00:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:01:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:01:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:02:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:02:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:02:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:02:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:02:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:03:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:03:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:04:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 14:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 14:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 14:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 14:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 14:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 14:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3542: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 14:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3542 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 14:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 14:07:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:07:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:07:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:07:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:07:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:07:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:08:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:08:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:08:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:08:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:09:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:11:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:11:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:11:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:11:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:12:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:13:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:13:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:13:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:13:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:14:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:14:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:14:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:15:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:16:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:17:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:18:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:18:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:18:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:18:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:19:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:19:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:20:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:20:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:20:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:20:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 14:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 14:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 14:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 14:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 14:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 14:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3543: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 14:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3543 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 14:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 14:22:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:22:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:22:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:23:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:24:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:24:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:25:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:27:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:28:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:28:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:28:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:29:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:29:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:29:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:29:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:30:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:30:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:31:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:31:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:31:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:32:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:32:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:32:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:32:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:33:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:34:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:34:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:34:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:35:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:35:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:35:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 14:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 14:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 14:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 14:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 14:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 14:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3544: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 14:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3544 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 14:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 14:36:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:37:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:37:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:38:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:38:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:38:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:39:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:40:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:40:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:40:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:40:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:40:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:41:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:42:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:42:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:42:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:43:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:43:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:43:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:43:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:43:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:44:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:44:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:44:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:44:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:46:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:46:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:46:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:47:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:47:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:47:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:47:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:47:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:48:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:48:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:48:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:48:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:48:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:49:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:50:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:50:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:50:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:50:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:50:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:51:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:51:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 14:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 14:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 14:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 14:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 14:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 14:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3545: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 14:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3545 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 14:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 14:52:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:52:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:52:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:52:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:52:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:53:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:54:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:54:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:55:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:55:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:56:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:56:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:56:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:56:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:56:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:57:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:57:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:57:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:58:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:59:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:59:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:59:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:59:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 14:59:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:00:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:00:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:00:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:01:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:02:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:02:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:02:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:03:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:03:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:03:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:04:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:05:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:05:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:05:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:06:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:06:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:06:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:06:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 15:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 15:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 15:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 15:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 15:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 15:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3546 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 15:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 15:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3546: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 15:07:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:07:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:07:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:07:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:09:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:09:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:09:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:09:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:09:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:10:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:11:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:12:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:13:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:14:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:14:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:14:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:14:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:15:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:15:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:15:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:15:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:17:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:18:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:18:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:18:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:19:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:19:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:19:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:19:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:20:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:20:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:20:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 15:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 15:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 15:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 15:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 15:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 15:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3547: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 15:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3547 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 15:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 15:21:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:23:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:23:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:23:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:23:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:24:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:24:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:24:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:25:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:26:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:26:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:26:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:26:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:28:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:28:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:28:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:28:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:29:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:29:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:29:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:30:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:30:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:30:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:30:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:31:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:31:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:32:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:32:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:32:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:33:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:33:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:33:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:33:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:33:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:34:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:35:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:35:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:35:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:35:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 15:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 15:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 15:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 15:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 15:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 15:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3548 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 15:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 15:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3548: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 15:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:37:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:37:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:37:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:38:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:39:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:39:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:40:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:42:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:42:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:42:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:42:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:43:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:43:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:43:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:43:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:44:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:45:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:46:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:47:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:47:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:47:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:48:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:48:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:48:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:49:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:51:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:51:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:51:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:51:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 15:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 15:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 15:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 15:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 15:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 15:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3549: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 15:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3549 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 15:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 15:52:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:52:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:52:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:52:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:52:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:53:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:55:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:56:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:56:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:56:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:56:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:56:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:57:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:58:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:58:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:58:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:59:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:59:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 15:59:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:00:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:01:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:01:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:01:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:03:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:03:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:03:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:03:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:04:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:04:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:04:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:05:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:05:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:05:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:05:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 16:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 16:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 16:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 16:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 16:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 16:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3550: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 16:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3550 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 16:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 16:06:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:07:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:07:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:08:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:09:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:09:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:10:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:10:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:10:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:11:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:13:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:13:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:13:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:13:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:14:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:14:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:14:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:14:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:14:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:15:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:15:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:15:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:15:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:16:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:16:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:16:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:16:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:17:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:17:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:17:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:17:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:18:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:19:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:19:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:19:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:20:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:20:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:20:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 16:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 16:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 16:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 16:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 16:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 16:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3551: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 16:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3551 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 16:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 16:22:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:22:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:23:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:23:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:23:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:23:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:24:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:25:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:25:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:25:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:25:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:26:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:26:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:26:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:27:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:27:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:27:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:28:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:29:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:30:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:30:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:30:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:30:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:31:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:31:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:31:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:31:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:31:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:32:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:32:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:32:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:32:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:32:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:33:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:34:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:35:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:35:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:36:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:36:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:36:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:36:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 16:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 16:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 16:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 16:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 16:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 16:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3552: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 16:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3552 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 16:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 16:36:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:37:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:38:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:38:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:38:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:39:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:39:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:39:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:39:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:40:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:41:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:42:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:42:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:43:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:43:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:43:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:44:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:44:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:44:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:47:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:48:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:49:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:49:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:49:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:49:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:49:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:50:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 16:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 16:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 16:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 16:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 16:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 16:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3553: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 16:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3553 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 16:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 16:52:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:52:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:53:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:54:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:54:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:55:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:56:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:56:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:56:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:56:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:57:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:57:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:57:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:58:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:59:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:59:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:59:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:59:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 16:59:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:00:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:01:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:01:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:01:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:01:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:01:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:02:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:02:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:02:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:03:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:03:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:03:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:03:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:03:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:04:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:04:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:04:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:04:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:05:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 17:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 17:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 17:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 17:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 17:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 17:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3554: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 17:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3554 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 17:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 17:07:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:07:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:07:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:07:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:08:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:09:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:09:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:09:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:09:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:09:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:10:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:10:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:10:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:10:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:10:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:11:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:12:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:12:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:12:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:12:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:13:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:13:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:13:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:13:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:16:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:16:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:16:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:17:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:18:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:18:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:18:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:19:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:19:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:19:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:20:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 17:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 17:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 17:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 17:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 17:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 17:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3555: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 17:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3555 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 17:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 17:21:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:22:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:22:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:23:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:24:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:24:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:25:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:25:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:25:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:25:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:26:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:27:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:29:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:30:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:30:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:30:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:30:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:31:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:31:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:31:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:31:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:31:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:32:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:33:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:33:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:33:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:33:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:33:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:35:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 17:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 17:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 17:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 17:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 17:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 17:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3556: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 17:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3556 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 17:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 17:36:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:37:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:37:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:37:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:37:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:38:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:39:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:40:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:41:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:41:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:41:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:41:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:41:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:42:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:42:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:42:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:43:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:44:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:44:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:44:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:45:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:45:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:45:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:46:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:46:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:46:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:46:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:47:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:47:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:47:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:47:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:47:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:48:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:48:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:48:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:49:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:49:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:49:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:49:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:50:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:50:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:50:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:50:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:51:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:51:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 17:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 17:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 17:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 17:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 17:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 17:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3557: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 17:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3557 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 17:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 17:52:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:52:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:52:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:52:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:53:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:54:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:54:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:54:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:55:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:55:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:55:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:55:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:55:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:56:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:56:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:56:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:56:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:57:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:58:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:58:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:58:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:59:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:59:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:59:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 17:59:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:00:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:00:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:00:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:00:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:00:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:01:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:02:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:03:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:04:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:05:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:05:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:05:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:05:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:06:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:06:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:06:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:06:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 18:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 18:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 18:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 18:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 18:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 18:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3558: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 18:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3558 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 18:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 18:07:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:07:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:09:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:10:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:10:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:11:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:12:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:13:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:13:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:14:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:15:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:15:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:16:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:16:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:16:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:17:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:17:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:17:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:17:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:17:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:18:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:19:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:19:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:19:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:19:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:20:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:20:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:20:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:20:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:21:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:21:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:21:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:21:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 18:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 18:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 18:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 18:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 18:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 18:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3559 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 18:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 18:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3559: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 18:22:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:23:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:24:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:24:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:25:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:27:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:27:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:27:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:27:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:28:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:29:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:30:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:30:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:30:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:30:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:30:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:31:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:31:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:31:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:31:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:31:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:32:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:34:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:34:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:34:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:35:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:35:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:35:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 18:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 18:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 18:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 18:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 18:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 18:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3560: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 18:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3560 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 18:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 18:37:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:37:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:37:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:38:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:38:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:38:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:38:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:38:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:39:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:39:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:39:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:39:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:39:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:40:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:41:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:42:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:42:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:42:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:42:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:43:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:43:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:43:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:43:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:43:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:44:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:44:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:44:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:44:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:45:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:46:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:47:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:48:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:49:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:49:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:49:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:49:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:49:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:50:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:50:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:50:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:50:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:51:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:51:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:51:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:51:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 18:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 18:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 18:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 18:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 18:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 18:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3561: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 18:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3561 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 18:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 18:52:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:52:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:52:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:52:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:53:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:53:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:54:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:55:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:56:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:57:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:57:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:57:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:57:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:58:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:58:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:58:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:59:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:59:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 18:59:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:00:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:00:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:00:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:00:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:01:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:01:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:01:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:02:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:02:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:02:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:02:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:03:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:04:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:05:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:05:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:05:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:05:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 19:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 19:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 19:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 19:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 19:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 19:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3562: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 19:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3562 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 19:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 19:07:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:07:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:07:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:07:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:08:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:09:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:09:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:09:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:09:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:09:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:10:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:11:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:11:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:11:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:12:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:12:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:12:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:12:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:13:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:14:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:14:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:14:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:14:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:14:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:15:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:15:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:15:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:15:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:16:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:16:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:16:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:16:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:17:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:18:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:18:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:18:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:18:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:18:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:19:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:20:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:20:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:20:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:20:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 19:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 19:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 19:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 19:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 19:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 19:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3563: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 19:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3563 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 19:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 19:21:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:22:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:22:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:22:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:23:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:23:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:23:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:23:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:24:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:24:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:24:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:24:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:24:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:25:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:25:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:25:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:26:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:26:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:27:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:27:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:28:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:29:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:29:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:29:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:29:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:30:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:31:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:31:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:31:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:31:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:31:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:32:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:32:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:32:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:33:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:33:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:33:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:33:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:33:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:34:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:35:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:35:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:35:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:35:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:35:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 19:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 19:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 19:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 19:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 19:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 19:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3564: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 19:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3564 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 19:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 19:37:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:37:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:37:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:38:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:38:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:38:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:38:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:39:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:39:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:39:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:39:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:40:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:40:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:40:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:40:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:41:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:41:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:42:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:43:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:43:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:44:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:44:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:46:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:46:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:46:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:47:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:47:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:47:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:47:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:48:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:48:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:48:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:48:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:48:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:49:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:49:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:49:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:49:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:49:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:50:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:50:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:50:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:50:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:51:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:51:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 19:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 19:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 19:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 19:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 19:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 19:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3565: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 19:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3565 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 19:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 19:52:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:53:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:53:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:53:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:53:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:53:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:55:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:55:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:56:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:56:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:57:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:57:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:57:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:58:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:58:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:58:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:58:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:59:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:59:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:59:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:59:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 19:59:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:00:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:02:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:02:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:02:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:02:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:02:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:04:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:04:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:04:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:04:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:04:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:05:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 20:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 20:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 20:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 20:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 20:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 20:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3566: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 20:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3566 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 20:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 20:07:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:07:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:07:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:07:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:08:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:08:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:08:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:08:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:09:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:10:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:10:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:10:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:10:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:11:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:13:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:14:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:14:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:14:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:14:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:14:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:15:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:15:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:15:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:15:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:16:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:16:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:16:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:17:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:17:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:17:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:18:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:18:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:18:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:19:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:20:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:20:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:20:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 20:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 20:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 20:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 20:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 20:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 20:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3567: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 20:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3567 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 20:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 20:21:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:22:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:23:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:23:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:23:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:23:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:24:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:24:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:24:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:25:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:26:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:26:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:26:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:26:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:26:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:27:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:27:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:27:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:27:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:29:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:30:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:30:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:31:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:31:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:32:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:32:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:33:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:33:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:33:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:34:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:35:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:35:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:35:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:35:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 20:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 20:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 20:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 20:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 20:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 20:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3568: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 20:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3568 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 20:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 20:36:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:37:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:38:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:38:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:38:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:38:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:39:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:40:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:40:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:40:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:41:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:42:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:42:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:42:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:42:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:42:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:43:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:44:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:44:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:44:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:44:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:44:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:45:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:46:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:46:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:46:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:47:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:48:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:49:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:49:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:49:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:51:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:51:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 20:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 20:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 20:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 20:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 20:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 20:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3569: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 20:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3569 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 20:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 20:52:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:52:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:52:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:52:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:52:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:53:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:54:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:54:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:54:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:55:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:55:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:55:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:56:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:56:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:56:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:57:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:58:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:58:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:59:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 20:59:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:00:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:01:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:01:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:01:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:01:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:01:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:02:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:03:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:04:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:04:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:04:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:04:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:04:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:05:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:06:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:06:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:06:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 21:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 21:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 21:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 21:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 21:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 21:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3570: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 21:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3570 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 21:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 21:06:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:07:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:07:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:07:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:07:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:07:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:08:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:09:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:10:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:11:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:11:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:11:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:11:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:11:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:13:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:14:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:14:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:14:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:14:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:15:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:15:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:15:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:15:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:15:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:16:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:17:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:18:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:18:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:18:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:18:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:18:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:19:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:20:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:20:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:20:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:20:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 21:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 21:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 21:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 21:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 21:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 21:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3571: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 21:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3571 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 21:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 21:22:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:22:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:22:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:22:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:22:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:23:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:23:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:23:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:23:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:24:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:25:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:25:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:25:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:25:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:26:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:26:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:26:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:26:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:26:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:27:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:27:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:27:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:27:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:27:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:28:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:29:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:29:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:29:36 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:29:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:30:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:30:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:30:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:30:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:30:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:31:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:31:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:31:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:32:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:32:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:32:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:32:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:32:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:33:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:33:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:33:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:33:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:33:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:34:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:34:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:34:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:34:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:35:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 21:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 21:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 21:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 21:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 21:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 21:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3572: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 21:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3572 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 21:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 21:37:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:37:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:37:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:37:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:37:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:37:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:38:54 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:39:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:40:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:40:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:40:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:40:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:40:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:41:09 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:41:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:42:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:42:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:42:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:43:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:43:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:43:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:43:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:43:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:01 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:44:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:45:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:46:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:46:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:46:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:46:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:47:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:47:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:47:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:47:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:47:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:48:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:48:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:48:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:48:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:48:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:49:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:49:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:49:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:50:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:50:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:50:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:51:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:51:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:51:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:51:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 21:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 21:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 21:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 21:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 21:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 21:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3573: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 21:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3573 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 21:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 21:52:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:52:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:52:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:52:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:53:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:53:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:53:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:54:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:54:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:54:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:54:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:54:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:55:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:55:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:55:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:55:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:55:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:56:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:56:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:56:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:57:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:58:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:58:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:58:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:58:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:58:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 21:59:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:00:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:00:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:00:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:00:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:00:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:01:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:01:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:01:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:02:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:02:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:02:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:02:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:02:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:03:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:03:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:03:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:04:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:05:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 22:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 22:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 22:06:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 22:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 22:06:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 22:06:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3574: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 22:06:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3574 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 22:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 22:07:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:07:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:07:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:26 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:08:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:09:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:10:21 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:10:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:10:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:10:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:11:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:11:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:11:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:11:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:11:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:12:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:12:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:12:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:12:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:13:20 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:13:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:13:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:14:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:14:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:15:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:16:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:16:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:16:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:16:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:17:55 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:18:12 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:18:33 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:18:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:19:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:19:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:19:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:19:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:02 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:20:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:21:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:21:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 22:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 22:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 22:21:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 22:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 22:21:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 22:21:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3575: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 22:21:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3575 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 22:21:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 22:21:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:22:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:22:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:23:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:23:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:23:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:23:48 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:24:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:24:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:22 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:25:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:26:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:26:38 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:27:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:27:39 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:28:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:28:47 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:28:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:28:59 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:29:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:29:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:29:27 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:29:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:29:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:30:16 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:30:32 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:30:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:30:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:31:04 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:31:10 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:31:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:31:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:31:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:32:50 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:34:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:34:30 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:34:37 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:34:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:35:00 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:35:18 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:35:40 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:35:46 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:36:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 22:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 22:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 22:36:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 22:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 22:36:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 22:36:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3576: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 22:36:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3576 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 22:36:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 22:37:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:37:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:37:43 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:38:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:38:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:40:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:40:44 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:40:51 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:40:57 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:15 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:28 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:34 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:41 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:41:52 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:42:05 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:42:11 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:42:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:43:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:43:14 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:43:45 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:44:23 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:44:58 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:07 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:13 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:29 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:42 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:49 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:45:56 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:46:06 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:46:19 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:46:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 22:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 22:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 22:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 22:51:54 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 22:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 22:51:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 22:51:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3577: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 22:51:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3577 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 22:51:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:06:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 23:06:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 23:06:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 23:06:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 23:06:55 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:06:55 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:06:55 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3578: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 23:06:55 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3578 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 23:06:55 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:21:55 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 15 23:21:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 23:21:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 23:21:55 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 23:21:55 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:21:55 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:21:55 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3579: /var/lib/pacemaker/pengine/pe-input-89.bz2
Feb 15 23:21:55 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3579 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-89.bz2): Complete
Feb 15 23:21:55 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:22:07 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI1' of type '.MSGFLOW'. 
Feb 15 23:22:12 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI1' (uuid='f1021881-ae0c-4fb8-a599-78fd8caa94ad',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:12 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI2' of type '.MSGFLOW'. 
Feb 15 23:22:17 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI2' (uuid='89d2e732-fe95-423b-8873-e169b6645e34',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:18 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI3' of type '.MSGFLOW'. 
Feb 15 23:22:21 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI3' (uuid='fe82987a-223b-4f13-950a-91f225a1bfe9',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:21 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI2' of type '.MSGFLOW'. 
Feb 15 23:22:21 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10601' is being 'stopped'. 
Feb 15 23:22:21 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI2' (uuid='115fcee7-94fc-4fdb-a34c-cf4a5f3214b7',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:21 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI1' of type '.MSGFLOW'. 
Feb 15 23:22:22 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10600' is being 'stopped'. 
Feb 15 23:22:22 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI1' (uuid='fea05524-e04b-43c3-9878-02264caaf5c1',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:22 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI4' of type '.MSGFLOW'. 
Feb 15 23:22:27 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI4' (uuid='30982ff9-74ff-4f18-97e1-a79a4ae878bb',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:27 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI3' of type '.MSGFLOW'. 
Feb 15 23:22:28 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10604' is being 'stopped'. 
Feb 15 23:22:28 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI3' (uuid='057daeec-35a1-4826-8389-461e2a8453a9',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:28 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI4' of type '.MSGFLOW'. 
Feb 15 23:22:29 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10605' is being 'stopped'. 
Feb 15 23:22:29 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI4' (uuid='e9e7bb35-27e3-4d16-aa0b-3b64a14c7927',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:35 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO4' of type '.MSGFLOW'. 
Feb 15 23:22:38 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO4' (uuid='27b2ce43-aa05-46ab-9e4f-4ba54e660b4e',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:38 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO1' of type '.MSGFLOW'. 
Feb 15 23:22:38 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO1' (uuid='189f9d64-aa0b-4433-a4c9-cc66245cf48a',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:38 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO2' of type '.MSGFLOW'. 
Feb 15 23:22:43 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO2' (uuid='910cf61a-58a4-45d5-8e4a-7bed87ec9db9',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:43 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO4' of type '.MSGFLOW'. 
Feb 15 23:22:43 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10607' is being 'stopped'. 
Feb 15 23:22:43 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO4' (uuid='c6c9db3c-1d29-49ad-ad20-2957305f1822',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:43 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO3' of type '.MSGFLOW'. 
Feb 15 23:22:44 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO3' (uuid='42d4f107-5a0d-48bb-be41-e3c3b60f7aa5',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:44 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO1' of type '.MSGFLOW'. 
Feb 15 23:22:46 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10602' is being 'stopped'. 
Feb 15 23:22:47 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO1' (uuid='e6e87a5c-4fb2-4f6d-9b5e-83505f69cf64',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:47 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO2' of type '.MSGFLOW'. 
Feb 15 23:22:48 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10603' is being 'stopped'. 
Feb 15 23:22:48 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO2' (uuid='0e793680-4b00-4eca-837f-574388937f5b',type='.MSGFLOW') successfully stopped. 
Feb 15 23:22:48 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2155I: About to 'stop ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO3' of type '.MSGFLOW'. 
Feb 15 23:22:52 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.162.2.17:10606' is being 'stopped'. 
Feb 15 23:22:53 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2271I: Deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO3' (uuid='8ff3c2f7-d454-4e43-b14d-82140cf752db',type='.MSGFLOW') successfully stopped. 
Feb 15 23:24:35 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:24:35 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for ha_lvm on kcsandnp01: unknown error (1)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    iib#011(kcsandnp01)
Feb 15 23:24:35 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: stop iib_stop_0 on kcsandnp01 (local)
Feb 15 23:24:35 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3580: /var/lib/pacemaker/pengine/pe-input-90.bz2
Feb 15 23:24:36 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:24:36 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63791] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 15 23:24:47 kcsandnp01 IIB[63623]: IBM Integration Bus v10006 (ESBXPRD01BK.anyid_interface_itmx) [Thread 63623] (Msg 1/1) BIP2204I: Integration server using process '63623' thread '63623' stopped. 
Feb 15 23:24:54 kcsandnp01 IIB[63477]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63580] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 15 23:24:56 kcsandnp01 IIB[63472]: IBM Integration Bus v10006 (ESBXPRD01BK) [Thread 63472] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 15 23:24:56 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=629, rc=0, cib-update=4227, confirmed=true)
Feb 15 23:24:56 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3580 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-90.bz2): Complete
Feb 15 23:24:56 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: do_lrm_invoke: Forcing the status of all resources to be redetected
Feb 15 23:26:07 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (<null>)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:26:07 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent delete 292: node=kcsandnp01, attr=probe_complete, id=<n/a>, set=(null), section=status
Feb 15 23:26:07 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent delete 294: node=kcsandnp01, attr=probe_complete, id=<n/a>, set=(null), section=status
Feb 15 23:26:07 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   ha_lvm#011(kcsandnp01)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   mqha_fs#011(kcsandnp01)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   vip#011(kcsandnp01)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   wmq#011(kcsandnp01)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   ping_gw:0#011(kcsandnp02)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: LogActions: Start   ping_gw:1#011(kcsandnp01)
Feb 15 23:26:07 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3581: /var/lib/pacemaker/pengine/pe-input-91.bz2
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 12: monitor vmfence_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: monitor vmfence_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 13: monitor ha_lvm_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 5: monitor ha_lvm_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 14: monitor mqha_fs_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 6: monitor mqha_fs_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 15: monitor vip_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 7: monitor vip_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vmfence_monitor_0: not running (node=kcsandnp01, call=645, rc=7, cib-update=4254, confirmed=true)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 16: monitor wmq_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 8: monitor wmq_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 17: monitor iib_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: monitor ping_gw:0_monitor_0 on kcsandnp02
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_0: ok (node=kcsandnp01, call=661, rc=0, cib-update=4255, confirmed=true)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 8 (wmq_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_0 'create' on (null): Event failed (magic=0:0;8:3581:7:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.484.24, source=match_graph_event:344, 0)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 8 (wmq_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_monitor_0: ok (node=kcsandnp01, call=657, rc=0, cib-update=4256, confirmed=true)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 7 (vip_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 7 (vip_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_monitor_0: ok (node=kcsandnp01, call=653, rc=0, cib-update=4257, confirmed=true)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 6 (mqha_fs_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 6 (mqha_fs_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_monitor_0: ok (node=kcsandnp01, call=649, rc=0, cib-update=4258, confirmed=true)
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 5 (ha_lvm_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:07 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 5 (ha_lvm_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:09 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 18 (ping_gw:0_monitor_0) on kcsandnp02 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:09 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 18 (ping_gw:0_monitor_0) on kcsandnp02 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 11: probe_complete probe_complete-kcsandnp02 on kcsandnp02 - no waiting
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3581 (Complete=15, Pending=0, Fired=0, Skipped=15, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-91.bz2): Stopped
Feb 15 23:26:09 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:   notice: LogActions: Start   ping_gw:1#011(kcsandnp01)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3582: /var/lib/pacemaker/pengine/pe-input-92.bz2
Feb 15 23:26:09 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:   notice: LogActions: Start   ping_gw:1#011(kcsandnp01)
Feb 15 23:26:09 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3583: /var/lib/pacemaker/pengine/pe-input-93.bz2
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 9: monitor ha_lvm_monitor_10000 on kcsandnp01 (local)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 12: monitor mqha_fs_monitor_10000 on kcsandnp01 (local)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 15: monitor vip_monitor_30000 on kcsandnp01 (local)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 25: monitor ping_gw_monitor_10000 on kcsandnp02
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=665, rc=0, cib-update=4261, confirmed=false)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_monitor_10000: ok (node=kcsandnp01, call=663, rc=0, cib-update=4262, confirmed=false)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 4: monitor iib_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:09 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 5: monitor ping_gw:1_monitor_0 on kcsandnp01 (local)
Feb 15 23:26:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_monitor_30000: ok (node=kcsandnp01, call=664, rc=0, cib-update=4263, confirmed=false)
Feb 15 23:26:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: ok (node=kcsandnp01, call=662, rc=0, cib-update=4264, confirmed=false)
Feb 15 23:26:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_0: not running (node=kcsandnp01, call=669, rc=7, cib-update=4265, confirmed=true)
Feb 15 23:26:10 kcsandnp01 crmd[2998]:   notice: process_lrm_event: kcsandnp01-iib_monitor_0:669 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is running\nhamqsi_monitor_broker_as: MQSI Integration node ESBXPRD01BK is not running\nhamqsi_monitor_broker_as: Integration node (ESBXPRD01BK) is not running correctly\n ]
Feb 15 23:26:12 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ping_gw_monitor_0: ok (node=kcsandnp01, call=674, rc=0, cib-update=4266, confirmed=true)
Feb 15 23:26:12 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 5 (ping_gw:1_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:12 kcsandnp01 crmd[2998]:   notice: abort_transition_graph: Transition aborted by ping_gw_monitor_0 'create' on (null): Event failed (magic=0:0;5:3583:7:c8f5e01a-cd7c-4fed-8d36-a42c1e8fff63, cib=0.484.39, source=match_graph_event:344, 0)
Feb 15 23:26:12 kcsandnp01 crmd[2998]:  warning: status_from_rc: Action 5 (ping_gw:1_monitor_0) on kcsandnp01 failed (target: 7 vs. rc: 0): Error
Feb 15 23:26:12 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 3: probe_complete probe_complete-kcsandnp01 on kcsandnp01 (local) - no waiting
Feb 15 23:26:12 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3583 (Complete=9, Pending=0, Fired=0, Skipped=3, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-93.bz2): Stopped
Feb 15 23:26:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:26:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:26:12 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3584: /var/lib/pacemaker/pengine/pe-input-94.bz2
Feb 15 23:26:12 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb 15 23:26:12 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 298: probe_complete=true
Feb 15 23:26:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:26:12 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:26:12 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 26: monitor ping_gw_monitor_10000 on kcsandnp01 (local)
Feb 15 23:26:12 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3585: /var/lib/pacemaker/pengine/pe-input-95.bz2
Feb 15 23:26:14 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ping_gw_monitor_10000: ok (node=kcsandnp01, call=675, rc=0, cib-update=4269, confirmed=false)
Feb 15 23:26:14 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3585 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-95.bz2): Complete
Feb 15 23:26:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:33:00 kcsandnp01 IIB[41484]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41484] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 15 23:33:00 kcsandnp01 IIB[41509]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41509] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 41514. 
Feb 15 23:33:00 kcsandnp01 IIB[41514]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41514] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 15 23:33:02 kcsandnp01 IIB[41514]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41514] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 15 23:33:03 kcsandnp01 IIB[41514]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41577] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 15 23:33:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:33:04 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41677] (Msg 1/1) BIP2208I: Integration server (64) started: process '41677'; thread '41677'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 15 23:33:08 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:33:12 kcsandnp01 IIB[41514]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 41672] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 15 23:33:19 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41967] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:33:19 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41967] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 15 23:33:20 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41967] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 15 23:33:25 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:33:26 kcsandnp01 IIB[41576]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 41680] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 15 23:33:28 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41963] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:33:28 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41963] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:33:28 kcsandnp01 IIB[41677]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 41965] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 15 23:33:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:34:31 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:35:03 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:40:14 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:40:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:40:14 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:40:14 kcsandnp01 pengine[2997]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 15 23:40:14 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start iib_start_0 on kcsandnp01 (local)
Feb 15 23:40:14 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3586: /var/lib/pacemaker/pengine/pe-input-96.bz2
Feb 15 23:40:15 kcsandnp01 IIB[47525]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47525] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 15 23:40:15 kcsandnp01 IIB[47565]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47565] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 47573. 
Feb 15 23:40:16 kcsandnp01 IIB[47573]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47573] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 15 23:40:17 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:40:18 kcsandnp01 IIB[47573]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47573] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 15 23:40:19 kcsandnp01 IIB[47573]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47695] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 15 23:40:19 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47735] (Msg 1/1) BIP2208I: Integration server (64) started: process '47735'; thread '47735'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 15 23:40:24 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:40:29 kcsandnp01 IIB[47573]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47732] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 15 23:40:34 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47912] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:40:34 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47912] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 15 23:40:35 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:40:35 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47912] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 15 23:40:42 kcsandnp01 IIB[47696]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 47737] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 15 23:40:45 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=676, rc=0, cib-update=4271, confirmed=true)
Feb 15 23:40:45 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 15 23:40:46 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47908] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:40:46 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=677, rc=0, cib-update=4272, confirmed=false)
Feb 15 23:40:46 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3586 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-96.bz2): Complete
Feb 15 23:40:46 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:40:47 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47908] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:40:47 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47910] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 15 23:40:53 kcsandnp01 kernel: JBD2: Detected IO errors while flushing file data on dm-4-8
Feb 15 23:43:34 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:43:34 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:   notice: LogActions: Move    ha_lvm#011(Started kcsandnp01 -> kcsandnp02)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:   notice: LogActions: Move    mqha_fs#011(Started kcsandnp01 -> kcsandnp02)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:   notice: LogActions: Move    vip#011(Started kcsandnp01 -> kcsandnp02)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:   notice: LogActions: Move    wmq#011(Started kcsandnp01 -> kcsandnp02)
Feb 15 23:43:34 kcsandnp01 pengine[2997]:   notice: LogActions: Move    iib#011(Started kcsandnp01 -> kcsandnp02)
Feb 15 23:43:34 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 24: stop iib_stop_0 on kcsandnp01 (local)
Feb 15 23:43:35 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3587: /var/lib/pacemaker/pengine/pe-input-97.bz2
Feb 15 23:43:35 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47898] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:43:35 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47898] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 15 23:43:55 kcsandnp01 IIB[47735]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 47735] (Msg 1/1) BIP2204I: Integration server using process '47735' thread '47735' stopped. 
Feb 15 23:44:01 kcsandnp01 IIB[47573]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47695] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 15 23:44:02 kcsandnp01 IIB[47565]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 47565] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 15 23:44:04 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=679, rc=0, cib-update=4274, confirmed=true)
Feb 15 23:44:04 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 21: stop wmq_stop_0 on kcsandnp01 (local)
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=681, rc=0, cib-update=4275, confirmed=true)
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: stop vip_stop_0 on kcsandnp01 (local)
Feb 15 23:44:17 kcsandnp01 IPaddr2(vip)[52904]: INFO: IP status = ok, IP_CIP=
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_stop_0: ok (node=kcsandnp01, call=683, rc=0, cib-update=4276, confirmed=true)
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 15: stop mqha_fs_stop_0 on kcsandnp01 (local)
Feb 15 23:44:17 kcsandnp01 Filesystem(mqha_fs)[52962]: INFO: Running stop for /dev/vg_share/mqhalv on /mqha
Feb 15 23:44:17 kcsandnp01 Filesystem(mqha_fs)[52962]: INFO: Trying to unmount /mqha
Feb 15 23:44:17 kcsandnp01 Filesystem(mqha_fs)[52962]: INFO: unmounted /mqha successfully
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_stop_0: ok (node=kcsandnp01, call=685, rc=0, cib-update=4277, confirmed=true)
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 12: stop ha_lvm_stop_0 on kcsandnp01 (local)
Feb 15 23:44:17 kcsandnp01 LVM(ha_lvm)[53041]: INFO: Deactivating volume group vg_share
Feb 15 23:44:17 kcsandnp01 LVM(ha_lvm)[53041]: INFO: 0 logical volume(s) in volume group "vg_share" now active
Feb 15 23:44:17 kcsandnp01 LVM(ha_lvm)[53041]: INFO: LVM Volume vg_share is not available (stopped)
Feb 15 23:44:17 kcsandnp01 LVM(ha_lvm)[53041]: INFO: Stripping tag, pacemaker
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_stop_0: ok (node=kcsandnp01, call=687, rc=0, cib-update=4278, confirmed=true)
Feb 15 23:44:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 13: start ha_lvm_start_0 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 14: monitor ha_lvm_monitor_10000 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 16: start mqha_fs_start_0 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 17: monitor mqha_fs_monitor_10000 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start vip_start_0 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor vip_monitor_30000 on kcsandnp02
Feb 15 23:44:19 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 22: start wmq_start_0 on kcsandnp02
Feb 15 23:44:20 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 23: monitor wmq_monitor_60000 on kcsandnp02
Feb 15 23:44:20 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 25: start iib_start_0 on kcsandnp02
Feb 15 23:44:47 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 26: monitor iib_monitor_60000 on kcsandnp02
Feb 15 23:44:48 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3587 (Complete=20, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-97.bz2): Complete
Feb 15 23:44:48 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:51:42 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:51:42 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: LogActions: Move    ha_lvm#011(Started kcsandnp02 -> kcsandnp01)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: LogActions: Move    mqha_fs#011(Started kcsandnp02 -> kcsandnp01)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: LogActions: Move    vip#011(Started kcsandnp02 -> kcsandnp01)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: LogActions: Move    wmq#011(Started kcsandnp02 -> kcsandnp01)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: LogActions: Move    iib#011(Started kcsandnp02 -> kcsandnp01)
Feb 15 23:51:42 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3588: /var/lib/pacemaker/pengine/pe-input-98.bz2
Feb 15 23:51:42 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 24: stop iib_stop_0 on kcsandnp02
Feb 15 23:52:10 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 21: stop wmq_stop_0 on kcsandnp02
Feb 15 23:52:13 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 18: stop vip_stop_0 on kcsandnp02
Feb 15 23:52:13 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 15: stop mqha_fs_stop_0 on kcsandnp02
Feb 15 23:52:14 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 12: stop ha_lvm_stop_0 on kcsandnp02
Feb 15 23:52:14 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 13: start ha_lvm_start_0 on kcsandnp01 (local)
Feb 15 23:52:14 kcsandnp01 LVM(ha_lvm)[54207]: INFO: Activating volume group vg_share
Feb 15 23:52:15 kcsandnp01 LVM(ha_lvm)[54207]: INFO: Reading all physical volumes. This may take a while... Found volume group "vg_share" using metadata type lvm2 Found volume group "vg00" using metadata type lvm2
Feb 15 23:52:15 kcsandnp01 LVM(ha_lvm)[54207]: INFO: New tag "pacemaker" added to vg_share
Feb 15 23:52:15 kcsandnp01 LVM(ha_lvm)[54207]: INFO: 1 logical volume(s) in volume group "vg_share" now active
Feb 15 23:52:15 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_start_0: ok (node=kcsandnp01, call=688, rc=0, cib-update=4280, confirmed=true)
Feb 15 23:52:15 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 14: monitor ha_lvm_monitor_10000 on kcsandnp01 (local)
Feb 15 23:52:15 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 16: start mqha_fs_start_0 on kcsandnp01 (local)
Feb 15 23:52:15 kcsandnp01 Filesystem(mqha_fs)[54314]: INFO: Running start for /dev/vg_share/mqhalv on /mqha
Feb 15 23:52:15 kcsandnp01 kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: 
Feb 15 23:52:15 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_start_0: ok (node=kcsandnp01, call=690, rc=0, cib-update=4281, confirmed=true)
Feb 15 23:52:15 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: ok (node=kcsandnp01, call=689, rc=0, cib-update=4282, confirmed=false)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 17: monitor mqha_fs_monitor_10000 on kcsandnp01 (local)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 19: start vip_start_0 on kcsandnp01 (local)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_monitor_10000: ok (node=kcsandnp01, call=691, rc=0, cib-update=4283, confirmed=false)
Feb 15 23:52:16 kcsandnp01 IPaddr2(vip)[54407]: INFO: Adding inet address 192.168.141.10/27 with broadcast address 192.168.141.31 to device eth0
Feb 15 23:52:16 kcsandnp01 IPaddr2(vip)[54407]: INFO: Bringing device eth0 up
Feb 15 23:52:16 kcsandnp01 IPaddr2(vip)[54407]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.141.10 eth0 192.168.141.10 auto not_used not_used
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_start_0: ok (node=kcsandnp01, call=692, rc=0, cib-update=4284, confirmed=true)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 20: monitor vip_monitor_30000 on kcsandnp01 (local)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 22: start wmq_start_0 on kcsandnp01 (local)
Feb 15 23:52:16 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_monitor_30000: ok (node=kcsandnp01, call=693, rc=0, cib-update=4285, confirmed=false)
Feb 15 23:52:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=694, rc=0, cib-update=4286, confirmed=true)
Feb 15 23:52:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 23: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 15 23:52:17 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 25: start iib_start_0 on kcsandnp01 (local)
Feb 15 23:52:17 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=695, rc=0, cib-update=4287, confirmed=false)
Feb 15 23:52:18 kcsandnp01 IIB[55293]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55293] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 15 23:52:19 kcsandnp01 IIB[55317]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55317] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 55322. 
Feb 15 23:52:19 kcsandnp01 IIB[55322]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55322] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 15 23:52:21 kcsandnp01 IIB[55322]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55322] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 15 23:52:22 kcsandnp01 IIB[55322]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55389] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 15 23:52:23 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55430] (Msg 1/1) BIP2208I: Integration server (64) started: process '55430'; thread '55430'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 15 23:52:33 kcsandnp01 IIB[55322]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55434] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 15 23:52:37 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55693] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:52:37 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55693] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 15 23:52:38 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55693] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 15 23:52:44 kcsandnp01 IIB[55390]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 55431] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 15 23:52:47 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55689] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:52:48 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=696, rc=0, cib-update=4288, confirmed=true)
Feb 15 23:52:48 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 26: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 15 23:52:48 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55689] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 15 23:52:49 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55691] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 15 23:52:50 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=697, rc=0, cib-update=4289, confirmed=false)
Feb 15 23:52:50 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3588 (Complete=20, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-98.bz2): Complete
Feb 15 23:52:50 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 15 23:58:41 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:58:41 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2153I: About to 'Change' an integration server. 
Feb 15 23:58:44 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2155I: About to 'change ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO1' of type '.MSGFLOW'. 
Feb 15 23:58:44 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP5731W: TCPIP Server Connection manager with connection details: '10.202.2.17:10602' is being 'stopped'. 
Feb 15 23:58:45 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 15 23:59:44 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 15 23:59:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    ha_lvm#011(kcsandnp01)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    mqha_fs#011(kcsandnp01)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    vip#011(kcsandnp01)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    wmq#011(kcsandnp01)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    iib#011(kcsandnp01)
Feb 15 23:59:44 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 16: stop iib_stop_0 on kcsandnp01 (local)
Feb 15 23:59:44 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3589: /var/lib/pacemaker/pengine/pe-input-99.bz2
Feb 15 23:59:45 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 15 23:59:45 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55590] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 16 00:00:05 kcsandnp01 IIB[55430]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 55430] (Msg 1/1) BIP2204I: Integration server using process '55430' thread '55430' stopped. 
Feb 16 00:00:12 kcsandnp01 IIB[55322]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55389] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 16 00:00:14 kcsandnp01 IIB[55317]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 55317] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 16 00:00:14 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=699, rc=0, cib-update=4291, confirmed=true)
Feb 16 00:00:14 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 15: stop wmq_stop_0 on kcsandnp01 (local)
Feb 16 00:00:18 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=701, rc=0, cib-update=4292, confirmed=true)
Feb 16 00:00:18 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 14: stop vip_stop_0 on kcsandnp01 (local)
Feb 16 00:00:18 kcsandnp01 IPaddr2(vip)[1051]: INFO: IP status = ok, IP_CIP=
Feb 16 00:00:18 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation vip_stop_0: ok (node=kcsandnp01, call=703, rc=0, cib-update=4293, confirmed=true)
Feb 16 00:00:18 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 13: stop mqha_fs_stop_0 on kcsandnp01 (local)
Feb 16 00:00:18 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: Running stop for /dev/vg_share/mqhalv on /mqha
Feb 16 00:00:18 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: Trying to unmount /mqha
Feb 16 00:00:18 kcsandnp01 Filesystem(mqha_fs)[1109]: ERROR: Couldn't unmount /mqha; trying cleanup with TERM
Feb 16 00:00:19 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: sending signal TERM to: esbadm   60796 60795  0 Feb15 pts/1    S+     0:00 -bash
Feb 16 00:00:20 kcsandnp01 Filesystem(mqha_fs)[1109]: ERROR: Couldn't unmount /mqha; trying cleanup with TERM
Feb 16 00:00:20 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: sending signal TERM to: esbadm   60796 60795  0 Feb15 pts/1    S+     0:00 -bash
Feb 16 00:00:21 kcsandnp01 Filesystem(mqha_fs)[1109]: ERROR: Couldn't unmount /mqha; trying cleanup with TERM
Feb 16 00:00:21 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: sending signal TERM to: esbadm   60796 60795  0 Feb15 pts/1    S+     0:00 -bash
Feb 16 00:00:22 kcsandnp01 Filesystem(mqha_fs)[1109]: ERROR: Couldn't unmount /mqha; trying cleanup with KILL
Feb 16 00:00:22 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: sending signal KILL to: esbadm   60796 60795  0 Feb15 pts/1    S+     0:00 -bash
Feb 16 00:00:23 kcsandnp01 Filesystem(mqha_fs)[1109]: INFO: unmounted /mqha successfully
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [ umount: /mqha: device is busy. ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [         (In some cases useful info about processes that use ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [          the device is found by lsof(8) or fuser(1)) ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [ umount: /mqha: device is busy. ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [         (In some cases useful info about processes that use ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [          the device is found by lsof(8) or fuser(1)) ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [ umount: /mqha: device is busy. ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [         (In some cases useful info about processes that use ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [          the device is found by lsof(8) or fuser(1)) ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [ umount: /mqha: device is busy. ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [         (In some cases useful info about processes that use ]
Feb 16 00:00:23 kcsandnp01 lrmd[2995]:   notice: operation_finished: mqha_fs_stop_0:1109:stderr [          the device is found by lsof(8) or fuser(1)) ]
Feb 16 00:00:23 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation mqha_fs_stop_0: ok (node=kcsandnp01, call=705, rc=0, cib-update=4294, confirmed=true)
Feb 16 00:00:23 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 12: stop ha_lvm_stop_0 on kcsandnp01 (local)
Feb 16 00:00:23 kcsandnp01 LVM(ha_lvm)[1284]: INFO: Deactivating volume group vg_share
Feb 16 00:00:23 kcsandnp01 LVM(ha_lvm)[1284]: INFO: 0 logical volume(s) in volume group "vg_share" now active
Feb 16 00:00:23 kcsandnp01 LVM(ha_lvm)[1284]: INFO: LVM Volume vg_share is not available (stopped)
Feb 16 00:00:23 kcsandnp01 LVM(ha_lvm)[1284]: INFO: Stripping tag, pacemaker
Feb 16 00:00:24 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ha_lvm_stop_0: ok (node=kcsandnp01, call=707, rc=0, cib-update=4295, confirmed=true)
Feb 16 00:00:24 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3589 (Complete=9, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-99.bz2): Complete
Feb 16 00:00:24 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:04:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:04:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:04:54 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 16 00:04:54 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    ping_gw:0#011(kcsandnp02)
Feb 16 00:04:54 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3590: /var/lib/pacemaker/pengine/pe-input-100.bz2
Feb 16 00:04:54 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 11: stop ping_gw_stop_0 on kcsandnp02
Feb 16 00:04:54 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3590 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-100.bz2): Complete
Feb 16 00:04:54 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:04:57 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:04:57 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp02 after 1000000 failures (max=1000000)
Feb 16 00:04:57 kcsandnp01 pengine[2997]:   notice: stage6: Scheduling Node kcsandnp02 for shutdown
Feb 16 00:04:57 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3591: /var/lib/pacemaker/pengine/pe-input-101.bz2
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3591 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-101.bz2): Complete
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: peer_update_callback: do_shutdown of kcsandnp02 (op 17) is complete
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:04:57 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:04:57 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3592: /var/lib/pacemaker/pengine/pe-input-102.bz2
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3592 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-102.bz2): Complete
Feb 16 00:04:57 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:04:57 kcsandnp01 stonith-ng[2994]:   notice: crm_update_peer_state: st_peer_update_callback: Node kcsandnp02[2] - state is now lost (was member)
Feb 16 00:04:57 kcsandnp01 cib[2993]:   notice: crm_update_peer_state: cib_peer_update_callback: Node kcsandnp02[2] - state is now lost (was member)
Feb 16 00:04:57 kcsandnp01 cib[2993]:   notice: crm_reap_dead_member: Removing kcsandnp02/2 from the membership list
Feb 16 00:04:57 kcsandnp01 cib[2993]:   notice: reap_crm_member: Purged 1 peers with id=2 and/or uname=(null) from the membership cache
Feb 16 00:04:59 kcsandnp01 corosync[2749]:   [QUORUM] Members[1]: 1
Feb 16 00:04:59 kcsandnp01 corosync[2749]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 16 00:04:59 kcsandnp01 crmd[2998]:   notice: crm_update_peer_state: cman_event_callback: Node kcsandnp02[2] - state is now lost (was member)
Feb 16 00:04:59 kcsandnp01 crmd[2998]:  warning: match_down_event: No match for shutdown action on kcsandnp02
Feb 16 00:04:59 kcsandnp01 corosync[2749]:   [CPG   ] chosen downlist: sender r(0) ip(10.1.204.8) ; members(old:2 left:1)
Feb 16 00:04:59 kcsandnp01 crmd[2998]:   notice: peer_update_callback: Stonith/shutdown of kcsandnp02 not matched
Feb 16 00:04:59 kcsandnp01 kernel: dlm: closing connection to node 2
Feb 16 00:04:59 kcsandnp01 corosync[2749]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb 16 00:04:59 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state ]
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (10)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1487060419)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (1000)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-vmfence (INFINITY)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-ha_lvm (1486892319)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (9)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vip (1484026787)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1487060435)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-ha_lvm (1)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb 16 00:04:59 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-vmfence (1484993011)
Feb 16 00:04:59 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:04:59 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3593: /var/lib/pacemaker/pengine/pe-input-103.bz2
Feb 16 00:04:59 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3593 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-103.bz2): Complete
Feb 16 00:04:59 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:06:11 kcsandnp01 init: tty (/dev/tty2) main process (3289) killed by TERM signal
Feb 16 00:06:11 kcsandnp01 init: tty (/dev/tty3) main process (3291) killed by TERM signal
Feb 16 00:06:11 kcsandnp01 init: tty (/dev/tty4) main process (3293) killed by TERM signal
Feb 16 00:06:11 kcsandnp01 init: tty (/dev/tty5) main process (3295) killed by TERM signal
Feb 16 00:06:11 kcsandnp01 init: tty (/dev/tty6) main process (3297) killed by TERM signal
Feb 16 00:06:12 kcsandnp01 pulseaudio[2102]: pid.c: Stale PID file, overwriting.
Feb 16 00:06:13 kcsandnp01 pulseaudio[2102]: main.c: Unable to contact D-Bus: org.freedesktop.DBus.Error.NoServer: Failed to connect to socket /tmp/dbus-igUT3mO9NQ: Connection refused
Feb 16 00:06:13 kcsandnp01 kernel: Removing vmci device
Feb 16 00:06:13 kcsandnp01 kernel: Resetting vmci device
Feb 16 00:06:13 kcsandnp01 kernel: Unregistered vmci device.
Feb 16 00:06:13 kcsandnp01 kernel: vmci 0000:00:07.7: PCI INT A disabled
Feb 16 00:06:13 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: standby (true)
Feb 16 00:06:13 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 327: standby=true
Feb 16 00:06:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:06:13 kcsandnp01 pacemaker: Waiting for shutdown of managed resources
Feb 16 00:06:13 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:06:13 kcsandnp01 pengine[2997]:   notice: LogActions: Stop    ping_gw:0#011(kcsandnp01)
Feb 16 00:06:13 kcsandnp01 crmd[2998]:   notice: te_rsc_command: Initiating action 9: stop ping_gw_stop_0 on kcsandnp01 (local)
Feb 16 00:06:13 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3594: /var/lib/pacemaker/pengine/pe-input-104.bz2
Feb 16 00:06:13 kcsandnp01 crmd[2998]:   notice: process_lrm_event: Operation ping_gw_stop_0: ok (node=kcsandnp01, call=709, rc=0, cib-update=4313, confirmed=true)
Feb 16 00:06:13 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3594 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-104.bz2): Complete
Feb 16 00:06:13 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:06:15 kcsandnp01 pacemaker: Leaving fence domain
Feb 16 00:06:16 kcsandnp01 pacemaker: Stopping fenced 2819
Feb 16 00:06:16 kcsandnp01 pacemaker: Signaling Pacemaker Cluster Manager to terminate
Feb 16 00:06:16 kcsandnp01 pacemaker: Waiting for cluster services to unload
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: pcmk_shutdown_worker: Shuting down Pacemaker
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping crmd: Sent -15 to process 2998
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: crm_shutdown: Requesting shutdown, upper limit is 1200000ms
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_SHUTDOWN cause=C_SHUTDOWN origin=crm_shutdown ]
Feb 16 00:06:16 kcsandnp01 attrd[2996]:   notice: attrd_trigger_update: Sending flush op to all hosts for: shutdown (1487178376)
Feb 16 00:06:16 kcsandnp01 attrd[2996]:   notice: attrd_perform_update: Sent update 329: shutdown=1487178376
Feb 16 00:06:16 kcsandnp01 pengine[2997]:  warning: common_apply_stickiness: Forcing vmfence away from kcsandnp01 after 1000000 failures (max=1000000)
Feb 16 00:06:16 kcsandnp01 pengine[2997]:   notice: stage6: Scheduling Node kcsandnp01 for shutdown
Feb 16 00:06:16 kcsandnp01 pengine[2997]:   notice: process_pe_message: Calculated Transition 3595: /var/lib/pacemaker/pengine/pe-input-105.bz2
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: run_graph: Transition 3595 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-105.bz2): Complete
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: lrm_state_verify_stopped: Stopped 0 recurring operations at shutdown... waiting (0 ops remaining)
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: do_lrm_control: Disconnected from the LRM
Feb 16 00:06:16 kcsandnp01 crmd[2998]:   notice: terminate_cs_connection: Disconnecting from Corosync
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping pengine: Sent -15 to process 2997
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping attrd: Sent -15 to process 2996
Feb 16 00:06:16 kcsandnp01 attrd[2996]:   notice: main: Exiting...
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping lrmd: Sent -15 to process 2995
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping stonith-ng: Sent -15 to process 2994
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: stop_child: Stopping cib: Sent -15 to process 2993
Feb 16 00:06:16 kcsandnp01 cib[2993]:   notice: terminate_cs_connection: Disconnecting from Corosync
Feb 16 00:06:16 kcsandnp01 pacemakerd[2987]:   notice: pcmk_shutdown_worker: Shutdown complete
Feb 16 00:06:18 kcsandnp01 kernel: dlm: closing connection to node 1
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Unloading all Corosync service engines.
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync extended virtual synchrony service
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync configuration service
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync cluster closed process group service v1.01
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync cluster config database access v1.01
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync profile loading service
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: openais checkpoint service B.01.01
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync CMAN membership service 2.90
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [SERV  ] Service engine unloaded: corosync cluster quorum service v0.1
Feb 16 00:06:18 kcsandnp01 corosync[2749]:   [MAIN  ] Corosync Cluster Engine exiting with status 0 at main.c:2055.
Feb 16 00:06:19 kcsandnp01 abrtd: Got signal 15, exiting
Feb 16 00:06:19 kcsandnp01 xinetd[2380]: Exiting...
Feb 16 00:06:19 kcsandnp01 acpid: exiting
Feb 16 00:06:21 kcsandnp01 console-kit-daemon[3345]: WARNING: no sender#012
Feb 16 00:06:21 kcsandnp01 rtkit-daemon[3476]: Demoting known real-time threads.
Feb 16 00:06:21 kcsandnp01 rtkit-daemon[3476]: Successfully demoted thread 2102 of process 2102 (/usr/bin/pulseaudio).
Feb 16 00:06:21 kcsandnp01 rtkit-daemon[3476]: Demoted 1 threads.
Feb 16 00:06:21 kcsandnp01 init: Disconnected from system bus
Feb 16 00:06:21 kcsandnp01 auditd[2164]: The audit daemon is exiting.
Feb 16 00:06:21 kcsandnp01 kernel: type=1305 audit(1487178381.749:3016220): audit_pid=0 old=2164 auid=4294967295 ses=4294967295 res=1
Feb 16 00:06:21 kcsandnp01 kernel: type=1305 audit(1487178381.862:3016221): auid=4294967295 ses=4294967295 op="remove rule" key="access" list=4 res=1
Feb 16 00:06:21 kcsandnp01 kernel: type=1305 audit(1487178381.862:3016222): auid=4294967295 ses=4294967295 op="remove rule" key="access" list=4 res=1
Feb 16 00:06:21 kcsandnp01 kernel: type=1305 audit(1487178381.867:3016223): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Feb 16 00:06:21 kcsandnp01 kernel: Kernel logging (proc) stopped.
Feb 16 00:06:21 kcsandnp01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2198" x-info="http://www.rsyslog.com"] exiting on signal 15.
Feb 16 00:18:23 kcsandnp01 kernel: imklog 5.8.10, log source = /proc/kmsg started.
Feb 16 00:18:23 kcsandnp01 rsyslogd: [origin software="rsyslogd" swVersion="5.8.10" x-pid="2224" x-info="http://www.rsyslog.com"] start
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys cpuset
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys cpu
Feb 16 00:18:23 kcsandnp01 kernel: Linux version 2.6.32-573.el6.x86_64 (mockbuild@x86-027.build.eng.bos.redhat.com) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Wed Jul 1 18:23:37 EDT 2015
Feb 16 00:18:23 kcsandnp01 kernel: Command line: ro root=/dev/mapper/vg00-lvol1 rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg00/lvol2 rd_NO_MD rd_LVM_LV=vg00/lvol1 SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Feb 16 00:18:23 kcsandnp01 kernel: KERNEL supported cpus:
Feb 16 00:18:23 kcsandnp01 kernel:  Intel GenuineIntel
Feb 16 00:18:23 kcsandnp01 kernel:  AMD AuthenticAMD
Feb 16 00:18:23 kcsandnp01 kernel:  Centaur CentaurHauls
Feb 16 00:18:23 kcsandnp01 kernel: Disabled fast string operations
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-provided physical RAM map:
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 0000000000000000 - 000000000009f000 (usable)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000000ca000 - 00000000000cc000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 0000000000100000 - 00000000bfee0000 (usable)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000bfee0000 - 00000000bfeff000 (ACPI data)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000bfeff000 - 00000000bff00000 (ACPI NVS)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000bff00000 - 00000000c0000000 (usable)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000f0000000 - 00000000f8000000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 00000000fffe0000 - 0000000100000000 (reserved)
Feb 16 00:18:23 kcsandnp01 kernel: BIOS-e820: 0000000100000000 - 0000000240000000 (usable)
Feb 16 00:18:23 kcsandnp01 kernel: DMI present.
Feb 16 00:18:23 kcsandnp01 kernel: SMBIOS version 2.4 @ 0xF6B30
Feb 16 00:18:23 kcsandnp01 kernel: Phoenix BIOS detected: BIOS may corrupt low RAM, working around it.
Feb 16 00:18:23 kcsandnp01 kernel: Hypervisor detected: VMware
Feb 16 00:18:23 kcsandnp01 kernel: No AGP bridge found
Feb 16 00:18:23 kcsandnp01 kernel: last_pfn = 0x240000 max_arch_pfn = 0x400000000
Feb 16 00:18:23 kcsandnp01 kernel: x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106
Feb 16 00:18:23 kcsandnp01 kernel: total RAM covered: 8192M
Feb 16 00:18:23 kcsandnp01 kernel: Found optimal setting for mtrr clean up
Feb 16 00:18:23 kcsandnp01 kernel: gran_size: 64K 	chunk_size: 64K 	num_reg: 4  	lose cover RAM: 0G
Feb 16 00:18:23 kcsandnp01 kernel: x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106
Feb 16 00:18:23 kcsandnp01 kernel: last_pfn = 0xc0000 max_arch_pfn = 0x400000000
Feb 16 00:18:23 kcsandnp01 kernel: init_memory_mapping: 0000000000000000-00000000c0000000
Feb 16 00:18:23 kcsandnp01 kernel: init_memory_mapping: 0000000100000000-0000000240000000
Feb 16 00:18:23 kcsandnp01 kernel: RAMDISK: 36524000 - 37fefe28
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Deleted _OSI()
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Deleted _OSI()
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: RSDP 00000000000f6b80 00024 (v02 PTLTD )
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: XSDT 00000000bfee060d 0005C (v01 INTEL  440BX    06040000 VMW  01324272)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: FACP 00000000bfefee73 000F4 (v04 INTEL  440BX    06040000 PTL  000F4240)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: DSDT 00000000bfee122c 1DC47 (v01 PTLTD  Custom   06040000 MSFT 03000001)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: FACS 00000000bfefffc0 00040
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: BOOT 00000000bfee1204 00028 (v01 PTLTD  $SBFTBL$ 06040000  LTP 00000001)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: APIC 00000000bfee0c3d 003C2 (v01 PTLTD  ? APIC   06040000  LTP 00000000)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: MCFG 00000000bfee0c01 0003C (v01 PTLTD  $PCITBL$ 06040000  LTP 00000001)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: SRAT 00000000bfee0709 004F8 (v02 VMWARE MEMPLUG  06040000 VMW  00000001)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: HPET 00000000bfee06d1 00038 (v01 VMWARE VMW HPET 06040000 VMW  00000001)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: WAET 00000000bfee06a9 00028 (v01 VMWARE VMW WAET 06040000 VMW  00000001)
Feb 16 00:18:23 kcsandnp01 kernel: Setting APIC routing to physical flat.
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 0 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 1 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 2 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 3 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 4 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 5 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 6 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 7 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 8 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 9 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 10 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 11 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 12 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 13 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 14 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 15 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 16 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 17 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 18 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 19 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 20 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 21 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 22 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 23 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 24 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 25 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 26 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 27 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 28 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 29 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 30 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 31 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 32 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 33 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 34 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 35 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 36 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 37 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 38 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 39 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 40 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 41 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 42 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 43 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 44 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 45 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 46 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 47 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 48 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 49 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 50 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 51 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 52 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 53 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 54 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 55 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 56 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 57 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 58 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 59 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 60 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 61 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 62 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: PXM 0 -> APIC 63 -> Node 0
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: Node 0 PXM 0 0-a0000
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: Node 0 PXM 0 100000-10000000
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: Node 0 PXM 0 10000000-c0000000
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: Node 0 PXM 0 100000000-240000000
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: Node 0 PXM 0 240000000-2040000000
Feb 16 00:18:23 kcsandnp01 kernel: SRAT: hot plug zone found 240000000 - 2040000000
Feb 16 00:18:23 kcsandnp01 kernel: Bootmem setup node 0 0000000000000000-0000000240000000
Feb 16 00:18:23 kcsandnp01 kernel:  NODE_DATA [0000000000057840 - 000000000008b83f]
Feb 16 00:18:23 kcsandnp01 kernel:  bootmap [0000000000100000 -  0000000000147fff] pages 48
Feb 16 00:18:23 kcsandnp01 kernel: (9 early reservations) ==> bootmem [0000000000 - 0240000000]
Feb 16 00:18:23 kcsandnp01 kernel:  #0 [0000000000 - 0000001000]   BIOS data page ==> [0000000000 - 0000001000]
Feb 16 00:18:23 kcsandnp01 kernel:  #1 [0000006000 - 0000008000]       TRAMPOLINE ==> [0000006000 - 0000008000]
Feb 16 00:18:23 kcsandnp01 kernel:  #2 [0001000000 - 000203cb24]    TEXT DATA BSS ==> [0001000000 - 000203cb24]
Feb 16 00:18:23 kcsandnp01 kernel:  #3 [0036524000 - 0037fefe28]          RAMDISK ==> [0036524000 - 0037fefe28]
Feb 16 00:18:23 kcsandnp01 kernel:  #4 [000009f000 - 0000100000]    BIOS reserved ==> [000009f000 - 0000100000]
Feb 16 00:18:23 kcsandnp01 kernel:  #5 [000203d000 - 000203df9d]              BRK ==> [000203d000 - 000203df9d]
Feb 16 00:18:23 kcsandnp01 kernel:  #6 [0000010000 - 0000012000]          PGTABLE ==> [0000010000 - 0000012000]
Feb 16 00:18:23 kcsandnp01 kernel:  #7 [0000012000 - 0000017000]          PGTABLE ==> [0000012000 - 0000017000]
Feb 16 00:18:23 kcsandnp01 kernel:  #8 [0000017000 - 0000057840]       MEMNODEMAP ==> [0000017000 - 0000057840]
Feb 16 00:18:23 kcsandnp01 kernel: found SMP MP-table at [ffff8800000f6bf0] f6bf0
Feb 16 00:18:23 kcsandnp01 kernel: Zone PFN ranges:
Feb 16 00:18:23 kcsandnp01 kernel:  DMA      0x00000010 -> 0x00001000
Feb 16 00:18:23 kcsandnp01 kernel:  DMA32    0x00001000 -> 0x00100000
Feb 16 00:18:23 kcsandnp01 kernel:  Normal   0x00100000 -> 0x00240000
Feb 16 00:18:23 kcsandnp01 kernel: Movable zone start PFN for each node
Feb 16 00:18:23 kcsandnp01 kernel: early_node_map[4] active PFN ranges
Feb 16 00:18:23 kcsandnp01 kernel:    0: 0x00000010 -> 0x0000009f
Feb 16 00:18:23 kcsandnp01 kernel:    0: 0x00000100 -> 0x000bfee0
Feb 16 00:18:23 kcsandnp01 kernel:    0: 0x000bff00 -> 0x000c0000
Feb 16 00:18:23 kcsandnp01 kernel:    0: 0x00100000 -> 0x00240000
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PM-Timer IO Port: 0x1008
Feb 16 00:18:23 kcsandnp01 kernel: Setting APIC routing to physical flat.
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] enabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x08] lapic_id[0x08] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x09] lapic_id[0x09] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x0a] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x0b] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x0c] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x0d] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x0e] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x0f] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x10] lapic_id[0x10] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x11] lapic_id[0x11] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x12] lapic_id[0x12] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x13] lapic_id[0x13] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x14] lapic_id[0x14] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x15] lapic_id[0x15] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x16] lapic_id[0x16] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x17] lapic_id[0x17] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x18] lapic_id[0x18] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x19] lapic_id[0x19] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x1a] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x1b] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x1c] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x1d] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x1e] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x1f] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x20] lapic_id[0x20] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x21] lapic_id[0x21] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x22] lapic_id[0x22] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x23] lapic_id[0x23] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x24] lapic_id[0x24] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x25] lapic_id[0x25] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x26] lapic_id[0x26] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x27] lapic_id[0x27] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x28] lapic_id[0x28] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x29] lapic_id[0x29] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2a] lapic_id[0x2a] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2b] lapic_id[0x2b] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2c] lapic_id[0x2c] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2d] lapic_id[0x2d] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2e] lapic_id[0x2e] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x2f] lapic_id[0x2f] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x30] lapic_id[0x30] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x31] lapic_id[0x31] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x32] lapic_id[0x32] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x33] lapic_id[0x33] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x34] lapic_id[0x34] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x35] lapic_id[0x35] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x36] lapic_id[0x36] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x37] lapic_id[0x37] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x38] lapic_id[0x38] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x39] lapic_id[0x39] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3a] lapic_id[0x3a] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3b] lapic_id[0x3b] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3c] lapic_id[0x3c] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3d] lapic_id[0x3d] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3e] lapic_id[0x3e] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC (acpi_id[0x3f] lapic_id[0x3f] disabled)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x00] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x01] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x02] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x03] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x04] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x05] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x06] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x07] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x08] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x09] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0a] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0b] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0c] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0d] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0e] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x0f] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x10] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x11] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x12] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x13] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x14] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x15] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x16] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x17] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x18] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x19] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1a] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1b] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1c] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1d] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1e] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x1f] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x20] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x21] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x22] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x23] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x24] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x25] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x26] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x27] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x28] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x29] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2a] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2b] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2c] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2d] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2e] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x2f] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x30] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x31] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x32] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x33] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x34] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x35] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x36] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x37] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x38] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x39] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3a] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3b] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3c] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3d] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3e] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: LAPIC_NMI (acpi_id[0x3f] high edge lint[0x1])
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: IOAPIC (id[0x40] address[0xfec00000] gsi_base[0])
Feb 16 00:18:23 kcsandnp01 kernel: IOAPIC[0]: apic_id 64, version 17, address 0xfec00000, GSI 0-23
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 high edge)
Feb 16 00:18:23 kcsandnp01 kernel: Using ACPI (MADT) for SMP configuration information
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: HPET id: 0x8086af01 base: 0xfed00000
Feb 16 00:18:23 kcsandnp01 kernel: SMP: Allowing 64 CPUs, 62 hotplug CPUs
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 000000000009f000 - 00000000000a0000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000000a0000 - 00000000000ca000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000000ca000 - 00000000000cc000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000000cc000 - 00000000000dc000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000000dc000 - 0000000000100000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000bfee0000 - 00000000bfeff000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000bfeff000 - 00000000bff00000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000c0000000 - 00000000f0000000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000f0000000 - 00000000f8000000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000f8000000 - 00000000fec00000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000fec00000 - 00000000fec10000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000fec10000 - 00000000fee00000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000fee00000 - 00000000fee01000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000fee01000 - 00000000fffe0000
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registered nosave memory: 00000000fffe0000 - 0000000100000000
Feb 16 00:18:23 kcsandnp01 kernel: Allocating PCI resources starting at c0000000 (gap: c0000000:30000000)
Feb 16 00:18:23 kcsandnp01 kernel: Booting paravirtualized kernel on bare hardware
Feb 16 00:18:23 kcsandnp01 kernel: NR_CPUS:4096 nr_cpumask_bits:64 nr_cpu_ids:64 nr_node_ids:1
Feb 16 00:18:23 kcsandnp01 kernel: PERCPU: Embedded 30 pages/cpu @ffff88002f600000 s91224 r8192 d23464 u131072
Feb 16 00:18:23 kcsandnp01 kernel: pcpu-alloc: s91224 r8192 d23464 u131072 alloc=1*2097152
Feb 16 00:18:23 kcsandnp01 kernel: pcpu-alloc: [0] 00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 
Feb 16 00:18:23 kcsandnp01 kernel: pcpu-alloc: [0] 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 
Feb 16 00:18:23 kcsandnp01 kernel: pcpu-alloc: [0] 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 
Feb 16 00:18:23 kcsandnp01 kernel: pcpu-alloc: [0] 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 
Feb 16 00:18:23 kcsandnp01 kernel: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 2064580
Feb 16 00:18:23 kcsandnp01 kernel: Policy zone: Normal
Feb 16 00:18:23 kcsandnp01 kernel: Kernel command line: ro root=/dev/mapper/vg00-lvol1 rd_NO_LUKS LANG=en_US.UTF-8 rd_LVM_LV=vg00/lvol2 rd_NO_MD rd_LVM_LV=vg00/lvol1 SYSFONT=latarcyrheb-sun16  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet
Feb 16 00:18:23 kcsandnp01 kernel: PID hash table entries: 4096 (order: 3, 32768 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: Checking aperture...
Feb 16 00:18:23 kcsandnp01 kernel: No AGP bridge found
Feb 16 00:18:23 kcsandnp01 kernel: Memory: 8154764k/9437184k available (5394k kernel code, 1049156k absent, 233264k reserved, 7014k data, 1296k init)
Feb 16 00:18:23 kcsandnp01 kernel: Hierarchical RCU implementation.
Feb 16 00:18:23 kcsandnp01 kernel: NR_IRQS:33024 nr_irqs:920
Feb 16 00:18:23 kcsandnp01 kernel: Extended CMOS year: 2000
Feb 16 00:18:23 kcsandnp01 kernel: Console: colour VGA+ 80x25
Feb 16 00:18:23 kcsandnp01 kernel: console [tty0] enabled
Feb 16 00:18:23 kcsandnp01 kernel: allocated 33554432 bytes of page_cgroup
Feb 16 00:18:23 kcsandnp01 kernel: please try 'cgroup_disable=memory' option if you don't want memory cgroups
Feb 16 00:18:23 kcsandnp01 kernel: TSC freq read from hypervisor : 2266.747 MHz
Feb 16 00:18:23 kcsandnp01 kernel: Detected 2266.747 MHz processor.
Feb 16 00:18:23 kcsandnp01 kernel: Calibrating delay loop (skipped) preset value.. 4533.49 BogoMIPS (lpj=2266747)
Feb 16 00:18:23 kcsandnp01 kernel: pid_max: default: 65536 minimum: 512
Feb 16 00:18:23 kcsandnp01 kernel: Security Framework initialized
Feb 16 00:18:23 kcsandnp01 kernel: SELinux:  Initializing.
Feb 16 00:18:23 kcsandnp01 kernel: Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: Mount-cache hash table entries: 256
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys ns
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys cpuacct
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys memory
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys devices
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys freezer
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys net_cls
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys blkio
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys perf_event
Feb 16 00:18:23 kcsandnp01 kernel: Initializing cgroup subsys net_prio
Feb 16 00:18:23 kcsandnp01 kernel: Disabled fast string operations
Feb 16 00:18:23 kcsandnp01 kernel: CPU: Physical Processor ID: 0
Feb 16 00:18:23 kcsandnp01 kernel: CPU: Processor Core ID: 0
Feb 16 00:18:23 kcsandnp01 kernel: mce: CPU supports 0 MCE banks
Feb 16 00:18:23 kcsandnp01 kernel: alternatives: switching to unfair spinlock
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Core revision 20090903
Feb 16 00:18:23 kcsandnp01 kernel: ftrace: converting mcount calls to 0f 1f 44 00 00
Feb 16 00:18:23 kcsandnp01 kernel: ftrace: allocating 22127 entries in 87 pages
Feb 16 00:18:23 kcsandnp01 kernel: x2apic not enabled, IRQ remapping init failed
Feb 16 00:18:23 kcsandnp01 kernel: APIC routing finalized to physical flat.
Feb 16 00:18:23 kcsandnp01 kernel: ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
Feb 16 00:18:23 kcsandnp01 kernel: CPU0: Intel(R) Xeon(R) CPU E7- 4860  @ 2.27GHz stepping 01
Feb 16 00:18:23 kcsandnp01 kernel: Performance Events: 16-deep LBR, Westmere events, core PMU driver.
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'cpu cycles' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'instructions' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'bus cycles' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'cache references' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'cache misses' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'branch instructions' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: CPUID marked event: 'branch misses' unavailable
Feb 16 00:18:23 kcsandnp01 kernel: ... version:                1
Feb 16 00:18:23 kcsandnp01 kernel: ... bit width:              48
Feb 16 00:18:23 kcsandnp01 kernel: ... generic registers:      4
Feb 16 00:18:23 kcsandnp01 kernel: ... value mask:             0000ffffffffffff
Feb 16 00:18:23 kcsandnp01 kernel: ... max period:             000000007fffffff
Feb 16 00:18:23 kcsandnp01 kernel: ... fixed-purpose events:   0
Feb 16 00:18:23 kcsandnp01 kernel: ... event mask:             000000000000000f
Feb 16 00:18:23 kcsandnp01 kernel: NMI watchdog disabled (cpu0): hardware events not enabled
Feb 16 00:18:23 kcsandnp01 kernel: Booting Node   0, Processors  #1
Feb 16 00:18:23 kcsandnp01 kernel: Disabled fast string operations
Feb 16 00:18:23 kcsandnp01 kernel: mce: CPU supports 0 MCE banks
Feb 16 00:18:23 kcsandnp01 kernel: Brought up 2 CPUs
Feb 16 00:18:23 kcsandnp01 kernel: Total of 2 processors activated (9066.98 BogoMIPS).
Feb 16 00:18:23 kcsandnp01 kernel: x86 PAT enabled: cpu 0, old 0x0, new 0x7010600070106
Feb 16 00:18:23 kcsandnp01 kernel: x86 PAT enabled: cpu 1, old 0x0, new 0x7010600070106
Feb 16 00:18:23 kcsandnp01 kernel: devtmpfs: initialized
Feb 16 00:18:23 kcsandnp01 kernel: PM: Registering ACPI NVS region at bfeff000 (4096 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: regulator: core version 0.5
Feb 16 00:18:23 kcsandnp01 kernel: NET: Registered protocol family 16
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: bus type pci registered
Feb 16 00:18:23 kcsandnp01 kernel: PCI: MCFG configuration 0: base f0000000 segment 0 buses 0 - 127
Feb 16 00:18:23 kcsandnp01 kernel: PCI: MCFG area at f0000000 reserved in E820
Feb 16 00:18:23 kcsandnp01 kernel: PCI: Using MMCONFIG at f0000000 - f7ffffff
Feb 16 00:18:23 kcsandnp01 kernel: PCI: Using configuration type 1 for base access
Feb 16 00:18:23 kcsandnp01 kernel: bio: create slab <bio-0> at 0
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: BIOS _OSI(Linux) query ignored
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Interpreter enabled
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: (supports S0 S1 S4 S5)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Using IOAPIC for interrupt routing
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: No dock devices found.
Feb 16 00:18:23 kcsandnp01 kernel: PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-7f])
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [mem 0x000a0000-0x000bffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [mem 0x000d0000-0x000d3fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [mem 0x000d4000-0x000d7fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [mem 0x000d8000-0x000dbfff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [mem 0xc0000000-0xfebfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [io  0x0000-0x0cf7]
Feb 16 00:18:23 kcsandnp01 kernel: pci_root PNP0A03:00: host bridge window [io  0x0d00-0xfeff]
Feb 16 00:18:23 kcsandnp01 kernel: PCI host bridge to bus 0000:00
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [mem 0x000d0000-0x000d3fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [mem 0x000d4000-0x000d7fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [mem 0x000d8000-0x000dbfff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7]
Feb 16 00:18:23 kcsandnp01 kernel: pci_bus 0000:00: root bus resource [io  0x0d00-0xfeff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:07.3: quirk: [io  0x1000-0x103f] claimed by PIIX4 ACPI
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:07.3: quirk: [io  0x1040-0x104f] claimed by PIIX4 SMB
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:11.0: PCI bridge to [bus 02-02] (subtractive decode)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:03:00.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:03:00.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:03:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0: PCI bridge to [bus 03-03]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1: PCI bridge to [bus 04-04]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2: PCI bridge to [bus 05-05]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3: PCI bridge to [bus 06-06]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4: PCI bridge to [bus 07-07]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5: PCI bridge to [bus 08-08]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6: PCI bridge to [bus 09-09]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7: PCI bridge to [bus 0a-0a]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:0b:00.0: PME# supported from D0 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:0b:00.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:0b:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0: PCI bridge to [bus 0b-0b]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1: PCI bridge to [bus 0c-0c]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2: PCI bridge to [bus 0d-0d]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3: PCI bridge to [bus 0e-0e]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4: PCI bridge to [bus 0f-0f]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5: PCI bridge to [bus 10-10]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6: PCI bridge to [bus 11-11]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7: PCI bridge to [bus 12-12]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:13:00.0: PME# supported from D0 D1 D2 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:13:00.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:13:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0: PCI bridge to [bus 13-13]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1: PCI bridge to [bus 14-14]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2: PCI bridge to [bus 15-15]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3: PCI bridge to [bus 16-16]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4: PCI bridge to [bus 17-17]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5: PCI bridge to [bus 18-18]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6: PCI bridge to [bus 19-19]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7: PCI bridge to [bus 1a-1a]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:1b:00.0: PME# supported from D0 D1 D2 D3hot D3cold
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:1b:00.0: PME# disabled
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:1b:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0: PCI bridge to [bus 1b-1b]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1: PCI bridge to [bus 1c-1c]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2: PCI bridge to [bus 1d-1d]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3: PCI bridge to [bus 1e-1e]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4: PCI bridge to [bus 1f-1f]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5: PCI bridge to [bus 20-20]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6: PCI bridge to [bus 21-21]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7: PCI bridge to [bus 22-22]
Feb 16 00:18:23 kcsandnp01 kernel: pci0000:00: Requesting ACPI _OSC control (0x1d)
Feb 16 00:18:23 kcsandnp01 kernel: pci0000:00: ACPI _OSC control (0x15) granted
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 *9 10 11 14 15)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 *5 6 7 9 10 11 14 15)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11 14 15)
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 14 15)
Feb 16 00:18:23 kcsandnp01 kernel: vgaarb: device added: PCI:0000:00:0f.0,decodes=io+mem,owns=io+mem,locks=none
Feb 16 00:18:23 kcsandnp01 kernel: vgaarb: loaded
Feb 16 00:18:23 kcsandnp01 kernel: vgaarb: bridge control possible 0000:00:0f.0
Feb 16 00:18:23 kcsandnp01 kernel: SCSI subsystem initialized
Feb 16 00:18:23 kcsandnp01 kernel: usbcore: registered new interface driver usbfs
Feb 16 00:18:23 kcsandnp01 kernel: usbcore: registered new interface driver hub
Feb 16 00:18:23 kcsandnp01 kernel: usbcore: registered new device driver usb
Feb 16 00:18:23 kcsandnp01 kernel: PCI: Using ACPI for IRQ routing
Feb 16 00:18:23 kcsandnp01 kernel: NetLabel: Initializing
Feb 16 00:18:23 kcsandnp01 kernel: NetLabel:  domain hash size = 128
Feb 16 00:18:23 kcsandnp01 kernel: NetLabel:  protocols = UNLABELED CIPSOv4
Feb 16 00:18:23 kcsandnp01 kernel: NetLabel:  unlabeled traffic allowed by default
Feb 16 00:18:23 kcsandnp01 kernel: HPET: 16 timers in total, 0 timers will be used for per-cpu timer
Feb 16 00:18:23 kcsandnp01 kernel: hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
Feb 16 00:18:23 kcsandnp01 kernel: hpet0: 16 comparators, 64-bit 14.318180 MHz counter
Feb 16 00:18:23 kcsandnp01 kernel: Switching to clocksource hpet
Feb 16 00:18:23 kcsandnp01 kernel: pnp: PnP ACPI init
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: bus type pnp registered
Feb 16 00:18:23 kcsandnp01 kernel: pnp: PnP ACPI: found 16 devices
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: ACPI bus type pnp unregistered
Feb 16 00:18:23 kcsandnp01 kernel: system 00:01: [io  0x1000-0x103f] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:01: [io  0x1040-0x104f] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:01: [io  0x0cf0-0x0cf1] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:08: [mem 0xfed00000-0xfed003ff] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:0d: [io  0xfce0-0xfcff] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:0d: [mem 0xf0000000-0xf7ffffff] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: system 00:0d: [mem 0xfe800000-0xfe9fffff] has been reserved
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:0f.0: BAR 6: assigned [mem 0xc0000000-0xc0007fff pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7: BAR 13: can't assign io (size 0x1000)
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:01.0: PCI bridge to [bus 01-01]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:11.0: PCI bridge to [bus 02-02]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:11.0:   bridge window [io  0x2000-0x3fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:11.0:   bridge window [mem 0xfd600000-0xfdffffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:11.0:   bridge window [mem 0xebb00000-0xebffffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:03:00.0: BAR 6: assigned [mem 0xeba00000-0xeba03fff pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0: PCI bridge to [bus 03-03]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0:   bridge window [io  0x4000-0x4fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0:   bridge window [mem 0xfd500000-0xfd5fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.0:   bridge window [mem 0xeba00000-0xebafffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1: PCI bridge to [bus 04-04]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1:   bridge window [io  0x8000-0x8fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1:   bridge window [mem 0xfd100000-0xfd1fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.1:   bridge window [mem 0xeb600000-0xeb6fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2: PCI bridge to [bus 05-05]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2:   bridge window [io  0xc000-0xcfff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2:   bridge window [mem 0xfcd00000-0xfcdfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.2:   bridge window [mem 0xeb200000-0xeb2fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3: PCI bridge to [bus 06-06]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3:   bridge window [mem 0xfc900000-0xfc9fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.3:   bridge window [mem 0xeae00000-0xeaefffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4: PCI bridge to [bus 07-07]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4:   bridge window [mem 0xfc500000-0xfc5fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.4:   bridge window [mem 0xeaa00000-0xeaafffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5: PCI bridge to [bus 08-08]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5:   bridge window [mem 0xfc100000-0xfc1fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.5:   bridge window [mem 0xea600000-0xea6fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6: PCI bridge to [bus 09-09]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6:   bridge window [mem 0xfbd00000-0xfbdfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.6:   bridge window [mem 0xea200000-0xea2fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7: PCI bridge to [bus 0a-0a]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7:   bridge window [mem 0xfb900000-0xfb9fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:15.7:   bridge window [mem 0xe9e00000-0xe9efffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:0b:00.0: BAR 6: assigned [mem 0xeb900000-0xeb903fff pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0: PCI bridge to [bus 0b-0b]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0:   bridge window [io  0x5000-0x5fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0:   bridge window [mem 0xfd400000-0xfd4fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.0:   bridge window [mem 0xeb900000-0xeb9fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1: PCI bridge to [bus 0c-0c]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1:   bridge window [io  0x9000-0x9fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1:   bridge window [mem 0xfd000000-0xfd0fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.1:   bridge window [mem 0xeb500000-0xeb5fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2: PCI bridge to [bus 0d-0d]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2:   bridge window [io  0xd000-0xdfff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2:   bridge window [mem 0xfcc00000-0xfccfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.2:   bridge window [mem 0xeb100000-0xeb1fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3: PCI bridge to [bus 0e-0e]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3:   bridge window [mem 0xfc800000-0xfc8fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.3:   bridge window [mem 0xead00000-0xeadfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4: PCI bridge to [bus 0f-0f]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4:   bridge window [mem 0xfc400000-0xfc4fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.4:   bridge window [mem 0xea900000-0xea9fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5: PCI bridge to [bus 10-10]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5:   bridge window [mem 0xfc000000-0xfc0fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.5:   bridge window [mem 0xea500000-0xea5fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6: PCI bridge to [bus 11-11]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6:   bridge window [mem 0xfbc00000-0xfbcfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.6:   bridge window [mem 0xea100000-0xea1fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7: PCI bridge to [bus 12-12]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7:   bridge window [mem 0xfb800000-0xfb8fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:16.7:   bridge window [mem 0xe9d00000-0xe9dfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:13:00.0: BAR 6: assigned [mem 0xeb800000-0xeb80ffff pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0: PCI bridge to [bus 13-13]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0:   bridge window [io  0x6000-0x6fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0:   bridge window [mem 0xfd300000-0xfd3fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.0:   bridge window [mem 0xeb800000-0xeb8fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1: PCI bridge to [bus 14-14]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1:   bridge window [io  0xa000-0xafff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1:   bridge window [mem 0xfcf00000-0xfcffffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.1:   bridge window [mem 0xeb400000-0xeb4fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2: PCI bridge to [bus 15-15]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2:   bridge window [io  0xe000-0xefff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2:   bridge window [mem 0xfcb00000-0xfcbfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.2:   bridge window [mem 0xeb000000-0xeb0fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3: PCI bridge to [bus 16-16]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3:   bridge window [mem 0xfc700000-0xfc7fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.3:   bridge window [mem 0xeac00000-0xeacfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4: PCI bridge to [bus 17-17]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4:   bridge window [mem 0xfc300000-0xfc3fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.4:   bridge window [mem 0xea800000-0xea8fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5: PCI bridge to [bus 18-18]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5:   bridge window [mem 0xfbf00000-0xfbffffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.5:   bridge window [mem 0xea400000-0xea4fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6: PCI bridge to [bus 19-19]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6:   bridge window [mem 0xfbb00000-0xfbbfffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.6:   bridge window [mem 0xea000000-0xea0fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7: PCI bridge to [bus 1a-1a]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7:   bridge window [mem 0xfb700000-0xfb7fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:17.7:   bridge window [mem 0xe9c00000-0xe9cfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:1b:00.0: BAR 6: assigned [mem 0xeb700000-0xeb70ffff pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0: PCI bridge to [bus 1b-1b]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0:   bridge window [io  0x7000-0x7fff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0:   bridge window [mem 0xfd200000-0xfd2fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.0:   bridge window [mem 0xeb700000-0xeb7fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1: PCI bridge to [bus 1c-1c]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1:   bridge window [io  0xb000-0xbfff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1:   bridge window [mem 0xfce00000-0xfcefffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.1:   bridge window [mem 0xeb300000-0xeb3fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2: PCI bridge to [bus 1d-1d]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2:   bridge window [mem 0xfca00000-0xfcafffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.2:   bridge window [mem 0xeaf00000-0xeaffffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3: PCI bridge to [bus 1e-1e]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3:   bridge window [mem 0xfc600000-0xfc6fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.3:   bridge window [mem 0xeab00000-0xeabfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4: PCI bridge to [bus 1f-1f]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4:   bridge window [mem 0xfc200000-0xfc2fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.4:   bridge window [mem 0xea700000-0xea7fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5: PCI bridge to [bus 20-20]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5:   bridge window [mem 0xfbe00000-0xfbefffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.5:   bridge window [mem 0xea300000-0xea3fffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6: PCI bridge to [bus 21-21]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6:   bridge window [mem 0xfba00000-0xfbafffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.6:   bridge window [mem 0xe9f00000-0xe9ffffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7: PCI bridge to [bus 22-22]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7:   bridge window [mem 0xfb600000-0xfb6fffff]
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:18.7:   bridge window [mem 0xe9b00000-0xe9bfffff 64bit pref]
Feb 16 00:18:23 kcsandnp01 kernel: NET: Registered protocol family 2
Feb 16 00:18:23 kcsandnp01 kernel: IP route cache hash table entries: 262144 (order: 9, 2097152 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: TCP: Hash tables configured (established 524288 bind 65536)
Feb 16 00:18:23 kcsandnp01 kernel: TCP reno registered
Feb 16 00:18:23 kcsandnp01 kernel: NET: Registered protocol family 1
Feb 16 00:18:23 kcsandnp01 kernel: pci 0000:00:00.0: Limiting direct PCI/PCI transfers
Feb 16 00:18:23 kcsandnp01 kernel: Trying to unpack rootfs image as initramfs...
Feb 16 00:18:23 kcsandnp01 kernel: Freeing initrd memory: 27439k freed
Feb 16 00:18:23 kcsandnp01 kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Feb 16 00:18:23 kcsandnp01 kernel: Placing 64MB software IO TLB between ffff880020000000 - ffff880024000000
Feb 16 00:18:23 kcsandnp01 kernel: software IO TLB at phys 0x20000000 - 0x24000000
Feb 16 00:18:23 kcsandnp01 kernel: Simple Boot Flag at 0x36 set to 0x80
Feb 16 00:18:23 kcsandnp01 kernel: futex hash table entries: 16384 (order: 8, 1048576 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: audit: initializing netlink socket (disabled)
Feb 16 00:18:23 kcsandnp01 kernel: type=2000 audit(1487204274.238:1): initialized
Feb 16 00:18:23 kcsandnp01 kernel: HugeTLB registered 2 MB page size, pre-allocated 0 pages
Feb 16 00:18:23 kcsandnp01 kernel: VFS: Disk quotas dquot_6.5.2
Feb 16 00:18:23 kcsandnp01 kernel: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
Feb 16 00:18:23 kcsandnp01 kernel: msgmni has been set to 15980
Feb 16 00:18:23 kcsandnp01 kernel: ksign: Installing public key data
Feb 16 00:18:23 kcsandnp01 kernel: Loading keyring
Feb 16 00:18:23 kcsandnp01 kernel: - Added public key F26C4F8C5052F303
Feb 16 00:18:23 kcsandnp01 kernel: - User ID: Red Hat, Inc. (Kernel Module GPG key)
Feb 16 00:18:23 kcsandnp01 kernel: - Added public key D4A26C9CCD09BEDA
Feb 16 00:18:23 kcsandnp01 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program <secalert@redhat.com>
Feb 16 00:18:23 kcsandnp01 kernel: - Added public key CB445E0DA3EA1F65
Feb 16 00:18:23 kcsandnp01 kernel: - User ID: Red Hat Enterprise Linux Driver Update Program (key 4) <secalert@redhat.com>
Feb 16 00:18:23 kcsandnp01 kernel: Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
Feb 16 00:18:23 kcsandnp01 kernel: io scheduler noop registered
Feb 16 00:18:23 kcsandnp01 kernel: io scheduler anticipatory registered
Feb 16 00:18:23 kcsandnp01 kernel: io scheduler deadline registered
Feb 16 00:18:23 kcsandnp01 kernel: io scheduler cfq registered (default)
Feb 16 00:18:23 kcsandnp01 kernel: pci_hotplug: PCI Hot Plug PCI Core version: 0.5
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:15.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:16.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:17.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.0:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.1:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.2:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.3:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.4:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.5:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.6:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp 0000:00:18.7:pcie04: HPC vendor_id 15ad device_id 7a0 ss_vid 0 ss_did 0
Feb 16 00:18:23 kcsandnp01 kernel: pciehp: PCI Express Hot Plug Controller Driver version: 0.4
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [32] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [33] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [34] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [35] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [36] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [37] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [38] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [39] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [40] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [41] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [42] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [43] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [44] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [45] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [46] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [47] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [48] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [49] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [50] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [51] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [52] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [53] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [54] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [55] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [56] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [57] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [58] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [59] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [60] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [61] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [62] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp: Slot [63] registered
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 160 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 192 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 224 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 256 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 161 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 162 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 163 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 164 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 165 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 166 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 167 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 193 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 194 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 195 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 196 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 197 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 198 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 199 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 225 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 226 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 227 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 228 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 229 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 230 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 231 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 257 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 258 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 259 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 260 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 261 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 262 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: acpiphp_glue: Slot 263 already registered by another hotplug driver
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: AC Adapter [ACAD] (on-line)
Feb 16 00:18:23 kcsandnp01 kernel: input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
Feb 16 00:18:23 kcsandnp01 kernel: ACPI: Power Button [PWRF]
Feb 16 00:18:23 kcsandnp01 kernel: Refined TSC clocksource calibration: 2266.737 MHz.
Feb 16 00:18:23 kcsandnp01 kernel: Switching to clocksource tsc
Feb 16 00:18:23 kcsandnp01 kernel: [Firmware Bug]: No valid trip found
Feb 16 00:18:23 kcsandnp01 kernel: System RAM resource 10000000 - bfffffff cannot be added
Feb 16 00:18:23 kcsandnp01 kernel: memory (0x10000 - 0x18000) online failed.
Feb 16 00:18:23 kcsandnp01 kernel: ACPI:memory_hp:Memory online failed for 0x10000000 - 0xc0000000
Feb 16 00:18:23 kcsandnp01 kernel: System RAM resource 100000000 - 23fffffff cannot be added
Feb 16 00:18:23 kcsandnp01 kernel: memory (0x100000 - 0x108000) online failed.
Feb 16 00:18:23 kcsandnp01 kernel: ACPI:memory_hp:Memory online failed for 0x100000000 - 0x240000000
Feb 16 00:18:23 kcsandnp01 kernel: GHES: HEST is not enabled!
Feb 16 00:18:23 kcsandnp01 kernel: Non-volatile memory driver v1.3
Feb 16 00:18:23 kcsandnp01 kernel: Linux agpgart interface v0.103
Feb 16 00:18:23 kcsandnp01 kernel: agpgart-intel 0000:00:00.0: Intel 440BX Chipset
Feb 16 00:18:23 kcsandnp01 kernel: agpgart-intel 0000:00:00.0: AGP aperture is 256M @ 0x0
Feb 16 00:18:23 kcsandnp01 kernel: crash memory driver: version 1.1
Feb 16 00:18:23 kcsandnp01 kernel: Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
Feb 16 00:18:23 kcsandnp01 kernel: serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Feb 16 00:18:23 kcsandnp01 kernel: serial8250: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Feb 16 00:18:23 kcsandnp01 kernel: 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a 16550A
Feb 16 00:18:23 kcsandnp01 kernel: 00:0b: ttyS1 at I/O 0x2f8 (irq = 3) is a 16550A
Feb 16 00:18:23 kcsandnp01 kernel: brd: module loaded
Feb 16 00:18:23 kcsandnp01 kernel: loop: module loaded
Feb 16 00:18:23 kcsandnp01 kernel: input: Macintosh mouse button emulation as /devices/virtual/input/input1
Feb 16 00:18:23 kcsandnp01 kernel: Fixed MDIO Bus: probed
Feb 16 00:18:23 kcsandnp01 kernel: ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
Feb 16 00:18:23 kcsandnp01 kernel: ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
Feb 16 00:18:23 kcsandnp01 kernel: uhci_hcd: USB Universal Host Controller Interface driver
Feb 16 00:18:23 kcsandnp01 kernel: PNP: PS/2 Controller [PNP0303:KBC,PNP0f13:MOUS] at 0x60,0x64 irq 1,12
Feb 16 00:18:23 kcsandnp01 kernel: serio: i8042 KBD port at 0x60,0x64 irq 1
Feb 16 00:18:23 kcsandnp01 kernel: serio: i8042 AUX port at 0x60,0x64 irq 12
Feb 16 00:18:23 kcsandnp01 kernel: mice: PS/2 mouse device common for all mice
Feb 16 00:18:23 kcsandnp01 kernel: rtc_cmos 00:04: rtc core: registered rtc_cmos as rtc0
Feb 16 00:18:23 kcsandnp01 kernel: rtc0: alarms up to one month, y3k, 114 bytes nvram, hpet irqs
Feb 16 00:18:23 kcsandnp01 kernel: cpuidle: using governor ladder
Feb 16 00:18:23 kcsandnp01 kernel: cpuidle: using governor menu
Feb 16 00:18:23 kcsandnp01 kernel: EFI Variables Facility v0.08 2004-May-17
Feb 16 00:18:23 kcsandnp01 kernel: usbcore: registered new interface driver hiddev
Feb 16 00:18:23 kcsandnp01 kernel: usbcore: registered new interface driver usbhid
Feb 16 00:18:23 kcsandnp01 kernel: usbhid: v2.6:USB HID core driver
Feb 16 00:18:23 kcsandnp01 kernel: GRE over IPv4 demultiplexor driver
Feb 16 00:18:23 kcsandnp01 kernel: TCP cubic registered
Feb 16 00:18:23 kcsandnp01 kernel: Initializing XFRM netlink socket
Feb 16 00:18:23 kcsandnp01 kernel: NET: Registered protocol family 17
Feb 16 00:18:23 kcsandnp01 kernel: registered taskstats version 1
Feb 16 00:18:23 kcsandnp01 kernel: input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
Feb 16 00:18:23 kcsandnp01 kernel: input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input3
Feb 16 00:18:23 kcsandnp01 kernel: rtc_cmos 00:04: setting system clock to 2017-02-16 00:18:07 UTC (1487204287)
Feb 16 00:18:23 kcsandnp01 kernel: Initalizing network drop monitor service
Feb 16 00:18:23 kcsandnp01 kernel: Freeing unused kernel memory: 1296k freed
Feb 16 00:18:23 kcsandnp01 kernel: Write protecting the kernel read-only data: 10240k
Feb 16 00:18:23 kcsandnp01 kernel: Freeing unused kernel memory: 732k freed
Feb 16 00:18:23 kcsandnp01 kernel: Freeing unused kernel memory: 1576k freed
Feb 16 00:18:23 kcsandnp01 kernel: dracut: dracut-004-388.el6
Feb 16 00:18:23 kcsandnp01 kernel: dracut: rd_NO_LUKS: removing cryptoluks activation
Feb 16 00:18:23 kcsandnp01 kernel: device-mapper: uevent: version 1.0.3
Feb 16 00:18:23 kcsandnp01 kernel: device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com
Feb 16 00:18:23 kcsandnp01 kernel: udev: starting version 147
Feb 16 00:18:23 kcsandnp01 kernel: dracut: Starting plymouth daemon
Feb 16 00:18:23 kcsandnp01 kernel: dracut: rd_NO_DM: removing DM RAID activation
Feb 16 00:18:23 kcsandnp01 kernel: dracut: rd_NO_MD: removing MD RAID activation
Feb 16 00:18:23 kcsandnp01 kernel: Fusion MPT base driver 3.04.20
Feb 16 00:18:23 kcsandnp01 kernel: Copyright (c) 1999-2008 LSI Corporation
Feb 16 00:18:23 kcsandnp01 kernel: Fusion MPT SAS Host driver 3.04.20
Feb 16 00:18:23 kcsandnp01 kernel: mptsas 0000:03:00.0: PCI INT A -> GSI 18 (level, low) -> IRQ 18
Feb 16 00:18:23 kcsandnp01 kernel: mptbase: ioc0: Initiating bringup
Feb 16 00:18:23 kcsandnp01 kernel: ioc0: LSISAS1068 B0: Capabilities={Initiator}
Feb 16 00:18:23 kcsandnp01 kernel: scsi0 : ioc0: LSISAS1068 B0, FwRev=01032920h, Ports=1, MaxQ=128, IRQ=18
Feb 16 00:18:23 kcsandnp01 kernel: mptsas: ioc0: attaching ssp device: fw_channel 0, fw_id 0, phy 0, sas_addr 0x5000c29610a42b47
Feb 16 00:18:23 kcsandnp01 kernel: scsi 0:0:0:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0 ANSI: 2
Feb 16 00:18:23 kcsandnp01 kernel: mptsas 0000:0b:00.0: PCI INT A -> GSI 19 (level, low) -> IRQ 19
Feb 16 00:18:23 kcsandnp01 kernel: mptbase: ioc1: Initiating bringup
Feb 16 00:18:23 kcsandnp01 kernel: ioc1: LSISAS1068 B0: Capabilities={Initiator}
Feb 16 00:18:23 kcsandnp01 kernel: scsi1 : ioc1: LSISAS1068 B0, FwRev=01032920h, Ports=1, MaxQ=128, IRQ=19
Feb 16 00:18:23 kcsandnp01 kernel: mptsas: ioc1: attaching ssp device: fw_channel 0, fw_id 0, phy 0, sas_addr 0x5000c29dbb4a83bf
Feb 16 00:18:23 kcsandnp01 kernel: scsi 1:0:0:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0 ANSI: 2
Feb 16 00:18:23 kcsandnp01 kernel: mptsas: ioc1: attaching ssp device: fw_channel 0, fw_id 1, phy 1, sas_addr 0x5000c296002ee417
Feb 16 00:18:23 kcsandnp01 kernel: scsi 1:0:1:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0 ANSI: 2
Feb 16 00:18:23 kcsandnp01 kernel: scsi2 : ata_piix
Feb 16 00:18:23 kcsandnp01 kernel: scsi3 : ata_piix
Feb 16 00:18:23 kcsandnp01 kernel: ata1: PATA max UDMA/33 cmd 0x1f0 ctl 0x3f6 bmdma 0x1060 irq 14
Feb 16 00:18:23 kcsandnp01 kernel: ata2: PATA max UDMA/33 cmd 0x170 ctl 0x376 bmdma 0x1068 irq 15
Feb 16 00:18:23 kcsandnp01 kernel: ata2.00: ATAPI: VMware Virtual IDE CDROM Drive, 00000001, max UDMA/33
Feb 16 00:18:23 kcsandnp01 kernel: ata2.00: configured for UDMA/33
Feb 16 00:18:23 kcsandnp01 kernel: scsi 3:0:0:0: CD-ROM            NECVMWar VMware IDE CDR10 1.00 PQ: 0 ANSI: 5
Feb 16 00:18:23 kcsandnp01 kernel: VMware vmxnet3 virtual NIC driver - version 1.1.30.0-k-NAPI
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:13:00.0: # of Tx queues : 2, # of Rx queues : 2
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:13:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Feb 16 00:18:23 kcsandnp01 kernel: features: sg csum vlan jf tso tsoIPv6 lro highDMA
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:13:00.0: eth0: NIC Link is Up 10000 Mbps
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:1b:00.0: # of Tx queues : 2, # of Rx queues : 2
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:1b:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
Feb 16 00:18:23 kcsandnp01 kernel: features: sg csum vlan jf tso tsoIPv6 lro highDMA
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:1b:00.0: eth1: NIC Link is Up 10000 Mbps
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Write Protect is off
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sda: sda1 sda2
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Write Protect is off
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: [sda] Attached SCSI disk
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sdb: sdb1
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: [sdb] Attached SCSI disk
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Write Protect is off
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sdc: unknown partition table
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Cache data unavailable
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Assuming drive cache: write through
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: [sdc] Attached SCSI disk
Feb 16 00:18:23 kcsandnp01 kernel: sr0: scsi3-mmc drive: 1x/1x writer dvd-ram cd/rw xa/form2 cdda tray
Feb 16 00:18:23 kcsandnp01 kernel: Uniform CD-ROM driver Revision: 3.20
Feb 16 00:18:23 kcsandnp01 kernel: dracut: Scanning devices sda2 sdb1  for LVM logical volumes vg00/lvol2 vg00/lvol1 
Feb 16 00:18:23 kcsandnp01 kernel: dracut: inactive '/dev/vg_share/mqhalv' [50.00 GiB] inherit
Feb 16 00:18:23 kcsandnp01 kernel: dracut: inactive '/dev/vg00/lvol1' [20.00 GiB] inherit
Feb 16 00:18:23 kcsandnp01 kernel: dracut: inactive '/dev/vg00/lvol2' [4.00 GiB] inherit
Feb 16 00:18:23 kcsandnp01 kernel: dracut: inactive '/dev/vg00/lvol3' [10.00 GiB] inherit
Feb 16 00:18:23 kcsandnp01 kernel: dracut: inactive '/dev/vg00/lvol4' [10.00 GiB] inherit
Feb 16 00:18:23 kcsandnp01 kernel: EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:18:23 kcsandnp01 kernel: dracut: Mounted root filesystem /dev/mapper/vg00-lvol1
Feb 16 00:18:23 kcsandnp01 kernel: SELinux:  Disabled at runtime.
Feb 16 00:18:23 kcsandnp01 kernel: type=1404 audit(1487204289.431:2): selinux=0 auid=4294967295 ses=4294967295
Feb 16 00:18:23 kcsandnp01 kernel: dracut: 
Feb 16 00:18:23 kcsandnp01 kernel: dracut: Switching root
Feb 16 00:18:23 kcsandnp01 kernel: udev: starting version 147
Feb 16 00:18:23 kcsandnp01 kernel: shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
Feb 16 00:18:23 kcsandnp01 kernel: piix4_smbus 0000:00:07.3: Host SMBus controller not enabled!
Feb 16 00:18:23 kcsandnp01 kernel: vmci: applying 16k kernel stack fix up
Feb 16 00:18:23 kcsandnp01 kernel: Probing for vmci/PCI.
Feb 16 00:18:23 kcsandnp01 kernel: vmci 0000:00:07.7: PCI INT A -> GSI 16 (level, low) -> IRQ 16
Feb 16 00:18:23 kcsandnp01 kernel: Found vmci/PCI at 0x1080, irq 16.
Feb 16 00:18:23 kcsandnp01 kernel: VMCI: using capabilities 0xc.
Feb 16 00:18:23 kcsandnp01 kernel: Registered vmci device.
Feb 16 00:18:23 kcsandnp01 kernel: microcode: CPU0 sig=0x20651, pf=0x1, revision=0x37
Feb 16 00:18:23 kcsandnp01 kernel: platform microcode: firmware: requesting intel-ucode/06-25-01
Feb 16 00:18:23 kcsandnp01 kernel: microcode: CPU1 sig=0x20651, pf=0x1, revision=0x37
Feb 16 00:18:23 kcsandnp01 kernel: platform microcode: firmware: requesting intel-ucode/06-25-01
Feb 16 00:18:23 kcsandnp01 kernel: Microcode Update Driver: v2.00 <tigran@aivazian.fsnet.co.uk>, Peter Oruba
Feb 16 00:18:23 kcsandnp01 kernel: sd 0:0:0:0: Attached scsi generic sg0 type 0
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:0:0: Attached scsi generic sg1 type 0
Feb 16 00:18:23 kcsandnp01 kernel: sd 1:0:1:0: Attached scsi generic sg2 type 0
Feb 16 00:18:23 kcsandnp01 kernel: sr 3:0:0:0: Attached scsi generic sg3 type 5
Feb 16 00:18:23 kcsandnp01 kernel: parport_pc 00:09: reported by Plug and Play ACPI
Feb 16 00:18:23 kcsandnp01 kernel: parport0: PC-style at 0x378, irq 7 [PCSPP,TRISTATE]
Feb 16 00:18:23 kcsandnp01 kernel: ppdev: user-space parallel port driver
Feb 16 00:18:23 kcsandnp01 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:18:23 kcsandnp01 kernel: EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:18:23 kcsandnp01 kernel: EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:18:23 kcsandnp01 kernel: Adding 4194300k swap on /dev/mapper/vg00-lvol2.  Priority:-1 extents:1 across:4194300k 
Feb 16 00:18:23 kcsandnp01 kernel: NET: Registered protocol family 10
Feb 16 00:18:23 kcsandnp01 kernel: lo: Disabled Privacy Extensions
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:13:00.0: eth0: intr type 3, mode 0, 3 vectors allocated
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:13:00.0: eth0: NIC Link is Up 10000 Mbps
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:1b:00.0: eth1: intr type 3, mode 0, 3 vectors allocated
Feb 16 00:18:23 kcsandnp01 kernel: vmxnet3 0000:1b:00.0: eth1: NIC Link is Up 10000 Mbps
Feb 16 00:18:23 kcsandnp01 kernel: type=1305 audit(1487179102.973:3): audit_pid=2190 old=0 auid=4294967295 ses=4294967295 res=1
Feb 16 00:18:23 kcsandnp01 acpid: starting up
Feb 16 00:18:23 kcsandnp01 acpid: 1 rule loaded
Feb 16 00:18:23 kcsandnp01 acpid: waiting for events: event logging is off
Feb 16 00:18:24 kcsandnp01 acpid: client connected from 2358[68:68]
Feb 16 00:18:24 kcsandnp01 acpid: 1 client rule loaded
Feb 16 00:18:24 kcsandnp01 xinetd[2406]: xinetd Version 2.3.14 started with libwrap loadavg labeled-networking options compiled in.
Feb 16 00:18:24 kcsandnp01 xinetd[2406]: Started working: 1 available service
Feb 16 00:18:24 kcsandnp01 ntpd[2416]: ntpd 4.2.6p5@1.2349-o Tue Apr 28 10:15:27 UTC 2015 (1)
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: proto: precision = 0.065 usec
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen and drop on 1 v6wildcard :: UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 2 lo 127.0.0.1 UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 3 eth0 192.168.141.11 UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 4 eth1 10.1.204.8 UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 5 lo ::1 UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 6 eth1 fe80::250:56ff:feb1:239b UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listen normally on 7 eth0 fe80::250:56ff:feb1:359f UDP 123
Feb 16 00:18:24 kcsandnp01 ntpd[2417]: Listening on routing socket on fd #24 for interface updates
Feb 16 00:18:25 kcsandnp01 abrtd: Init complete, entering main loop
Feb 16 00:18:39 kcsandnp01 kernel: SCTP: Hash tables configured (established 65536 bind 65536)
Feb 16 00:18:39 kcsandnp01 kernel: DLM (built Jul  1 2015 18:24:22) installed
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [MAIN  ] Corosync Cluster Engine ('1.4.7'): started and ready to provide service.
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [MAIN  ] Corosync built-in features: nss dbus rdma snmp
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [MAIN  ] Successfully read config from /etc/cluster/cluster.conf
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [MAIN  ] Successfully parsed cman config
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [TOTEM ] Initializing transport (UDP/IP Unicast).
Feb 16 00:18:39 kcsandnp01 corosync[3172]:   [TOTEM ] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [TOTEM ] The network interface [10.1.204.8] is now up.
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [QUORUM] Using quorum provider quorum_cman
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [CMAN  ] CMAN 3.0.12.1 (built Mar  5 2015 07:51:44) started
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync CMAN membership service 2.90
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: openais checkpoint service B.01.01
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync extended virtual synchrony service
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync configuration service
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync cluster closed process group service v1.01
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync cluster config database access v1.01
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync profile loading service
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [QUORUM] Using quorum provider quorum_cman
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [SERV  ] Service engine loaded: corosync cluster quorum service v0.1
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [MAIN  ] Compatibility mode set to whitetank.  Using V1 and V2 of the synchronization engine.
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [TOTEM ] adding new UDPU member {10.1.204.8}
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [TOTEM ] adding new UDPU member {10.1.204.9}
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [CMAN  ] quorum regained, resuming activity
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [QUORUM] This node is within the primary component and will provide service.
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [QUORUM] Members[1]: 1
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [QUORUM] Members[1]: 1
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [CPG   ] chosen downlist: sender r(0) ip(10.1.204.8) ; members(old:0 left:0)
Feb 16 00:18:40 kcsandnp01 corosync[3172]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb 16 00:18:43 kcsandnp01 fenced[3239]: fenced 3.0.12.1 started
Feb 16 00:18:44 kcsandnp01 dlm_controld[3255]: dlm_controld 3.0.12.1 started
Feb 16 00:18:45 kcsandnp01 gfs_controld[3316]: gfs_controld 3.0.12.1 started
Feb 16 00:19:30 kcsandnp01 pacemaker: Starting Pacemaker Cluster Manager
Feb 16 00:19:30 kcsandnp01 pacemakerd[3413]:   notice: crm_add_logfile: Additional logging available in /var/log/pacemaker.log
Feb 16 00:19:30 kcsandnp01 pacemakerd[3413]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 pacemakerd[3413]:   notice: main: Starting Pacemaker 1.1.11 (Build: 97629de):  generated-manpages agent-manpages ascii-docs ncurses libqb-logging libqb-ipc nagios  corosync-plugin cman acls
Feb 16 00:19:30 kcsandnp01 cib[3419]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 attrd[3422]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 attrd[3422]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
Feb 16 00:19:30 kcsandnp01 attrd[3422]:   notice: main: Starting mainloop...
Feb 16 00:19:30 kcsandnp01 lrmd[3421]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 stonith-ng[3420]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 stonith-ng[3420]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
Feb 16 00:19:30 kcsandnp01 pengine[3423]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 cib[3419]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
Feb 16 00:19:30 kcsandnp01 crmd[3425]:   notice: crm_add_logfile: Additional logging available in /var/log/cluster/corosync.log
Feb 16 00:19:30 kcsandnp01 crmd[3425]:   notice: main: CRM Git Version: 97629de
Feb 16 00:19:30 kcsandnp01 crmd[3425]:   notice: crm_cluster_connect: Connecting to cluster infrastructure: cman
Feb 16 00:19:30 kcsandnp01 crmd[3425]:   notice: cman_event_callback: Membership 420: quorum acquired
Feb 16 00:19:30 kcsandnp01 crmd[3425]:   notice: crm_update_peer_state: cman_event_callback: Node kcsandnp01[1] - state is now member (was (null))
Feb 16 00:19:31 kcsandnp01 crmd[3425]:   notice: do_started: The local CRM is operational
Feb 16 00:19:31 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_started ]
Feb 16 00:19:31 kcsandnp01 stonith-ng[3420]:   notice: setup_cib: Watching for stonith topology changes
Feb 16 00:19:33 kcsandnp01 stonith-ng[3420]:   notice: stonith_device_register: Added 'vmfence' to the device list (1 active devices)
Feb 16 00:19:36 kcsandnp01 fenced[3239]: fencing node kcsandnp02
Feb 16 00:19:36 kcsandnp01 fence_pcmk[3500]: Requesting Pacemaker fence kcsandnp02 (reset)
Feb 16 00:19:36 kcsandnp01 stonith_admin[3501]:   notice: crm_log_args: Invoked: stonith_admin --reboot kcsandnp02 --tolerance 5s --tag cman 
Feb 16 00:19:36 kcsandnp01 stonith-ng[3420]:   notice: handle_request: Client stonith_admin.cman.3501.9d2c2d50 wants to fence (reboot) 'kcsandnp02' with device '(any)'
Feb 16 00:19:36 kcsandnp01 stonith-ng[3420]:   notice: initiate_remote_stonith_op: Initiating remote operation reboot for kcsandnp02: f52f4439-0222-4aa7-be96-f05e389e73c6 (0)
Feb 16 00:19:36 kcsandnp01 stonith-ng[3420]:   notice: can_fence_host_with_device: vmfence can fence (reboot) kcsandnp02: static-list
Feb 16 00:19:36 kcsandnp01 stonith-ng[3420]:   notice: can_fence_host_with_device: vmfence can fence (reboot) kcsandnp02: static-list
Feb 16 00:19:39 kcsandnp01 polkitd[3642]: started daemon version 0.96 using authority implementation `local' version `0.96'
Feb 16 00:19:39 kcsandnp01 rtkit-daemon[3649]: Sucessfully made thread 3647 of process 3647 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Feb 16 00:19:52 kcsandnp01 crmd[3425]:  warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Feb 16 00:19:52 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped ]
Feb 16 00:19:52 kcsandnp01 crmd[3425]:  warning: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_INTEGRATION
Feb 16 00:19:52 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
Feb 16 00:19:52 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 3: shutdown=0
Feb 16 00:19:52 kcsandnp01 attrd[3422]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Feb 16 00:19:52 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
Feb 16 00:19:52 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 5: shutdown=0
Feb 16 00:19:52 kcsandnp01 pengine[3423]:  warning: stage6: Scheduling Node kcsandnp02 for STONITH
Feb 16 00:19:52 kcsandnp01 pengine[3423]:   notice: LogActions: Start   vmfence#011(kcsandnp01)
Feb 16 00:19:52 kcsandnp01 pengine[3423]:   notice: LogActions: Start   ping_gw:0#011(kcsandnp01)
Feb 16 00:19:52 kcsandnp01 pengine[3423]:  warning: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-warn-8.bz2
Feb 16 00:19:52 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 4: monitor vmfence_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:52 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 5: monitor ha_lvm_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:52 kcsandnp01 crmd[3425]:   notice: te_fence_node: Executing reboot fencing operation (24) on kcsandnp02 (timeout=180000)
Feb 16 00:19:52 kcsandnp01 stonith-ng[3420]:   notice: handle_request: Client crmd.3425.a031dcf7 wants to fence (reboot) 'kcsandnp02' with device '(any)'
Feb 16 00:19:52 kcsandnp01 stonith-ng[3420]:   notice: merge_duplicates: Merging stonith action reboot for node kcsandnp02 originating from client crmd.3425.15dca67a with identical request from stonith_admin.cman.3501@kcsandnp01.f52f4439 (144s)
Feb 16 00:19:52 kcsandnp01 LVM(ha_lvm)[3661]: WARNING: LVM Volume vg_share is not available (stopped)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vmfence_monitor_0: not running (node=kcsandnp01, call=5, rc=7, cib-update=29, confirmed=true)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_monitor_0: not running (node=kcsandnp01, call=9, rc=7, cib-update=30, confirmed=true)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 6: monitor mqha_fs_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 7: monitor vip_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:53 kcsandnp01 Filesystem(mqha_fs)[3682]: WARNING: Couldn't find device [/dev/vg_share/mqhalv]. Expected /dev/??? to exist
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_monitor_0: not running (node=kcsandnp01, call=17, rc=7, cib-update=31, confirmed=true)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_monitor_0: not running (node=kcsandnp01, call=13, rc=7, cib-update=32, confirmed=true)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 8: monitor wmq_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 9: monitor iib_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_monitor_0: not running (node=kcsandnp01, call=21, rc=7, cib-update=33, confirmed=true)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_0:21 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Status not available\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 10: monitor ping_gw:0_monitor_0 on kcsandnp01 (local)
Feb 16 00:19:53 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ping_gw_monitor_0: not running (node=kcsandnp01, call=30, rc=7, cib-update=34, confirmed=true)
Feb 16 00:19:54 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_monitor_0: not running (node=kcsandnp01, call=25, rc=7, cib-update=35, confirmed=true)
Feb 16 00:19:54 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-iib_monitor_0:25 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 16 00:19:54 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 3: probe_complete probe_complete-kcsandnp01 on kcsandnp01 (local) - no waiting
Feb 16 00:19:54 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb 16 00:19:54 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 7: probe_complete=true
Feb 16 00:19:54 kcsandnp01 crmd[3425]:   notice: abort_transition_graph: Transition aborted by status-kcsandnp01-probe_complete, probe_complete=true: Transient attribute change (create cib=0.488.12, source=te_update_diff:391, path=/cib/status/node_state[@id='kcsandnp01']/transient_attributes[@id='kcsandnp01']/instance_attributes[@id='status-kcsandnp01'], 0)
Feb 16 00:19:58 kcsandnp01 stonith-ng[3420]:   notice: log_operation: Operation 'reboot' [3503] (call 2 from stonith_admin.cman.3501) for host 'kcsandnp02' with device 'vmfence' returned: 0 (OK)
Feb 16 00:19:58 kcsandnp01 stonith-ng[3420]:  warning: get_xpath_object: No match for //@st_delegate in /st-reply
Feb 16 00:19:58 kcsandnp01 stonith-ng[3420]:   notice: remote_op_done: Operation reboot of kcsandnp02 by kcsandnp01 for stonith_admin.cman.3501@kcsandnp01.f52f4439: OK
Feb 16 00:19:58 kcsandnp01 stonith-ng[3420]:   notice: remote_op_done: Operation reboot of kcsandnp02 by kcsandnp01 for crmd.3425@kcsandnp01.15dca67a: OK
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Peer kcsandnp02 was terminated (reboot) by kcsandnp01 for kcsandnp01: OK (ref=f52f4439-0222-4aa7-be96-f05e389e73c6) by client stonith_admin.cman.3501
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Notified CMAN that 'kcsandnp02' is now fenced
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Confirmed CMAN fencing event for 'kcsandnp02'
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: crm_update_peer_state: crmd_peer_down: Node kcsandnp02[2] - state is now lost (was (null))
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_callback: Stonith operation 2/24:0:0:5bdde6d8-16e7-416b-8d88-a2a4133735ab: OK (0)
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Peer kcsandnp02 was terminated (reboot) by kcsandnp01 for kcsandnp01: OK (ref=15dca67a-7b0e-4470-917c-db41628c18b5) by client crmd.3425
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Notified CMAN that 'kcsandnp02' is now fenced
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: tengine_stonith_notify: Confirmed CMAN fencing event for 'kcsandnp02'
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 0 (Complete=11, Pending=0, Fired=0, Skipped=6, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-warn-8.bz2): Stopped
Feb 16 00:19:58 kcsandnp01 fenced[3239]: fence kcsandnp02 success
Feb 16 00:19:58 kcsandnp01 pengine[3423]:   notice: LogActions: Start   vmfence#011(kcsandnp01)
Feb 16 00:19:58 kcsandnp01 pengine[3423]:   notice: LogActions: Start   ping_gw:0#011(kcsandnp01)
Feb 16 00:19:58 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 1: /var/lib/pacemaker/pengine/pe-input-106.bz2
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 4: start vmfence_start_0 on kcsandnp01 (local)
Feb 16 00:19:58 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 10: start ping_gw_start_0 on kcsandnp01 (local)
Feb 16 00:20:00 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ping_gw_start_0: ok (node=kcsandnp01, call=32, rc=0, cib-update=47, confirmed=true)
Feb 16 00:20:00 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 11: monitor ping_gw_monitor_10000 on kcsandnp01 (local)
Feb 16 00:20:02 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ping_gw_monitor_10000: ok (node=kcsandnp01, call=33, rc=0, cib-update=48, confirmed=false)
Feb 16 00:20:05 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (1000)
Feb 16 00:20:05 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 9: pingd=1000
Feb 16 00:20:05 kcsandnp01 crmd[3425]:   notice: abort_transition_graph: Transition aborted by status-kcsandnp01-pingd, pingd=1000: Transient attribute change (create cib=0.488.17, source=te_update_diff:391, path=/cib/status/node_state[@id='kcsandnp01']/transient_attributes[@id='kcsandnp01']/instance_attributes[@id='status-kcsandnp01'], 0)
Feb 16 00:20:13 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vmfence_start_0: ok (node=kcsandnp01, call=31, rc=0, cib-update=49, confirmed=true)
Feb 16 00:20:13 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 1 (Complete=5, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-106.bz2): Stopped
Feb 16 00:20:13 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 2: /var/lib/pacemaker/pengine/pe-input-107.bz2
Feb 16 00:20:13 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 7: monitor vmfence_monitor_60000 on kcsandnp01 (local)
Feb 16 00:20:27 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vmfence_monitor_60000: ok (node=kcsandnp01, call=34, rc=0, cib-update=51, confirmed=false)
Feb 16 00:20:27 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 2 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-107.bz2): Complete
Feb 16 00:20:27 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:21:20 kcsandnp01 corosync[3172]:   [TOTEM ] A processor joined or left the membership and a new membership was formed.
Feb 16 00:21:20 kcsandnp01 corosync[3172]:   [QUORUM] Members[2]: 1 2
Feb 16 00:21:20 kcsandnp01 corosync[3172]:   [QUORUM] Members[2]: 1 2
Feb 16 00:21:20 kcsandnp01 crmd[3425]:   notice: crm_update_peer_state: cman_event_callback: Node kcsandnp02[2] - state is now member (was lost)
Feb 16 00:21:20 kcsandnp01 corosync[3172]:   [CPG   ] chosen downlist: sender r(0) ip(10.1.204.8) ; members(old:1 left:0)
Feb 16 00:21:20 kcsandnp01 corosync[3172]:   [MAIN  ] Completed service synchronization, ready to provide service.
Feb 16 00:21:26 kcsandnp01 stonith-ng[3420]:   notice: crm_update_peer_state: st_peer_update_callback: Node kcsandnp02[2] - state is now member (was (null))
Feb 16 00:21:26 kcsandnp01 cib[3419]:   notice: crm_update_peer_state: cib_peer_update_callback: Node kcsandnp02[2] - state is now member (was (null))
Feb 16 00:21:27 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=peer_update_callback ]
Feb 16 00:21:32 kcsandnp01 attrd[3422]:   notice: attrd_local_callback: Sending full refresh (origin=crmd)
Feb 16 00:21:32 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: shutdown (0)
Feb 16 00:21:32 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Feb 16 00:21:32 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: pingd (1000)
Feb 16 00:21:33 kcsandnp01 pengine[3423]:   notice: LogActions: Start   ping_gw:1#011(kcsandnp02)
Feb 16 00:21:33 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 7: monitor vmfence_monitor_0 on kcsandnp02
Feb 16 00:21:33 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 8: monitor ha_lvm_monitor_0 on kcsandnp02
Feb 16 00:21:33 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 3: /var/lib/pacemaker/pengine/pe-input-108.bz2
Feb 16 00:21:34 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 9: monitor mqha_fs_monitor_0 on kcsandnp02
Feb 16 00:21:34 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 10: monitor vip_monitor_0 on kcsandnp02
Feb 16 00:21:35 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 11: monitor wmq_monitor_0 on kcsandnp02
Feb 16 00:21:35 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 12: monitor iib_monitor_0 on kcsandnp02
Feb 16 00:21:37 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 13: monitor ping_gw:1_monitor_0 on kcsandnp02
Feb 16 00:21:37 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 6: probe_complete probe_complete-kcsandnp02 on kcsandnp02 - no waiting
Feb 16 00:21:37 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 22: start ping_gw:1_start_0 on kcsandnp02
Feb 16 00:21:37 kcsandnp01 crmd[3425]:   notice: abort_transition_graph: Transition aborted by status-kcsandnp02-probe_complete, probe_complete=true: Transient attribute change (create cib=0.488.35, source=te_update_diff:391, path=/cib/status/node_state[@id='kcsandnp02']/transient_attributes[@id='kcsandnp02']/instance_attributes[@id='status-kcsandnp02'], 0)
Feb 16 00:21:39 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 3 (Complete=12, Pending=0, Fired=0, Skipped=1, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-108.bz2): Stopped
Feb 16 00:21:39 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 4: /var/lib/pacemaker/pengine/pe-input-109.bz2
Feb 16 00:21:39 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 17: monitor ping_gw_monitor_10000 on kcsandnp02
Feb 16 00:21:41 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 4 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-109.bz2): Complete
Feb 16 00:21:41 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:21:44 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:21:44 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 5: /var/lib/pacemaker/pengine/pe-input-110.bz2
Feb 16 00:21:44 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 5 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-110.bz2): Complete
Feb 16 00:21:44 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:36:44 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 00:36:44 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 6: /var/lib/pacemaker/pengine/pe-input-110.bz2
Feb 16 00:36:44 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 6 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-110.bz2): Complete
Feb 16 00:36:44 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:41:54 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: LogActions: Start   ha_lvm#011(kcsandnp01)
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: LogActions: Start   mqha_fs#011(kcsandnp01)
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: LogActions: Start   vip#011(kcsandnp01)
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: LogActions: Start   wmq#011(kcsandnp01)
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 16 00:41:54 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 10: start ha_lvm_start_0 on kcsandnp01 (local)
Feb 16 00:41:54 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-111.bz2
Feb 16 00:41:55 kcsandnp01 LVM(ha_lvm)[8248]: INFO: Activating volume group vg_share
Feb 16 00:41:55 kcsandnp01 LVM(ha_lvm)[8248]: INFO: Reading all physical volumes. This may take a while... Found volume group "vg_share" using metadata type lvm2 Found volume group "vg00" using metadata type lvm2
Feb 16 00:41:55 kcsandnp01 LVM(ha_lvm)[8248]: INFO: New tag "pacemaker" added to vg_share
Feb 16 00:41:55 kcsandnp01 LVM(ha_lvm)[8248]: INFO: 1 logical volume(s) in volume group "vg_share" now active
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_start_0: ok (node=kcsandnp01, call=35, rc=0, cib-update=85, confirmed=true)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 11: monitor ha_lvm_monitor_10000 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 12: start mqha_fs_start_0 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 Filesystem(mqha_fs)[8355]: INFO: Running start for /dev/vg_share/mqhalv on /mqha
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: ok (node=kcsandnp01, call=36, rc=0, cib-update=86, confirmed=false)
Feb 16 00:41:55 kcsandnp01 kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_start_0: ok (node=kcsandnp01, call=37, rc=0, cib-update=87, confirmed=true)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 13: monitor mqha_fs_monitor_10000 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 14: start vip_start_0 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_monitor_10000: ok (node=kcsandnp01, call=38, rc=0, cib-update=88, confirmed=false)
Feb 16 00:41:55 kcsandnp01 IPaddr2(vip)[8448]: INFO: Adding inet address 192.168.141.10/27 with broadcast address 192.168.141.31 to device eth0
Feb 16 00:41:55 kcsandnp01 IPaddr2(vip)[8448]: INFO: Bringing device eth0 up
Feb 16 00:41:55 kcsandnp01 IPaddr2(vip)[8448]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.141.10 eth0 192.168.141.10 auto not_used not_used
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_start_0: ok (node=kcsandnp01, call=39, rc=0, cib-update=89, confirmed=true)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 15: monitor vip_monitor_30000 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 16: start wmq_start_0 on kcsandnp01 (local)
Feb 16 00:41:55 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_monitor_30000: ok (node=kcsandnp01, call=40, rc=0, cib-update=90, confirmed=false)
Feb 16 00:41:57 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=41, rc=0, cib-update=91, confirmed=true)
Feb 16 00:41:57 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 17: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 16 00:41:57 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 18: start iib_start_0 on kcsandnp01 (local)
Feb 16 00:41:57 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=42, rc=0, cib-update=92, confirmed=false)
Feb 16 00:42:00 kcsandnp01 IIB[9335]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9335] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 16 00:42:01 kcsandnp01 IIB[9366]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9366] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 9371. 
Feb 16 00:42:03 kcsandnp01 IIB[9371]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9371] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 16 00:42:06 kcsandnp01 IIB[9371]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9371] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 16 00:42:06 kcsandnp01 IIB[9371]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9516] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 16 00:42:07 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9556] (Msg 1/1) BIP2208I: Integration server (64) started: process '9556'; thread '9556'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 16 00:42:12 kcsandnp01 IIB[9371]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9547] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 16 00:42:20 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9748] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 16 00:42:20 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9748] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 16 00:42:21 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9748] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 16 00:42:24 kcsandnp01 IIB[9517]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 9576] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 16 00:42:28 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9744] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 00:42:28 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9744] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 00:42:28 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9746] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 16 00:42:32 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=43, rc=0, cib-update=93, confirmed=true)
Feb 16 00:42:32 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 19: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 16 00:42:33 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=44, rc=0, cib-update=94, confirmed=false)
Feb 16 00:42:33 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 7 (Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-111.bz2): Complete
Feb 16 00:42:33 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:43:58 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: LogActions: Move    ha_lvm#011(Started kcsandnp01 -> kcsandnp02)
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: LogActions: Move    mqha_fs#011(Started kcsandnp01 -> kcsandnp02)
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: LogActions: Move    vip#011(Started kcsandnp01 -> kcsandnp02)
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: LogActions: Move    wmq#011(Started kcsandnp01 -> kcsandnp02)
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: LogActions: Move    iib#011(Started kcsandnp01 -> kcsandnp02)
Feb 16 00:43:58 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 27: stop iib_stop_0 on kcsandnp01 (local)
Feb 16 00:43:58 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 8: /var/lib/pacemaker/pengine/pe-input-112.bz2
Feb 16 00:43:59 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9717] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 16 00:43:59 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9717] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 16 00:44:11 kcsandnp01 IIB[9556]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 9556] (Msg 1/1) BIP2204I: Integration server using process '9556' thread '9556' stopped. 
Feb 16 00:44:17 kcsandnp01 IIB[9371]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9516] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 16 00:44:18 kcsandnp01 IIB[9366]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 9366] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 16 00:44:19 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=46, rc=0, cib-update=96, confirmed=true)
Feb 16 00:44:19 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 24: stop wmq_stop_0 on kcsandnp01 (local)
Feb 16 00:44:22 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=48, rc=0, cib-update=97, confirmed=true)
Feb 16 00:44:22 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 21: stop vip_stop_0 on kcsandnp01 (local)
Feb 16 00:44:22 kcsandnp01 IPaddr2(vip)[13075]: INFO: IP status = ok, IP_CIP=
Feb 16 00:44:22 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_stop_0: ok (node=kcsandnp01, call=50, rc=0, cib-update=98, confirmed=true)
Feb 16 00:44:22 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 18: stop mqha_fs_stop_0 on kcsandnp01 (local)
Feb 16 00:44:22 kcsandnp01 Filesystem(mqha_fs)[13133]: INFO: Running stop for /dev/vg_share/mqhalv on /mqha
Feb 16 00:44:22 kcsandnp01 Filesystem(mqha_fs)[13133]: INFO: Trying to unmount /mqha
Feb 16 00:44:23 kcsandnp01 Filesystem(mqha_fs)[13133]: INFO: unmounted /mqha successfully
Feb 16 00:44:23 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_stop_0: ok (node=kcsandnp01, call=52, rc=0, cib-update=99, confirmed=true)
Feb 16 00:44:23 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 15: stop ha_lvm_stop_0 on kcsandnp01 (local)
Feb 16 00:44:23 kcsandnp01 LVM(ha_lvm)[13212]: INFO: Deactivating volume group vg_share
Feb 16 00:44:23 kcsandnp01 LVM(ha_lvm)[13212]: INFO: 0 logical volume(s) in volume group "vg_share" now active
Feb 16 00:44:23 kcsandnp01 LVM(ha_lvm)[13212]: INFO: LVM Volume vg_share is not available (stopped)
Feb 16 00:44:23 kcsandnp01 LVM(ha_lvm)[13212]: INFO: Stripping tag, pacemaker
Feb 16 00:44:23 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_stop_0: ok (node=kcsandnp01, call=54, rc=0, cib-update=100, confirmed=true)
Feb 16 00:44:23 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 16: start ha_lvm_start_0 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 17: monitor ha_lvm_monitor_10000 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 19: start mqha_fs_start_0 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 20: monitor mqha_fs_monitor_10000 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 22: start vip_start_0 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 23: monitor vip_monitor_30000 on kcsandnp02
Feb 16 00:44:24 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 25: start wmq_start_0 on kcsandnp02
Feb 16 00:44:27 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 26: monitor wmq_monitor_60000 on kcsandnp02
Feb 16 00:44:27 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 28: start iib_start_0 on kcsandnp02
Feb 16 00:45:22 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 29: monitor iib_monitor_60000 on kcsandnp02
Feb 16 00:45:23 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 8 (Complete=20, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-112.bz2): Complete
Feb 16 00:45:23 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:47:36 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: LogActions: Move    ha_lvm#011(Started kcsandnp02 -> kcsandnp01)
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: LogActions: Move    mqha_fs#011(Started kcsandnp02 -> kcsandnp01)
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: LogActions: Move    vip#011(Started kcsandnp02 -> kcsandnp01)
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: LogActions: Move    wmq#011(Started kcsandnp02 -> kcsandnp01)
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: LogActions: Move    iib#011(Started kcsandnp02 -> kcsandnp01)
Feb 16 00:47:36 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 27: stop iib_stop_0 on kcsandnp02
Feb 16 00:47:36 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 9: /var/lib/pacemaker/pengine/pe-input-113.bz2
Feb 16 00:48:02 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 24: stop wmq_stop_0 on kcsandnp02
Feb 16 00:48:05 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 21: stop vip_stop_0 on kcsandnp02
Feb 16 00:48:05 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 18: stop mqha_fs_stop_0 on kcsandnp02
Feb 16 00:48:06 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 15: stop ha_lvm_stop_0 on kcsandnp02
Feb 16 00:48:06 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 16: start ha_lvm_start_0 on kcsandnp01 (local)
Feb 16 00:48:06 kcsandnp01 LVM(ha_lvm)[13701]: INFO: Activating volume group vg_share
Feb 16 00:48:06 kcsandnp01 LVM(ha_lvm)[13701]: INFO: Reading all physical volumes. This may take a while... Found volume group "vg_share" using metadata type lvm2 Found volume group "vg00" using metadata type lvm2
Feb 16 00:48:06 kcsandnp01 LVM(ha_lvm)[13701]: INFO: New tag "pacemaker" added to vg_share
Feb 16 00:48:06 kcsandnp01 LVM(ha_lvm)[13701]: INFO: 1 logical volume(s) in volume group "vg_share" now active
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_start_0: ok (node=kcsandnp01, call=55, rc=0, cib-update=102, confirmed=true)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 17: monitor ha_lvm_monitor_10000 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 19: start mqha_fs_start_0 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 Filesystem(mqha_fs)[13808]: INFO: Running start for /dev/vg_share/mqhalv on /mqha
Feb 16 00:48:07 kcsandnp01 kernel: EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: 
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_start_0: ok (node=kcsandnp01, call=57, rc=0, cib-update=103, confirmed=true)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation ha_lvm_monitor_10000: ok (node=kcsandnp01, call=56, rc=0, cib-update=104, confirmed=false)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 20: monitor mqha_fs_monitor_10000 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 22: start vip_start_0 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation mqha_fs_monitor_10000: ok (node=kcsandnp01, call=58, rc=0, cib-update=105, confirmed=false)
Feb 16 00:48:07 kcsandnp01 IPaddr2(vip)[13901]: INFO: Adding inet address 192.168.141.10/27 with broadcast address 192.168.141.31 to device eth0
Feb 16 00:48:07 kcsandnp01 IPaddr2(vip)[13901]: INFO: Bringing device eth0 up
Feb 16 00:48:07 kcsandnp01 IPaddr2(vip)[13901]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-192.168.141.10 eth0 192.168.141.10 auto not_used not_used
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_start_0: ok (node=kcsandnp01, call=59, rc=0, cib-update=106, confirmed=true)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 23: monitor vip_monitor_30000 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 25: start wmq_start_0 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation vip_monitor_30000: ok (node=kcsandnp01, call=60, rc=0, cib-update=107, confirmed=false)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=61, rc=0, cib-update=108, confirmed=true)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 26: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 28: start iib_start_0 on kcsandnp01 (local)
Feb 16 00:48:07 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=62, rc=0, cib-update=109, confirmed=false)
Feb 16 00:48:08 kcsandnp01 IIB[14785]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14785] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 16 00:48:08 kcsandnp01 IIB[14828]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14828] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 14833. 
Feb 16 00:48:09 kcsandnp01 IIB[14833]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14833] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 16 00:48:10 kcsandnp01 IIB[14833]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14833] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 16 00:48:12 kcsandnp01 IIB[14833]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14900] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 16 00:48:13 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 14948] (Msg 1/1) BIP2208I: Integration server (64) started: process '14948'; thread '14948'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 16 00:48:17 kcsandnp01 IIB[14833]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14950] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 16 00:48:22 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15117] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 16 00:48:22 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15117] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 16 00:48:23 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15117] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 16 00:48:26 kcsandnp01 IIB[14901]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 14926] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 16 00:48:29 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15110] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 00:48:30 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15110] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 00:48:31 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15112] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 16 00:48:39 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=63, rc=0, cib-update=110, confirmed=true)
Feb 16 00:48:39 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 29: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 16 00:48:40 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=64, rc=0, cib-update=111, confirmed=false)
Feb 16 00:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 9 (Complete=20, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-113.bz2): Complete
Feb 16 00:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 00:57:25 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO1' of type '.MSGFLOW'. 
Feb 16 00:57:27 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO1' of type '.MSGFLOW'. 
Feb 16 00:57:30 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO2' of type '.MSGFLOW'. 
Feb 16 00:57:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO2' of type '.MSGFLOW'. 
Feb 16 00:57:37 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO3' of type '.MSGFLOW'. 
Feb 16 00:57:41 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO3' of type '.MSGFLOW'. 
Feb 16 00:57:45 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Req_PO4' of type '.MSGFLOW'. 
Feb 16 00:57:47 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Outbound_Res_PO4' of type '.MSGFLOW'. 
Feb 16 00:57:54 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI1' of type '.MSGFLOW'. 
Feb 16 00:57:56 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI1' of type '.MSGFLOW'. 
Feb 16 00:57:58 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI2' of type '.MSGFLOW'. 
Feb 16 00:58:01 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI2' of type '.MSGFLOW'. 
Feb 16 00:58:04 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI3' of type '.MSGFLOW'. 
Feb 16 00:58:06 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI3' of type '.MSGFLOW'. 
Feb 16 00:58:08 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Req_PI4' of type '.MSGFLOW'. 
Feb 16 00:58:10 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2155I: About to 'start ' the deployed resource 'AnyID_CrTfr_ITMX_Inbound_Res_PI4' of type '.MSGFLOW'. 
Feb 16 01:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 01:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 10: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 01:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 10 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 01:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 01:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 01:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 11 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 01:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 01:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 11: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 01:25:01 kcsandnp01 crmd[3425]:   notice: throttle_handle_load: High CPU load detected: 3.420000
Feb 16 01:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 01:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 12: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 01:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 12 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 01:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 01:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 01:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 13: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 01:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 13 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 01:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 02:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 02:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 14: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 02:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 14 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 02:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 02:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 02:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 15 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 02:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 02:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 15: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 02:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 02:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 16: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 02:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 16 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 02:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 02:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 02:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 17: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 02:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 17 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 02:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 02:50:40 kcsandnp01 kernel: hrtimer: interrupt took 1198225 ns
Feb 16 03:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 03:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 18: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 03:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 18 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 03:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 03:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 03:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 19: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 03:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 19 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 03:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 03:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 03:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 20: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 03:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 20 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 03:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 03:45:44 kcsandnp01 rhsmd: In order for Subscription Manager to provide your system with updates, your system must be registered with the Customer Portal. Please enter your Red Hat login to ensure your system is up-to-date.
Feb 16 03:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 03:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 21 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 03:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 03:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 21: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 04:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 04:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 22: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 04:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 22 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 04:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 04:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 04:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 23: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 04:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 23 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 04:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 04:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 04:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 24: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 04:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 24 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 04:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 04:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 04:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 25 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 04:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 04:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 25: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 05:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 05:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 26: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 05:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 26 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 05:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 05:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 05:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 27: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 05:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 27 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 05:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 05:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 05:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 28: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 05:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 28 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 05:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 05:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 05:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 29: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 05:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 29 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 05:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 06:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 06:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 30: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 06:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 30 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 06:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 06:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 06:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 31: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 06:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 31 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 06:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 06:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 06:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 32: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 06:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 32 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 06:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 06:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 06:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 33: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 06:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 33 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 06:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 07:02:54 kcsandnp01 auditd[2190]: Audit daemon rotating log files
Feb 16 07:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 07:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 34: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 07:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 34 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 07:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 07:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 07:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 35 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 07:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 07:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 35: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 07:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 07:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 36: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 07:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 36 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 07:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 07:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 07:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 37: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 07:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 37 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 07:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 08:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 08:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 38: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 08:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 38 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 08:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 08:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 08:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 39: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 08:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 39 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 08:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 08:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 08:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 40 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 08:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 08:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 40: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 08:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 08:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 41 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 08:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 08:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 41: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 09:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 09:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 42: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 09:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 42 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 09:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 09:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 09:18:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 43: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 09:18:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 43 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 09:18:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 09:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 09:33:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 44: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 09:33:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 44 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 09:33:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 09:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 09:48:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 45: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 09:48:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 45 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 09:48:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 10:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 10:03:40 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 46: /var/lib/pacemaker/pengine/pe-input-114.bz2
Feb 16 10:03:40 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 46 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-114.bz2): Complete
Feb 16 10:03:40 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: reservation conflict
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb]  Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 16 10:06:22 kcsandnp01 kernel: sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 00 0e 25 3f 00 00 08 00
Feb 16 10:06:22 kcsandnp01 kernel: end_request: critical nexus error, dev sdb, sector 927039
Feb 16 10:06:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:22 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:22 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27546] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1'. 
Feb 16 10:06:22 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27546] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI1.MQ In: ANYID_TO_ITMX_PI1' to queue 'ANYID_TO_ITMX_PI1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:23 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 16 10:06:23 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/1) BIP2153I: About to 'Stop' an integration server. 
Feb 16 10:06:24 kcsandnp01 IIB[14833]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14900] (Msg 1/1) BIP9155I: The MQTT pub/sub broker, configured to listen on port '11883', has stopped. 
Feb 16 10:06:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:24 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:24 kcsandnp01 IIB[14828]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14828] (Msg 1/1) BIP2015E: The IBM Integration Bus service detected that executable bipbroker terminated, exit code 11. 
Feb 16 10:06:24 kcsandnp01 IIB[14828]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14828] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 45278. 
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14141.0.FDC
Feb 16 10:06:25 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14049.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14049.0.FDC
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26423] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26423] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15131] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15131] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Res.MQ In: ANYID_RES_FROM_ITMX' to queue 'ANYID_RES_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15132] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15132] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14049.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14049.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14101.0.FDC
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26469] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26469] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15134] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15130] (Msg 1/1) BIP2112E: Integration node internal error: diagnostic information '2', '2009'. 
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27673] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2'. 
Feb 16 10:06:33 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27673] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI2.MQ In: ANYID_TO_ITMX_PI2' to queue 'ANYID_TO_ITMX_PI2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:33 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27768] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27768] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI3.MQ In: ANYID_TO_ITMX_PI3' to queue 'ANYID_TO_ITMX_PI3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27897] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27897] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Inbound_Res_PI4.MQ In: ANYID_TO_ITMX_PI4' to queue 'ANYID_TO_ITMX_PI4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26371] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26371] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_TO_ITMX_PO1' to queue 'ANYID_TO_ITMX_PO1' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26464] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26464] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO2.MQ In: ANYID_TO_ITMX_PO2' to queue 'ANYID_TO_ITMX_PO2' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26612] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26612] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_TO_ITMX_PO3' to queue 'ANYID_TO_ITMX_PO3' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26790] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26790] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_TO_ITMX_PO4' to queue 'ANYID_TO_ITMX_PO4' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15123] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15123] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_ITMX_Reply_0810.MQIn: ITMX_IN_0810' to queue 'ITMX_IN_0810' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26793] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26793] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO4.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26390] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26390] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26380] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26380] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_LogMsg_DB.MQ In: ANYID_ITMX_MSG_LOG' to queue 'ANYID_ITMX_MSG_LOG' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27681] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27681] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27903] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27903] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_Distribute_ITMX_Req.MQ In: ANYID_REQ_FROM_ITMX' to queue 'ANYID_REQ_FROM_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26374] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26374] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO1.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26615] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX'. 
Feb 16 10:06:34 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 26615] (Msg 2/2) BIP2680E: Failed to setup a MQ control callback for component 'AnyID_CrTfr_ITMX_Outbound_Req_PO3.MQ In: ANYID_REQ_TO_ITMX' to queue 'ANYID_REQ_TO_ITMX' on queue manager 'ESBXPRD01': MQCC=2; MQRC=2009. 
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:34 kcsandnp01 MQSeries: FFST record created in /var/mqm/errors/AMQ14086.0.FDC
Feb 16 10:06:35 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27868] (Msg 1/3) BIP2628W: Exception condition detected on input node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.TCPIP Client In Rcv ISO'. 
Feb 16 10:06:35 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27868] (Msg 2/3) BIP2230E: Error detected whilst processing a message in node 'AnyID_CrTfr_ITMX_Inbound_Req_PI4.MQ Out: ANYID_ITMX_MSG_LOG '. 
Feb 16 10:06:35 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 27868] (Msg 3/3) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15086] (Msg 1/2) BIP2124E: The integration server has detected a condition on thread number '15086' with name 'MQQM Monitor Thread', which requires it to shutdown and restart. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15086] (Msg 2/2) BIP2674E: A broken connection to WebSphere MQ queue manager has been reported with WebSphere MQ reason code '2059'. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQ' transport. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using the 'MQTT' transport. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15087] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status/ExecutionGroup/anyid_interface_itmx' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.14948', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 16 10:06:43 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_monitor_60000: not running (node=kcsandnp01, call=64, rc=7, cib-update=149, confirmed=false)
Feb 16 10:06:43 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:64 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 16 10:06:43 kcsandnp01 crmd[3425]:  warning: update_failcount: Updating failcount for iib on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1487214403)
Feb 16 10:06:43 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]
Feb 16 10:06:43 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-iib (1)
Feb 16 10:06:43 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 25: fail-count-iib=1
Feb 16 10:06:43 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-iib (1487214403)
Feb 16 10:06:43 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 27: last-failure-iib=1487214403
Feb 16 10:06:43 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:06:43 kcsandnp01 pengine[3423]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 16 10:06:43 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 47: /var/lib/pacemaker/pengine/pe-input-115.bz2
Feb 16 10:06:43 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:06:43 kcsandnp01 pengine[3423]:   notice: LogActions: Recover iib#011(Started kcsandnp01)
Feb 16 10:06:43 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 5: stop iib_stop_0 on kcsandnp01 (local)
Feb 16 10:06:43 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-iib_monitor_60000:64 [ hamqsi_monitor_broker_as: Queue Manager ESBXPRD01 is not responsive\nhamqsi_monitor_broker_as: Queue manager (ESBXPRD01) is not running correctly\n ]
Feb 16 10:06:43 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 48: /var/lib/pacemaker/pengine/pe-input-116.bz2
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15134] (Msg 1/2) BIP2628W: Exception condition detected on input node 'AnyID_ITMX_NetMgmt_0800.MQ In: ANYID_ITMX_MGMT_CMD'. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15134] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15130] (Msg 1/2) BIP2628W: Exception condition detected on input node 'ESB_LoadCache.ANYID_CACHE_CMD.IN'. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15130] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15121] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15121] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15120] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 16 10:06:43 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15120] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:44 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15122] (Msg 1/2) BIP4617E: The TimeoutNotification node 'AnyID_ITMX_NetMgmt_0800.Timeout Notification' failed to access its internal store on queue 'SYSTEM.BROKER.TIMEOUT.QUEUE'. 
Feb 16 10:06:44 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15122] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:44 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15119] (Msg 1/2) BIP3115E: An internal error has occurred in message flow node or resource manager 'HTTP Input'. 
Feb 16 10:06:44 kcsandnp01 IIB[14948]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 15119] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:44 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 1/1) BIP8236I: A standby instance of integration node ESBXPRD01BK has been started against the standby queue manager ESBXPRD01. The active integration node instance and active queue manager instance are running elsewhere. 
Feb 16 10:06:49 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic '$SYS/Broker/ESBXPRD01BK/Status' using the 'MQ' transport. 
Feb 16 10:06:49 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 2/2) BIP2678E: Failed to make a server connection to queue manager 'ESBXPRD01': MQCC=2; MQRC=2059. 
Feb 16 10:06:51 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 1/2) BIP4078E: An error occurred publishing integration node event on topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using the 'MQTT' transport. 
Feb 16 10:06:51 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 2/2) BIP4311E: An MQTT Publish using 'pubsub_AdminEvents_MQTT_localhost_11883_pubsubDefault' detected an exception when it was attempting to publish to topic 'IBM/IntegrationBus/ESBXPRD01BK/Status' using server connection 'tcp://localhost:11883', Client id 'ESBXPRD01BK.45278', Security identity username '' and Security identity 'pubsubDefault'.  The exception is 'Client is not connected'.  The exception stack trace is '[org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:31), org.eclipse.paho.client.mqttv3.internal.ClientComms.sendNoWait(ClientComms.java:143), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:859), org.eclipse.paho.client.mqttv3.MqttAsyncClient.publish(MqttAsyncClient.java:837), com.ibm.broker.connector.mqtt.MQTTOutputInteraction.send(MQTTOutputInteraction.java:126)]'. 
Feb 16 10:06:51 kcsandnp01 IIB[45278]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 45278] (Msg 1/1) BIP2807E: The integration node received a negative response to a 'Stop' publish event. 
Feb 16 10:06:51 kcsandnp01 IIB[14828]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 14828] (Msg 1/1) BIP2002W: The IBM Integration Bus service has stopped. 
Feb 16 10:07:04 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_monitor_60000: not running (node=kcsandnp01, call=62, rc=7, cib-update=153, confirmed=false)
Feb 16 10:07:04 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:62 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 16 10:07:04 kcsandnp01 crmd[3425]:   notice: abort_transition_graph: Transition aborted by wmq_monitor_60000 'create' on kcsandnp01: Old event (magic=0:7;26:9:0:5bdde6d8-16e7-416b-8d88-a2a4133735ab, cib=0.491.19, source=process_graph_event:555, 0)
Feb 16 10:07:04 kcsandnp01 crmd[3425]:  warning: update_failcount: Updating failcount for wmq on kcsandnp01 after failed monitor: rc=7 (update=value++, time=1487214424)
Feb 16 10:07:04 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-wmq (1)
Feb 16 10:07:04 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 29: fail-count-wmq=1
Feb 16 10:07:04 kcsandnp01 attrd[3422]:   notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-wmq (1487214424)
Feb 16 10:07:04 kcsandnp01 attrd[3422]:   notice: attrd_perform_update: Sent update 31: last-failure-wmq=1487214424
Feb 16 10:07:47 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_stop_0: ok (node=kcsandnp01, call=66, rc=0, cib-update=154, confirmed=true)
Feb 16 10:07:47 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 48 (Complete=2, Pending=0, Fired=0, Skipped=6, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-116.bz2): Stopped
Feb 16 10:07:47 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:07:47 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 10:07:47 kcsandnp01 pengine[3423]:   notice: LogActions: Recover wmq#011(Started kcsandnp01)
Feb 16 10:07:47 kcsandnp01 pengine[3423]:   notice: LogActions: Start   iib#011(kcsandnp01)
Feb 16 10:07:47 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 49: /var/lib/pacemaker/pengine/pe-input-117.bz2
Feb 16 10:07:47 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 7: stop wmq_stop_0 on kcsandnp01 (local)
Feb 16 10:07:47 kcsandnp01 crmd[3425]:   notice: process_lrm_event: kcsandnp01-wmq_monitor_60000:62 [ Checking Queue Manager Status:\nGetting status of queue manager ESBXPRD01\nQueue manager ESBXPRD01 status is: Ended preemptively\033[60G[\033[0;32m  OK  \033[0;39m]\r\n ]
Feb 16 10:07:48 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_stop_0: ok (node=kcsandnp01, call=68, rc=0, cib-update=156, confirmed=true)
Feb 16 10:07:48 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 21: start wmq_start_0 on kcsandnp01 (local)
Feb 16 10:07:49 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_start_0: ok (node=kcsandnp01, call=69, rc=0, cib-update=157, confirmed=true)
Feb 16 10:07:49 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 6: monitor wmq_monitor_60000 on kcsandnp01 (local)
Feb 16 10:07:49 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 22: start iib_start_0 on kcsandnp01 (local)
Feb 16 10:07:49 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation wmq_monitor_60000: ok (node=kcsandnp01, call=70, rc=0, cib-update=158, confirmed=false)
Feb 16 10:07:49 kcsandnp01 abrt[47149]: Saved core dump of pid 14948 (/opt/ibm/mqsi/iib-10.0.0.7/server/bin/DataFlowEngine) to /var/spool/abrt/ccpp-2017-02-16-10:07:38-14948 (1479761920 bytes)
Feb 16 10:07:49 kcsandnp01 abrtd: Directory 'ccpp-2017-02-16-10:07:38-14948' creation detected
Feb 16 10:07:49 kcsandnp01 abrtd: Executable '/opt/ibm/mqsi/iib-10.0.0.7/server/bin/DataFlowEngine' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Feb 16 10:07:49 kcsandnp01 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2017-02-16-10:07:38-14948' exited with 1
Feb 16 10:07:49 kcsandnp01 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2017-02-16-10:07:38-14948'
Feb 16 10:07:49 kcsandnp01 IIB[48088]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48088] (Msg 1/1) BIP8873I: Starting the component verification for component 'ESBXPRD01BK'. 
Feb 16 10:07:50 kcsandnp01 IIB[48113]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48113] (Msg 1/1) BIP2001I: The IBM Integration Bus service has started at version 10007; process ID 48118. 
Feb 16 10:07:50 kcsandnp01 IIB[48118]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48118] (Msg 1/1) BIP2866I: IBM Integration Bus administration security is active. 
Feb 16 10:07:52 kcsandnp01 IIB[48118]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48118] (Msg 1/1) BIP2777I: Authentication for the web administration is configured to use the local web user registry. 
Feb 16 10:07:52 kcsandnp01 IIB[48118]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48185] (Msg 1/1) BIP9153I: The MQTT pub/sub broker has started successfully, configured to listen on port 11883. 
Feb 16 10:07:53 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48241] (Msg 1/1) BIP2208I: Integration server (64) started: process '48241'; thread '48241'; additional information: integrationNodeName 'ESBXPRD01BK' (operation mode 'standard'); integrationServerUUID '2d5e5bba-1e8d-4739-b3c9-2bcb04bf2eed'; integrationServerLabel 'anyid_interface_itmx'; queueManagerName 'ESBXPRD01'; trusted 'false'; userId 'esbadm'; migrationNeeded 'false'; integrationNodeUUID 'b9ad0ab8-8531-11e6-8548-0a01cc080000'; filePath '/opt/ibm/mqsi/iib-10.0.0.7/server'; workPath '/mqha/IIB/ESBXPRD01BK/mqsi'; ICU Converter Path ''; ordinality '1'. 
Feb 16 10:07:58 kcsandnp01 IIB[48118]: IBM Integration Bus v10007 (ESBXPRD01BK) [Thread 48242] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '4414' for 'WebAdmin http' connections. 
Feb 16 10:08:02 kcsandnp01 crmd[3425]:   notice: throttle_handle_load: High CPU load detected: 20.240000
Feb 16 10:08:02 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48402] (Msg 1/1) BIP2152I: Configuration message received from integration node. 
Feb 16 10:08:02 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48402] (Msg 1/1) BIP2153I: About to 'Start' an integration server. 
Feb 16 10:08:04 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48402] (Msg 1/1) BIP2154I: Integration server finished with Configuration message. 
Feb 16 10:08:10 kcsandnp01 IIB[48186]: IBM Integration Bus v10007 (ESBXPRD01BK.HTTPListener) [Thread 48244] (Msg 1/1) BIP3132I: The HTTP Listener has started listening on port '7080' for 'http' connections. 
Feb 16 10:08:14 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48398] (Msg 1/1) BIP7162I: Catalog server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 10:08:15 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48398] (Msg 1/1) BIP7163I: Container server 'ESBXPRD01BK_kcsandnp01_2860' in integration server 'anyid_interface_itmx' has started successfully. 
Feb 16 10:08:15 kcsandnp01 IIB[48241]: IBM Integration Bus v10007 (ESBXPRD01BK.anyid_interface_itmx) [Thread 48400] (Msg 1/1) BIP7155I: The integration server has established a connection to the integration node global cache. 
Feb 16 10:08:19 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_start_0: ok (node=kcsandnp01, call=71, rc=0, cib-update=159, confirmed=true)
Feb 16 10:08:19 kcsandnp01 crmd[3425]:   notice: te_rsc_command: Initiating action 23: monitor iib_monitor_60000 on kcsandnp01 (local)
Feb 16 10:08:22 kcsandnp01 crmd[3425]:   notice: process_lrm_event: Operation iib_monitor_60000: ok (node=kcsandnp01, call=72, rc=0, cib-update=160, confirmed=false)
Feb 16 10:08:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 49 (Complete=10, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-117.bz2): Complete
Feb 16 10:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 10:08:32 kcsandnp01 crmd[3425]:   notice: throttle_handle_load: High CPU load detected: 12.910000
Feb 16 10:09:02 kcsandnp01 crmd[3425]:   notice: throttle_handle_load: High CPU load detected: 7.880000
Feb 16 10:09:32 kcsandnp01 crmd[3425]:   notice: throttle_handle_load: High CPU load detected: 4.930000
Feb 16 10:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 10:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 10:23:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 50: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 10:23:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 50 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 10:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 10:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 10:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 10:38:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 51 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 10:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 10:38:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 51: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 10:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 10:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 10:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 10:53:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 52: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 10:53:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 52 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 10:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 11:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 11:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 11:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 11:08:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 53: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 11:08:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 53 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 11:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 11:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 11:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 11:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 11:23:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 54: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 11:23:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 54 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 11:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 11:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 11:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 11:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 11:38:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 55: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 11:38:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 55 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 11:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 11:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 11:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 11:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 11:53:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 56 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 11:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 11:53:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 56: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 12:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 12:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 12:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 12:08:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 57: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 12:08:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 57 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 12:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 12:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 12:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 12:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 12:23:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 58: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 12:23:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 58 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 12:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 12:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 12:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 12:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 12:38:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 59: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 12:38:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 59 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 12:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 12:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 12:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 12:53:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 12:53:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 60: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 12:53:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 60 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 12:53:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 13:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 13:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 13:08:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 13:08:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 61: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 13:08:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 61 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 13:08:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 13:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 13:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 13:23:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 13:23:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 62: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 13:23:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 62 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 13:23:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb 16 13:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]
Feb 16 13:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for iib on kcsandnp01: not running (7)
Feb 16 13:38:22 kcsandnp01 pengine[3423]:  warning: unpack_rsc_op_failure: Processing failed op monitor for wmq on kcsandnp01: not running (7)
Feb 16 13:38:22 kcsandnp01 pengine[3423]:   notice: process_pe_message: Calculated Transition 63: /var/lib/pacemaker/pengine/pe-input-118.bz2
Feb 16 13:38:22 kcsandnp01 crmd[3425]:   notice: run_graph: Transition 63 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-118.bz2): Complete
Feb 16 13:38:22 kcsandnp01 crmd[3425]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
