Filesystem remounted as read-only when performing hardware RAID failover tests on HS23 in RHEL6.3
Issue
- On an IBM BladeCenter HS23p 7875AC1 with a LSI SAS2004 RAID controller in a RAID 1 (mirror) configuration, when performing a failure test by removing a physical disk from a running system, the below messages are recorded in syslog, and the mounted filesystem goes read-only.
Oct 4 11:31:02 server4 kernel: lpfc 0000:16:00.2: 0:(0):0110 ELS response tag x792 completes Data: x0 x0 x0 x150080 x80000000 x6 x4
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 00 e8 10 00 00 00 10 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 0
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 1
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 01 68 13 e8 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 1048701
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 01 69 22 98 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 1057363
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 01 68 10 80 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 1048592
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 00 e8 1e 88 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 465
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 00 e8 1f a0 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 500
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 00 e9 21 58 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 8747
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 01 68 13 28 00 00 08 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-2, logical block 1048677
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-2
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:26 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 01 f0 51 d0 00 00 38 00
Oct 4 11:37:26 server4 kernel: Buffer I/O error on device dm-3, logical block 329786
Oct 4 11:37:26 server4 kernel: lost page write due to I/O error on dm-3
Oct 4 11:37:27 server4 kernel: JBD2: Detected IO errors while flushing file data on dm-3-8
Oct 4 11:37:27 server4 kernel: sd 0:1:0:0: [sda] Unhandled error code
Oct 4 11:37:27 server4 kernel: sd 0:1:0:0: [sda] Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
Oct 4 11:37:27 server4 kernel: sd 0:1:0:0: [sda] CDB: Write(10): 2a 00 02 0c 4b e8 00 00 28 00
Oct 4 11:37:27 server4 kernel: end_request: I/O error, dev sda, sector 34360288
Oct 4 11:37:27 server4 kernel: Aborting journal on device dm-3-8.
Oct 4 11:37:27 server4 kernel: EXT4-fs error (device dm-3): ext4_journal_start_sb: Detected aborted journal
Oct 4 11:37:27 server4 kernel: EXT4-fs (dm-3): Remounting filesystem read-only
Oct 4 11:37:27 server4 kernel: end_request: I/O error, dev sda, sector 34344960
Oct 4 11:37:27 server4 kernel: JBD2: I/O error detected when updating journal superblock for dm-3-8.
Oct 4 11:37:27 server4 kernel: end_request: I/O error, dev sda, sector 32526848
Oct 4 11:37:27 server4 kernel: JBD2: Detected IO errors while flushing file data on dm-3-8
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 4989768
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 4989760
Oct 4 11:37:30 server4 kernel: Aborting journal on device dm-0-8.
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 19699272
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 15279104
Oct 4 11:37:30 server4 kernel: JBD2: Detected IO errors while flushing file data on dm-2-8
Oct 4 11:37:30 server4 kernel: Aborting journal on device dm-2-8.
Oct 4 11:37:30 server4 kernel: EXT4-fs error (device dm-2): ext4_journal_start_sb: Detected aborted journal
Oct 4 11:37:30 server4 kernel: EXT4-fs (dm-2): Remounting filesystem read-only
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 4984832
Oct 4 11:37:30 server4 kernel: JBD2: I/O error detected when updating journal superblock for dm-0-8.
Oct 4 11:37:30 server4 kernel: end_request: I/O error, dev sda, sector 19664896
Oct 4 11:37:30 server4 kernel: JBD2: I/O error detected when updating journal superblock for dm-2-8.
Oct 4 11:37:39 server4 ipmievd: Drive Slot sensor Drive 1 - Drive Present Deasserted
Oct 4 11:37:53 server4 kernel: mpt2sas0: _ctl_do_mpt_command: timeout
Oct 4 11:37:53 server4 kernel: mf:
Oct 4 11:37:53 server4 kernel: #011000000b5 00000000 00000000 ffde2e0d 00fc0000 00000018 00000000 000000ff
Oct 4 11:37:53 server4 kernel: #01100000000 00000006 00000000 00000000 00000000 00000000 00000000 02000000
Oct 4 11:37:53 server4 kernel: #01100000012 000000ff 00000000 00000000 00000000 00000000 00000000 00000000
Oct 4 11:37:53 server4 kernel: mpt2sas0: issue target reset: handle = (0x00b5)
Oct 4 11:37:55 server4 kernel: mpt2sas0: removing handle(0x0009), sas_addr(0x5000c5004bd78a15)
Oct 4 11:38:19 server4 ipmievd: Drive Slot sensor Drive 1 - Drive Fault Asserted
Oct 4 11:38:19 server4 ipmievd: Drive Slot sensor Drive 2 - In Critical Array Asserted
Oct 4 11:38:54 server4 kernel: EXT4-fs error (device dm-0): ext4_journal_start_sb: Detected aborted journal
Oct 4 11:38:54 server4 kernel: EXT4-fs (dm-0): Remounting filesystem read-only
Environment
- Red Hat Enterprise Linux 6.3
- IBM BladeCenter HS23p 7875AC1
- LSI SAS2004 RAID 1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.