Ceph - Multiple OSD's assert with FAILED assert(0 == "unexpected aio error")
Issue
- Multiple OSDs in a Ceph cluster all assert around the same time frame with the same FAILED assert(0 == "unexpected aio error") thread. In the logs (/var/log/ceph) for the asserted OSDs the following stacks can be seen:
-1> 2016-08-24 11:52:10.618075 7fae1f5eb700 -1 journal aio to 4600467456~28672 wrote 18446744073709551611
0> 2016-08-24 11:52:10.619336 7fae1f5eb700 -1 os/FileJournal.cc: In function 'void FileJournal::write_finish_thread_entry()' thread 7fae1f5eb700 time 2016-08-24 11:52:10.618118
os/FileJournal.cc: 1426: FAILED assert(0 == "unexpected aio error")
ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf66e5]
2: (FileJournal::write_finish_thread_entry()+0x695) [0x9cffe5]
3: (FileJournal::WriteFinisher::entry()+0xd) [0x8d730d]
4: (()+0x7df5) [0x7fae2a155df5]
5: (clone()+0x6d) [0x7fae28c381ad]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
- The messages and dmesg logs will show similar I/O errors on the physical device and its partitions.
[21068556.629355] Buffer I/O error on device sdb1, logical block 2563312
[21068556.991689] end_request: I/O error, dev sdb, sector 20508544
[21068556.998911] Buffer I/O error on device sdb1, logical block 2563312
[21068557.361649] end_request: I/O error, dev sdb, sector 41015168
[21068557.368914] Buffer I/O error on device sdb2, logical block 2563312
[21068557.487791] end_request: I/O error, dev sdb, sector 41015168
[21068557.487793] Buffer I/O error on device sdb2, logical block 2563312
[21068557.557865] end_request: I/O error, dev sdb, sector 61523840
[21068557.557866] Buffer I/O error on device sdb3, logical block 2563568
[21068557.623013] end_request: I/O error, dev sdb, sector 61523840
[21068557.623014] Buffer I/O error on device sdb3, logical block 2563568
[21068557.692899] end_request: I/O error, dev sdb, sector 82030464
[21068557.692901] Buffer I/O error on device sdb4, logical block 2563312
[21068557.754902] end_request: I/O error, dev sdb, sector 82030464
[21068557.754903] Buffer I/O error on device sdb4, logical block 2563312
[21068557.821931] end_request: I/O error, dev sdb, sector 102539136
[21068557.821933] Buffer I/O error on device sdb5, logical block 2563568
[21068557.883958] end_request: I/O error, dev sdb, sector 102539136
[21068557.883960] Buffer I/O error on device sdb5, logical block 2563568
Environment
- Red Hat Enterprise Linux 7.x
- Red Hat Ceph Storage 1.3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.