Ceph - Multiple OSD's assert with FAILED assert(0 == "unexpected aio error")

Solution Verified - Updated -

Issue

  • Multiple OSDs in a Ceph cluster all assert around the same time frame with the same FAILED assert(0 == "unexpected aio error") thread. In the logs (/var/log/ceph) for the asserted OSDs the following stacks can be seen:
   -1> 2016-08-24 11:52:10.618075 7fae1f5eb700 -1 journal aio to 4600467456~28672 wrote 18446744073709551611
     0> 2016-08-24 11:52:10.619336 7fae1f5eb700 -1 os/FileJournal.cc: In function 'void FileJournal::write_finish_thread_entry()' thread 7fae1f5eb700 time 2016-08-24 11:52:10.618118
os/FileJournal.cc: 1426: FAILED assert(0 == "unexpected aio error")

 ceph version 0.94.1 (e4bfad3a3c51054df7e537a724c8d0bf9be972ff)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xaf66e5]
 2: (FileJournal::write_finish_thread_entry()+0x695) [0x9cffe5]
 3: (FileJournal::WriteFinisher::entry()+0xd) [0x8d730d]
 4: (()+0x7df5) [0x7fae2a155df5]
 5: (clone()+0x6d) [0x7fae28c381ad]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
  • The messages and dmesg logs will show similar I/O errors on the physical device and its partitions.
[21068556.629355] Buffer I/O error on device sdb1, logical block 2563312
[21068556.991689] end_request: I/O error, dev sdb, sector 20508544
[21068556.998911] Buffer I/O error on device sdb1, logical block 2563312
[21068557.361649] end_request: I/O error, dev sdb, sector 41015168
[21068557.368914] Buffer I/O error on device sdb2, logical block 2563312
[21068557.487791] end_request: I/O error, dev sdb, sector 41015168
[21068557.487793] Buffer I/O error on device sdb2, logical block 2563312
[21068557.557865] end_request: I/O error, dev sdb, sector 61523840
[21068557.557866] Buffer I/O error on device sdb3, logical block 2563568
[21068557.623013] end_request: I/O error, dev sdb, sector 61523840
[21068557.623014] Buffer I/O error on device sdb3, logical block 2563568
[21068557.692899] end_request: I/O error, dev sdb, sector 82030464
[21068557.692901] Buffer I/O error on device sdb4, logical block 2563312
[21068557.754902] end_request: I/O error, dev sdb, sector 82030464
[21068557.754903] Buffer I/O error on device sdb4, logical block 2563312
[21068557.821931] end_request: I/O error, dev sdb, sector 102539136
[21068557.821933] Buffer I/O error on device sdb5, logical block 2563568
[21068557.883958] end_request: I/O error, dev sdb, sector 102539136
[21068557.883960] Buffer I/O error on device sdb5, logical block 2563568

Environment

  • Red Hat Enterprise Linux 7.x
  • Red Hat Ceph Storage 1.3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content