ActiveMQ: clients block when storage filesystem is changed to read-only by the kernel

Issue

In the event of serious filesystem corruption, the Linux kernel might take protective measures by setting the filesystem read-only. If it does this, it will do so even if some processes have open file handles on the filesystem. Of course, these processes will no longer be able to write.

If this change to read-only occurs on a filesystem that is hosting an ActiveMQ KahaDB message store, then the broker will behave in a suboptimal way. In particular, clients of the broker will not be disconnected, and new clients will still be able to connect, even though they will not be able to carry out any messaging operations.

This means that, in a highly-available mesh configuration, clients that might ideally be able to reconnect to a working broker, in the event of a particular broker experiencing a filesystem failure, do not do so. A single failure can, in the worst case, bring down the entire mesh.