Red Hat Training

A Red Hat training course is available for Red Hat Gluster Storage

Chapter 3. Notable Bug Fixes

This chapter describes bugs fixed in this release of Red Hat Gluster Storage that have significant impact on users.
BZ#1341934
When a file that had a hard link was marked as bad, the bad file marker was not being removed from the inode. This meant that self-heal attempted to open the file in order to heal it, the heal failed with an EIO error, and the hard link was not recovered. The bad file marker on the inode is now removed during lookup so that files with hard links recover successfully.
BZ#1359619
Previously if glusterd took more than 120 seconds to process 'gluster volume status all clients --xml' then the XML output that was returned was incomplete. This could result in gstatus failures when attempting to parse the incomplete result. This update ensures parseable results are returned.
BZ#1414663
The rpc.statd process runs as the rpcuser user by default. Previously, the /var/lib/nfs/statd directory was not owned by the rpcuser user or group. This meant that when the rpc.statd process started before NFS-Ganesha had started, the rpc.statd process was unable to read or write client state to the /var/lib/nfs/statd directory. This directory and its files are now created with rpcuser as the owning user and group so that the rpc.statd process can access and maintain client state.
BZ#1278336
Previously, during virtual IP failover, the TCP packets sent by a client and received by the server may be out of sequence because of previous failures to close the TCP socket. This could result in mount points becoming unresponsive. Portblock resource agents now 'tickle' TCP connections to ensure that packets are in sequence after failover.
BZ#1344675
Export configuration files were stored separately in each node. This could lead to situations where a volume was being exported differently on different nodes, and therefore developed inconsistencies after failing over to another node. Configuration details are now stored in a shared meta volume so that volumes are exported identically by all nodes.
BZ#1348949
NFS-Ganesha now places ganesha.conf, ganesha-ha.conf, and export configuration files in a shared storage directory to make it easier to manage export configuration details.
BZ#1348954
The HA_VOL_SERVER parameter in the ganesha-ha.conf file is no longer used by Red Hat Gluster Storage.
BZ#1333885
When a client attempted to connect using SSL and the connection failed, the client identifier was not part of the log message. The client identifier is now included in the log message to make it easier to determine which client was attempting to connect.
BZ#1340608
Red Hat Gluster Storage now provides support for Samba to enable Transport Layer Security (SSL) on a management connection between the smbd and glusterd services. Libgfapi now checks for the /var/lib/glusterd/secure-access file before making an RPC connection and enables SSL on the management connection if the file is present.
BZ#1347251
Online rolling upgrades were not possible from Red Hat Gluster Storage 3.1.x to 3.1.y (where y is more recent than x) because of client limitations. Red Hat Gluster Storage 3.2 enables online rolling upgrades from 3.2.x to 3.2.y (where y is more recent than x).
BZ#1352805
The thread pool limit for the rebalance process was static and set to 40. This meant that machines with more than 40 cores crashed when the rebalance process attempted to create more than 40 threads and access more memory than was allocated to the stack. The thread pool limit is now dynamic, and is determined based on the number of available cores.
BZ#1286572
Previously, the output of the 'gluster volume rebalance VOLNAME status' command attempted to display status details for fix-layout operations despite this operation not populating the output with details of rebalanced files, scanned files, failures, and size. The unpopulated sections have now been removed so that only status and elapsed time on each node are shown.
BZ#1294035
When a new brick was added, Red Hat Gluster Storage did not propagate permissions on the root directory of a brick during a heal operation. This resulted in errors because of incorrect permissions. This has been corrected so that permissions are correctly propagated.
BZ#1404989
Previously, the 'gluster volume add-brick' command failed on some nodes when a distributed volume was converted into a replicated volume, and the volume was not mounted, and no lookup had been performed. This could result in inconsistent data across gluster nodes. To avoid this situation, the 'gluster volume add-brick' command is no longer allowed when the replica count has increased and there any replica bricks are unavailable.
BZ#1405000
Previously, if the rm command was still running when the remove-brick rebalance operation was triggered, the rebalance operation aborted when it received the resulting ENOENT and ESTALE errors. These errors are now ignored so that the rebalance process can complete as expected in this situation.
BZ#1343320
After a connection failure, the Gluster FUSE client did not correctly clean up threads spawned to handle the failure, which resulted in a memory leak. This caused a crash when the process was killed to reclaim memory. Threads are now cleaned up correctly and the process is no longer killed, preventing the crash.
BZ#1337863
Previously, when SSL was enabled for a Red Hat Gluster Storage volume, thread cleanup code did not always finish running before a new thread with the same ID was generated. When this happened, clients with I/O in progress experienced hung behavior. New threads are now generated after the detach flag is enabled during thread creation, so that these race conditions are avoided, and I/O no longer hangs for clients.
BZ#1400365
The maximum length for the slave user was set to the value of __POSIX_LOGIN_NAME_MAX (9 including the NULL byte). This meant that the glusterd service failed upon geo-replication mountbroker setup when the slave user length was more than 8 characters long. LOGIN_NAME_MAX is now used instead, allowing for up to 256 characters including the NULL byte. This prevents failure of the glusterd service in this situation.
BZ#1412883
Sometimes when a brick crashed, a race condition occurred that caused corruption of extended attributes on the htime file located at BRICKPATH/.glusterfs/changelogs/htime/HTIME.TIMESTAMP. This caused geo-replication to enter a "Faulty" state because of the resulting Changelog Exception. This has been corrected by ensuring that every time the brick processes come up, the extended attributes on the htime file are checked for corruption, and fixed if corruption exists.
BZ#1240333
Concurrent rename and lookup operations on a directory caused both old and new directories to be healed. At the end of the heal operation, both directories existed and had the same GFID. This meant that clients were sometimes unable to access the contents of the directory. The distributed hash table algorithm has been updated so that this issue no longer occurs.
BZ#1383898
Previously, when the gsyncd_template.conf file could not be read by the non-root account used in the geo-replication process, the default values were used instead. Because the default values do not include a complete path for the gluster binary, geo-replication fails. A check has been added to the process so that if the gsyncd_template.conf file cannot be read, an appropriate error is logged.
BZ#1205162
When a geo-replication session was deleted, the sync time attribute on the root directory of the brick was not reset to zero. This meant that when a new geo-replication session was created, the stale sync time attribute caused the sync process to ignore all files created up until the stale sync time, and start syncing from that time. A new reset-sync-time option has been added to the session delete command so that administrators can reset the sync time attribute is to zero if required.
BZ#1340756
Previously, when the rsync command failed with an error, geo-replication attempted to retrieve the error status after the child rsync process was already closed. This caused geo-replication to fail with an elines error. The elines attribute in the error object is now initialized correctly so that this failure does not occur.
BZ#1344826
When an rsync operation is retried, the geo-replication process attempted to clean up GFIDs from the rsync queue that were already unlinked during the previous sync attempt. This resulted in a KeyError. The geo-replication process now checks for the existence of a GFID before attempting to unlink a file and remove it from the rsync queue, preventing this failure.
BZ#1344908
Previously, enabling User Serviceable Snapshots (USS) caused a graph switch, which meant that when data was copied from the .snaps directory to the master volume, the first copy operation is not synchronized to the slave volume/s. The GFID access translator now correctly handles nameless lookups during the graph switch, and all data copied from the .snaps directory is correctly synced from the master volume to the slave volume/s.
BZ#1347625
If geo-replication status was requested after an upgrade but before glusterd was started again, an empty monitor.status was created and the session status was listed as 'Started'. This meant that when glusterd restarted, because monitor.status was empty, a fresh geo-replication session started instead of the previous session being resumed. This has been corrected so that an empty monitor.status results in an error, and geo-replication status is listed as 'Stopped' after an upgrade but before glusterd restarts.
BZ#1364422
If no changelog entries existed for a requested time range, the changelog history API did not return a distinguished error. This meant that applications (e.g geo-rep) using the history API assumed a general failure and re-attempted the operation. This could cause applications to loop. A specific error is now returned in this situation so that the application falls back to another mechanism like 'hybrid crawl' in geo-replication use cases.
BZ#1315544
Previously, when a NFS client unmounted all volumes, Red Hat Gluster Storage Native NFS server freed a structure that was still being used, which resulted in a segmentation fault on the server (use-after-free). The server now does not free the structure while the mount service is available, so the segmentation fault no longer occurs.
BZ#1337811
Previously, when 'showmount' was run, the structure of data passed from the mount protocol meant that the groupnodes defined in the nfs.rpc-auth-allow volume option were handled as a single string, which caused errors when the string of groupnodes was longer than 255 characters. This single string is now handled as a list of strings so that 'showmount' receives the correct number of hostnames.
BZ#1404996
As of Red Hat Gluster Storage 3.2, Gluster Native NFS server is disabled when creating a new volume. Administrators can either use NFS-Ganesha with new volumes, or continue using Native NFS server by enabling it manually. Existing volumes that use Gluster Native NFS are not modified in order to prevent disruptions during upgrade.
BZ#1351949
Previously, when the nominated volfile server in a Red Hat Gluster Storage cluster became unavailable, any configuration changes that would result in changes to the volfile were not passed to clients until the volfile server became available again. This update ensures that when a volfile server becomes unavailable, another server takes over the role of volfile server, so that clients do not need to wait to receive updates from the original volfile server.
BZ#1241314
The enable-shared-storage option always appeared disabled when the 'volume get VOLNAME enable-shared-storage' command was run, regardless of the actual state. This has been corrected so that the enable-shared-state option's state is shown accurately.
BZ#1263090
Previously, glusterd managed its portmap table so that ports that had previously been allocated to one daemon could not be reused by other daemons after the original daemon no longer used it, for example, after a brick was removed. This update ensures that ports can be reused by another daemon after they become available.
BZ#1306120
Previously, the glusterd log file was named etc-glusterfs-glusterd.vol.log. This update changes the file name to glusterd.log. The file is still found in the /var/log/glusterfs directory.
BZ#1336267
When a node is rebooted, each brick has its identity verified as it resumes operation. Previously, this was done with address resolution. However, this meant that when a cluster had a large number of bricks, there were a very large number of address lookups, which created contention and sometimes meant that bricks failed to restart. This update changes the brick verification method to use a brick's UUID rather than its address, which reduces contention and ensures that all brick processes restart after a reboot.
BZ#1340995
Previously, when glusterd was restarted, bricks were started even when server quorum was not met. This update ensures that bricks are stopped if server quorum is no longer met, or if server quorum is disabled, to ensure that bricks in maintenance are not started incorrectly.
BZ#1351732
Previously, when a server node was unavailable, the client details of the bricks on that node were not displayed when the 'gluster volume status VOLNAME clients' command was run. This has been corrected and client details are now displayed as expected.
BZ#1367472
Previously if a cluster was upgraded from Red Hat Gluster Storage 3.0.x to a version greater than or equal to 3.1.1, and any volumes had quotas enabled, attempts to run peer probe were rejected. This update ensures that peer probe can run as expected.
BZ#1379790
If a long running "pre" operation was terminated with a CTRL+C key at the keyboard, the local processes terminated, but remote processes continued to run until they had completed, because the "delete" command context could not affect remote processes. The remote command is now executed by forcing allocation of a terminal to the remote processes executed via ssh. With a terminal attached to the remote process, a local keyboard interrupt is propagated immediately to the remote processes so that they abort operations and terminate as desired by the user.
BZ#1318000
The glusterd service expected that all files in the /var/lib/glusterd/snaps/snapshot name directory were volumes. This meant that when a snapshot was taken of an NFS-Ganesha volume, glusterd interpreted the configuration file as an invalid volume and did not start. glusterd now starts correctly in this situation.
BZ#1374166
A file descriptor leak meant that a file that was removed from a volume exported using NFS-Ganesha was not removed from the underlying storage. The file descriptor leak has been corrected so that files removed from a mounted volume are also removed from the underlying storage.
BZ#1371475
Red Hat Gluster Storage now provides support for NFS-Ganesha to enable Transport Layer Security (SSL) on a management connection between the Ganesha and glusterd services. Libgfapi now checks for the /var/lib/glusterd/secure-access file before making an RPC connection and enables SSL on the management connection if the file is present.
BZ#1392895
Previously, when the pacemaker and corosync packages were updated, the update process did not remove the /etc/cluster/corosync.conf file. This meant that setup of high availability functionality failed after an upgrade. This file is now removed as part of upgrade so that setup succeeds.
BZ#1392299
In some situations, read operations were skipped by the io-cache translator, which led to a hung client mount. This has been corrected so that the client mount process works as expected for read operations.
BZ#1403840
Previously, split-brain resolution commands issued from the command line interface did not work when client-side self-heals were disabled, and returned incorrect output that indicated the file was not in split brain. This update ensures that split-brain resolution commands work regardless of whether client-side heal or the self-heal daemon are enabled.
BZ#1403180
When a directory is healed, the replication module creates the directory on the brick that needs healing, and then marks on the source brick that the newly created directory needs healing. Previously, there was a possibility for race conditions to develop if the source brick became unavailable after the creation of the new directory and before the directory was marked as requiring healing. In this situation, when I/O occurs in the new directory, the healing direction was reversed, leading to deletion of the files on the source brick. The order of operations has been reversed so that marking the source directory now occurs before directory creation. This ensures that race conditions do not develop.
BZ#1417177
Earlier, the split-brain resolution commands would erroneously resolve split-brains if two bricks that blamed each other were available, but a correct source brick was unavailable. This has now been corrected so that split-brain resolution commands will work only when all bricks are available and true split-brain conditions are present.
BZ#1284873
Enumerating large directories on a Samba client issued a large number of file operations to the gluster volume, slowing directory enumeration performance. Gluster caching has been improved to increase performance in this situation.
BZ#1412554
Premature requests to create shards before the .shard directory had been initialized caused virtual machines to pause when bricks were added because directory layout information was not yet available in memory. Shard creation now includes checks at various points to ensure that virtual machines operate normally.
BZ#1361759
Red Hat Gluster Storage now includes expedited demotion functionality to ensure that tiered volumes remain stable and available to clients when the hi-watermark value has been breached and continuing writes threaten to consume the remaining space on the hot tier before the hi-watermark breach event is triggered. Demotions are now triggered at most 5 seconds after a hi-watermark breach event. Administrators can use the cluster.tier-query-limit volume parameter to specify the number of records extracted from the heat database during a single run of the expedited demotion process. Note that there is still potential for the hot tier to become completely full, for example, if the sum of the disk space retained for metadata and the hi-watermark value are greater than the total size of the hot tier.
BZ#1427783
The metadata cache translator has been updated to improve Red Hat Gluster Storage performance when reading small files.
BZ#1388464
The 'gluster volume attach-tier' and 'gluster volume detach-tier' commands are considered deprecated in favor of the new commands, 'gluster volume tier VOLNAME attach' and gluster volume tier VOLNAME detach'. This update includes extra warning message output when the older commands are used in order to give users advance notice of the deprecation and eventual removal of the older commands.
BZ#1200927
It was found that glusterfs-server RPM package would write file with predictable name into world readable /tmp directory. A local attacker could potentially use this flaw to escalate their privileges to root by modifying the shell script during the installation of the glusterfs-server package.
BZ#1328191
Nagios expected and attempted to process performance data even when performance data had not yet been collected. This caused a crash in the Nagios plugin. This update ensures that Nagios checks whether data exists before attempting to process it, and displays an appropriate error message instead of crashing.
BZ#1351749
When Nagios retrieved live status information, this information was limited to a total of 8192 bytes. In clusters with large numbers of volumes, this meant that the data retrieved was not complete and therefore could not be parsed by Nagios. This resulted in a crash of the Nagios plugin. This update increases the allowed length for live status data in order to ensure that a complete result is retrieved and avoid the situation that led to the crash.
BZ#1372691
Previously, old RPMs included a -doc subpackage which was not included in the new RPMs. This -doc subpackage was not removed during an update. With this fix, the -doc subpackage is erased by an update.
BZ#1380695
Incorrect SELinux rules resulted in failed Gluster Native NFS functionality on Red Hat Enterprise Linux 6 based installations of Red Hat Gluster Storage. These rules have now been updated so that Gluster Native NFS works as expected.
BZ#1354661
Previously, SElinux denied binding to socket listener. This caused the ganesha.nfsd fail to start. With this fix the SElinux rules are updated and the issue is resolved.
BZ#1240258
Previously, NFS-ganesha mapped all anonymous users to uid 4294967294. This value is different from the nfsnobody value of 65534. With this fix all the anonymous uid and gid are mapped to nfsnobody by default.
BZ#1327195
On reboot, the NFS-ganesha export configuration for the volume were not copied from the online nodes. Due to this, the configuration for a volume in the NFS-ganesha cluster was out of sync. With this release this issue is fixed.
BZ#1331559
Previously, SELinux blocked the gluster brick processes to create non-regular socket files. Due to this, users were unable to create socket type files on gluster volume. With this fix, SELinux rules have been added to provide relevant permissions to gluster brick process and files of type socket can be created on nfs mount of gluster volumes.
BZ#1338068
Previously, there were few memory leaks while creating and removing files on a volume exported via NFS-Ganesha server. Due to this, NFS-Ganesha server might have gotten OOM killed, depending on the system memory limits. With this fix, the memory leaks issue has been addressed and the server shall no longer become unavailable while creating/removing large number of files.
BZ#1379329
Previously, there was an fd-leak on the file on which lock operations have been performed from a nfs-mount of the volume that is exported via NFS-Ganesha server. When that file is deleted, it was not removed from .glusterfs/unlink folder at the backend consuming memory. With this fix, all the files that are removed from the mount point shall be removed completely from the backend as well.
BZ#1377275
The gluster packages have been upgraded to upstream version (3.8.4), which provides various bug fixes and enhancements over the previous version.