-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for Red Hat Gluster Storage
Chapter 3. Notable Bug Fixes
This chapter describes bugs fixed in this release of Red Hat Gluster Storage that have significant impact on users.
- BZ#1341934
- When a file that had a hard link was marked as bad, the bad file marker was not being removed from the inode. This meant that self-heal attempted to open the file in order to heal it, the heal failed with an EIO error, and the hard link was not recovered. The bad file marker on the inode is now removed during lookup so that files with hard links recover successfully.
- BZ#1359619
- Previously if glusterd took more than 120 seconds to process 'gluster volume status all clients --xml' then the XML output that was returned was incomplete. This could result in gstatus failures when attempting to parse the incomplete result. This update ensures parseable results are returned.
- BZ#1414663
- The rpc.statd process runs as the rpcuser user by default. Previously, the /var/lib/nfs/statd directory was not owned by the rpcuser user or group. This meant that when the rpc.statd process started before NFS-Ganesha had started, the rpc.statd process was unable to read or write client state to the /var/lib/nfs/statd directory. This directory and its files are now created with rpcuser as the owning user and group so that the rpc.statd process can access and maintain client state.
- BZ#1278336
- Previously, during virtual IP failover, the TCP packets sent by a client and received by the server may be out of sequence because of previous failures to close the TCP socket. This could result in mount points becoming unresponsive. Portblock resource agents now 'tickle' TCP connections to ensure that packets are in sequence after failover.
- BZ#1344675
- Export configuration files were stored separately in each node. This could lead to situations where a volume was being exported differently on different nodes, and therefore developed inconsistencies after failing over to another node. Configuration details are now stored in a shared meta volume so that volumes are exported identically by all nodes.
- BZ#1348949
- NFS-Ganesha now places ganesha.conf, ganesha-ha.conf, and export configuration files in a shared storage directory to make it easier to manage export configuration details.
- BZ#1348954
- The HA_VOL_SERVER parameter in the ganesha-ha.conf file is no longer used by Red Hat Gluster Storage.
- BZ#1333885
- When a client attempted to connect using SSL and the connection failed, the client identifier was not part of the log message. The client identifier is now included in the log message to make it easier to determine which client was attempting to connect.
- BZ#1340608
- Red Hat Gluster Storage now provides support for Samba to enable Transport Layer Security (SSL) on a management connection between the smbd and glusterd services. Libgfapi now checks for the /var/lib/glusterd/secure-access file before making an RPC connection and enables SSL on the management connection if the file is present.
- BZ#1347251
- Online rolling upgrades were not possible from Red Hat Gluster Storage 3.1.x to 3.1.y (where y is more recent than x) because of client limitations. Red Hat Gluster Storage 3.2 enables online rolling upgrades from 3.2.x to 3.2.y (where y is more recent than x).
- BZ#1352805
- The thread pool limit for the rebalance process was static and set to 40. This meant that machines with more than 40 cores crashed when the rebalance process attempted to create more than 40 threads and access more memory than was allocated to the stack. The thread pool limit is now dynamic, and is determined based on the number of available cores.
- BZ#1286572
- Previously, the output of the 'gluster volume rebalance VOLNAME status' command attempted to display status details for fix-layout operations despite this operation not populating the output with details of rebalanced files, scanned files, failures, and size. The unpopulated sections have now been removed so that only status and elapsed time on each node are shown.
- BZ#1294035
- When a new brick was added, Red Hat Gluster Storage did not propagate permissions on the root directory of a brick during a heal operation. This resulted in errors because of incorrect permissions. This has been corrected so that permissions are correctly propagated.
- BZ#1404989
- Previously, the 'gluster volume add-brick' command failed on some nodes when a distributed volume was converted into a replicated volume, and the volume was not mounted, and no lookup had been performed. This could result in inconsistent data across gluster nodes. To avoid this situation, the 'gluster volume add-brick' command is no longer allowed when the replica count has increased and there any replica bricks are unavailable.
- BZ#1405000
- Previously, if the rm command was still running when the remove-brick rebalance operation was triggered, the rebalance operation aborted when it received the resulting ENOENT and ESTALE errors. These errors are now ignored so that the rebalance process can complete as expected in this situation.
- BZ#1343320
- After a connection failure, the Gluster FUSE client did not correctly clean up threads spawned to handle the failure, which resulted in a memory leak. This caused a crash when the process was killed to reclaim memory. Threads are now cleaned up correctly and the process is no longer killed, preventing the crash.
- BZ#1337863
- Previously, when SSL was enabled for a Red Hat Gluster Storage volume, thread cleanup code did not always finish running before a new thread with the same ID was generated. When this happened, clients with I/O in progress experienced hung behavior. New threads are now generated after the detach flag is enabled during thread creation, so that these race conditions are avoided, and I/O no longer hangs for clients.
- BZ#1400365
- The maximum length for the slave user was set to the value of __POSIX_LOGIN_NAME_MAX (9 including the NULL byte). This meant that the glusterd service failed upon geo-replication mountbroker setup when the slave user length was more than 8 characters long. LOGIN_NAME_MAX is now used instead, allowing for up to 256 characters including the NULL byte. This prevents failure of the glusterd service in this situation.
- BZ#1412883
- Sometimes when a brick crashed, a race condition occurred that caused corruption of extended attributes on the htime file located at BRICKPATH/.glusterfs/changelogs/htime/HTIME.TIMESTAMP. This caused geo-replication to enter a "Faulty" state because of the resulting Changelog Exception. This has been corrected by ensuring that every time the brick processes come up, the extended attributes on the htime file are checked for corruption, and fixed if corruption exists.
- BZ#1240333
- Concurrent rename and lookup operations on a directory caused both old and new directories to be healed. At the end of the heal operation, both directories existed and had the same GFID. This meant that clients were sometimes unable to access the contents of the directory. The distributed hash table algorithm has been updated so that this issue no longer occurs.
- BZ#1383898
- Previously, when the gsyncd_template.conf file could not be read by the non-root account used in the geo-replication process, the default values were used instead. Because the default values do not include a complete path for the gluster binary, geo-replication fails. A check has been added to the process so that if the gsyncd_template.conf file cannot be read, an appropriate error is logged.
- BZ#1205162
- When a geo-replication session was deleted, the sync time attribute on the root directory of the brick was not reset to zero. This meant that when a new geo-replication session was created, the stale sync time attribute caused the sync process to ignore all files created up until the stale sync time, and start syncing from that time. A new reset-sync-time option has been added to the session delete command so that administrators can reset the sync time attribute is to zero if required.
- BZ#1340756
- Previously, when the rsync command failed with an error, geo-replication attempted to retrieve the error status after the child rsync process was already closed. This caused geo-replication to fail with an elines error. The elines attribute in the error object is now initialized correctly so that this failure does not occur.
- BZ#1344826
- When an rsync operation is retried, the geo-replication process attempted to clean up GFIDs from the rsync queue that were already unlinked during the previous sync attempt. This resulted in a KeyError. The geo-replication process now checks for the existence of a GFID before attempting to unlink a file and remove it from the rsync queue, preventing this failure.
- BZ#1344908
- Previously, enabling User Serviceable Snapshots (USS) caused a graph switch, which meant that when data was copied from the .snaps directory to the master volume, the first copy operation is not synchronized to the slave volume/s. The GFID access translator now correctly handles nameless lookups during the graph switch, and all data copied from the .snaps directory is correctly synced from the master volume to the slave volume/s.
- BZ#1347625
- If geo-replication status was requested after an upgrade but before glusterd was started again, an empty monitor.status was created and the session status was listed as 'Started'. This meant that when glusterd restarted, because monitor.status was empty, a fresh geo-replication session started instead of the previous session being resumed. This has been corrected so that an empty monitor.status results in an error, and geo-replication status is listed as 'Stopped' after an upgrade but before glusterd restarts.
- BZ#1364422
- If no changelog entries existed for a requested time range, the changelog history API did not return a distinguished error. This meant that applications (e.g geo-rep) using the history API assumed a general failure and re-attempted the operation. This could cause applications to loop. A specific error is now returned in this situation so that the application falls back to another mechanism like 'hybrid crawl' in geo-replication use cases.
- BZ#1315544
- Previously, when a NFS client unmounted all volumes, Red Hat Gluster Storage Native NFS server freed a structure that was still being used, which resulted in a segmentation fault on the server (use-after-free). The server now does not free the structure while the mount service is available, so the segmentation fault no longer occurs.
- BZ#1337811
- Previously, when 'showmount' was run, the structure of data passed from the mount protocol meant that the groupnodes defined in the nfs.rpc-auth-allow volume option were handled as a single string, which caused errors when the string of groupnodes was longer than 255 characters. This single string is now handled as a list of strings so that 'showmount' receives the correct number of hostnames.
- BZ#1404996
- As of Red Hat Gluster Storage 3.2, Gluster Native NFS server is disabled when creating a new volume. Administrators can either use NFS-Ganesha with new volumes, or continue using Native NFS server by enabling it manually. Existing volumes that use Gluster Native NFS are not modified in order to prevent disruptions during upgrade.
- BZ#1351949
- Previously, when the nominated volfile server in a Red Hat Gluster Storage cluster became unavailable, any configuration changes that would result in changes to the volfile were not passed to clients until the volfile server became available again. This update ensures that when a volfile server becomes unavailable, another server takes over the role of volfile server, so that clients do not need to wait to receive updates from the original volfile server.
- BZ#1241314
- The enable-shared-storage option always appeared disabled when the 'volume get VOLNAME enable-shared-storage' command was run, regardless of the actual state. This has been corrected so that the enable-shared-state option's state is shown accurately.
- BZ#1263090
- Previously, glusterd managed its portmap table so that ports that had previously been allocated to one daemon could not be reused by other daemons after the original daemon no longer used it, for example, after a brick was removed. This update ensures that ports can be reused by another daemon after they become available.
- BZ#1306120
- Previously, the glusterd log file was named etc-glusterfs-glusterd.vol.log. This update changes the file name to glusterd.log. The file is still found in the /var/log/glusterfs directory.
- BZ#1336267
- When a node is rebooted, each brick has its identity verified as it resumes operation. Previously, this was done with address resolution. However, this meant that when a cluster had a large number of bricks, there were a very large number of address lookups, which created contention and sometimes meant that bricks failed to restart. This update changes the brick verification method to use a brick's UUID rather than its address, which reduces contention and ensures that all brick processes restart after a reboot.
- BZ#1340995
- Previously, when glusterd was restarted, bricks were started even when server quorum was not met. This update ensures that bricks are stopped if server quorum is no longer met, or if server quorum is disabled, to ensure that bricks in maintenance are not started incorrectly.
- BZ#1351732
- Previously, when a server node was unavailable, the client details of the bricks on that node were not displayed when the 'gluster volume status VOLNAME clients' command was run. This has been corrected and client details are now displayed as expected.
- BZ#1367472
- Previously if a cluster was upgraded from Red Hat Gluster Storage 3.0.x to a version greater than or equal to 3.1.1, and any volumes had quotas enabled, attempts to run peer probe were rejected. This update ensures that peer probe can run as expected.
- BZ#1379790
- If a long running "pre" operation was terminated with a CTRL+C key at the keyboard, the local processes terminated, but remote processes continued to run until they had completed, because the "delete" command context could not affect remote processes. The remote command is now executed by forcing allocation of a terminal to the remote processes executed via ssh. With a terminal attached to the remote process, a local keyboard interrupt is propagated immediately to the remote processes so that they abort operations and terminate as desired by the user.
- BZ#1318000
- The glusterd service expected that all files in the /var/lib/glusterd/snaps/snapshot name directory were volumes. This meant that when a snapshot was taken of an NFS-Ganesha volume, glusterd interpreted the configuration file as an invalid volume and did not start. glusterd now starts correctly in this situation.
- BZ#1374166
- A file descriptor leak meant that a file that was removed from a volume exported using NFS-Ganesha was not removed from the underlying storage. The file descriptor leak has been corrected so that files removed from a mounted volume are also removed from the underlying storage.
- BZ#1371475
- Red Hat Gluster Storage now provides support for NFS-Ganesha to enable Transport Layer Security (SSL) on a management connection between the Ganesha and glusterd services. Libgfapi now checks for the /var/lib/glusterd/secure-access file before making an RPC connection and enables SSL on the management connection if the file is present.
- BZ#1392895
- Previously, when the pacemaker and corosync packages were updated, the update process did not remove the /etc/cluster/corosync.conf file. This meant that setup of high availability functionality failed after an upgrade. This file is now removed as part of upgrade so that setup succeeds.
- BZ#1392299
- In some situations, read operations were skipped by the io-cache translator, which led to a hung client mount. This has been corrected so that the client mount process works as expected for read operations.
- BZ#1403840
- Previously, split-brain resolution commands issued from the command line interface did not work when client-side self-heals were disabled, and returned incorrect output that indicated the file was not in split brain. This update ensures that split-brain resolution commands work regardless of whether client-side heal or the self-heal daemon are enabled.
- BZ#1403180
- When a directory is healed, the replication module creates the directory on the brick that needs healing, and then marks on the source brick that the newly created directory needs healing. Previously, there was a possibility for race conditions to develop if the source brick became unavailable after the creation of the new directory and before the directory was marked as requiring healing. In this situation, when I/O occurs in the new directory, the healing direction was reversed, leading to deletion of the files on the source brick. The order of operations has been reversed so that marking the source directory now occurs before directory creation. This ensures that race conditions do not develop.
- BZ#1417177
- Earlier, the split-brain resolution commands would erroneously resolve split-brains if two bricks that blamed each other were available, but a correct source brick was unavailable. This has now been corrected so that split-brain resolution commands will work only when all bricks are available and true split-brain conditions are present.
- BZ#1284873
- Enumerating large directories on a Samba client issued a large number of file operations to the gluster volume, slowing directory enumeration performance. Gluster caching has been improved to increase performance in this situation.
- BZ#1412554
- Premature requests to create shards before the .shard directory had been initialized caused virtual machines to pause when bricks were added because directory layout information was not yet available in memory. Shard creation now includes checks at various points to ensure that virtual machines operate normally.
- BZ#1361759
- Red Hat Gluster Storage now includes expedited demotion functionality to ensure that tiered volumes remain stable and available to clients when the hi-watermark value has been breached and continuing writes threaten to consume the remaining space on the hot tier before the hi-watermark breach event is triggered. Demotions are now triggered at most 5 seconds after a hi-watermark breach event. Administrators can use the cluster.tier-query-limit volume parameter to specify the number of records extracted from the heat database during a single run of the expedited demotion process. Note that there is still potential for the hot tier to become completely full, for example, if the sum of the disk space retained for metadata and the hi-watermark value are greater than the total size of the hot tier.
- BZ#1427783
- The metadata cache translator has been updated to improve Red Hat Gluster Storage performance when reading small files.
- BZ#1388464
- The 'gluster volume attach-tier' and 'gluster volume detach-tier' commands are considered deprecated in favor of the new commands, 'gluster volume tier VOLNAME attach' and gluster volume tier VOLNAME detach'. This update includes extra warning message output when the older commands are used in order to give users advance notice of the deprecation and eventual removal of the older commands.
- BZ#1200927
- It was found that glusterfs-server RPM package would write file with predictable name into world readable /tmp directory. A local attacker could potentially use this flaw to escalate their privileges to root by modifying the shell script during the installation of the glusterfs-server package.
- BZ#1328191
- Nagios expected and attempted to process performance data even when performance data had not yet been collected. This caused a crash in the Nagios plugin. This update ensures that Nagios checks whether data exists before attempting to process it, and displays an appropriate error message instead of crashing.
- BZ#1351749
- When Nagios retrieved live status information, this information was limited to a total of 8192 bytes. In clusters with large numbers of volumes, this meant that the data retrieved was not complete and therefore could not be parsed by Nagios. This resulted in a crash of the Nagios plugin. This update increases the allowed length for live status data in order to ensure that a complete result is retrieved and avoid the situation that led to the crash.
- BZ#1372691
- Previously, old RPMs included a -doc subpackage which was not included in the new RPMs. This -doc subpackage was not removed during an update. With this fix, the -doc subpackage is erased by an update.
- BZ#1380695
- Incorrect SELinux rules resulted in failed Gluster Native NFS functionality on Red Hat Enterprise Linux 6 based installations of Red Hat Gluster Storage. These rules have now been updated so that Gluster Native NFS works as expected.
- BZ#1354661
- Previously, SElinux denied binding to socket listener. This caused the ganesha.nfsd fail to start. With this fix the SElinux rules are updated and the issue is resolved.
- BZ#1240258
- Previously, NFS-ganesha mapped all anonymous users to uid 4294967294. This value is different from the nfsnobody value of 65534. With this fix all the anonymous uid and gid are mapped to nfsnobody by default.
- BZ#1327195
- On reboot, the NFS-ganesha export configuration for the volume were not copied from the online nodes. Due to this, the configuration for a volume in the NFS-ganesha cluster was out of sync. With this release this issue is fixed.
- BZ#1331559
- Previously, SELinux blocked the gluster brick processes to create non-regular socket files. Due to this, users were unable to create socket type files on gluster volume. With this fix, SELinux rules have been added to provide relevant permissions to gluster brick process and files of type socket can be created on nfs mount of gluster volumes.
- BZ#1338068
- Previously, there were few memory leaks while creating and removing files on a volume exported via NFS-Ganesha server. Due to this, NFS-Ganesha server might have gotten OOM killed, depending on the system memory limits. With this fix, the memory leaks issue has been addressed and the server shall no longer become unavailable while creating/removing large number of files.
- BZ#1379329
- Previously, there was an fd-leak on the file on which lock operations have been performed from a nfs-mount of the volume that is exported via NFS-Ganesha server. When that file is deleted, it was not removed from .glusterfs/unlink folder at the backend consuming memory. With this fix, all the files that are removed from the mount point shall be removed completely from the backend as well.
- BZ#1377275
- The gluster packages have been upgraded to upstream version (3.8.4), which provides various bug fixes and enhancements over the previous version.