Why File is getting corrupted after adding RHS nodes in Red Hat Gluster Storage 2.1 ?

Issue

• Why File is getting corrupted after adding RHS nodes in Red Hat Gluster Storage 2.1 ?

• Experienced file corruption when reading files, but not all files and only files residing in a single directory

~ Snippet from gluster client logs,

[2015-02-09 17:51:50.427542] I [dht-common.c:623:dht_revalidate_cbk] 0-glashart-dht: mismatching layouts for /npvr
[2015-02-09 17:51:51.790093] I [dht-layout.c:726:dht_layout_dir_mismatch] 0-glashart-dht: /npvr - disk layout missing
[2015-02-09 17:51:51.790161] I [dht-common.c:623:dht_revalidate_cbk] 0-glashart-dht: mismatching layouts for /npvr
[2015-02-09 17:51:51.790276] I [dht-layout.c:726:dht_layout_dir_mismatch] 0-glashart-dht: /npvr - disk layout missing
[2015-02-09 17:51:51.790309] I [dht-common.c:623:dht_revalidate_cbk] 0-glashart-dht: mismatching layouts for /npvr
[2015-02-09 17:51:52.347564] I [dht-common.c:1017:dht_lookup_everywhere_cbk] 0-glashart-dht: deleting stale linkfile /videos7/foxsp1000900.mpg on glashart-replicate-11
[2015-02-09 17:51:52.348424] W [client-rpc-fops.c:645:client3_3_unlink_cbk] 0-glashart-client-22: remote operation failed: Permission denied
[2015-02-09 17:51:52.348582] W [client-rpc-fops.c:645:client3_3_unlink_cbk] 0-glashart-client-23: remote operation failed: Permission denied
[2015-02-09 17:51:52.349403] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-glashart-client-20: remote operation failed: Permission denied. Path: /videos7/foxsp1000900.mpg (00000000-0000-0000-0000-000000000000)
[2015-02-09 17:51:52.349473] W [client-rpc-fops.c:259:client3_3_mknod_cbk] 0-glashart-client-21: remote operation failed: Permission denied. Path: /videos7/foxsp1000900.mpg (00000000-0000-0000-0000-000000000000)
[2015-02-09 17:51:52.349883] E [dht-helper.c:1052:dht_inode_ctx_get] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+
0x75) [0x7f223a55c705] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7f223a54304e] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_inode_ctx_layout_set+0x34) [0x7f223a544394]))) 0-glashart-dht: invalid argument: inode
[2015-02-09 17:51:52.349942] E [dht-helper.c:1071:dht_inode_ctx_set] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_lookup_linkfile_create_cbk+
0x75) [0x7f223a55c705] (-->/usr/lib64/glusterfs/3.4.2/xlator/cluster/distribute.so(dht_layout_preset+0x5e) [0x7f223a54304e] (-->/usr/lib64/glusterfs/3.4.2/xlato
r/cluster/distribute.so(dht_inode_ctx_layout_set+0x52) [0x7f223a5443b2]))) 0-glashart-dht: invalid argument: inode

[2015-02-09 13:57:08.659344] E [server-rpc-fops.c:1111:server_unlink_cbk] 0-glashart-server: 97089: UNLINK /videos7/kidstv1003491.mpg (ec7dea99-820e-4d37-a1ec-5130ae31714f/kidstv1003491.mpg) ==> (Permission denied)

• Cannot start a rebalance fix-layout, displaying the following message,

volume rebalance: glashart: failed: Volume glashart has one or more connected clients of a version lower than RHS-2.1 update 5. Starting rebalance in this state could lead to data loss.
Please disconnect those clients before attempting this command again.