RHGS volume gets unmounted while copying large data to the volume
Issue
- The gluster volume is getting unmounted automatically
-
The
/var/log/messageshowing glusterfs getting killed by OOM killerJul 21 13:00:18 rhgs-clnt kernel: lowmem_reserve[]: 0 0 5050 5050 Jul 21 13:00:18 rhgs-clnt kernel: Node 0 Normal free:42260kB min:42316kB low:52892kB high:63472kB active_anon:4298296kB inactive_anon:716488kB active_file:348kB inactive_file:960kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:5171200kB mlocked:0kB dirty:0kB writeback:0kB mapped:452kB shmem:36kB slab_reclaimable:11016kB slab_unreclaimable:66248kB kernel_stack:3200kB pagetables:28620kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jul 21 13:00:18 rhgs-clnt kernel: lowmem_reserve[]: 0 0 0 0 Jul 21 13:00:18 rhgs-clnt kernel: Node 0 DMA: 1*4kB 1*8kB 0*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15660kB Jul 21 13:00:18 rhgs-clnt kernel: Node 0 DMA32: 703*4kB 486*8kB 382*16kB 311*32kB 181*64kB 61*128kB 13*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 45484kB Jul 21 13:00:18 rhgs-clnt kernel: Node 0 Normal: 10558*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42232kB Jul 21 13:00:18 rhgs-clnt kernel: 76956 total pagecache pages Jul 21 13:00:18 rhgs-clnt kernel: 76572 pages in swap cache Jul 21 13:00:18 rhgs-clnt kernel: Swap cache stats: add 11613481, delete 11536909, find 19061073/20276356 Jul 21 13:00:18 rhgs-clnt kernel: Free swap = 0kB Jul 21 13:00:18 rhgs-clnt kernel: Total swap = 1048572kB Jul 21 13:00:18 rhgs-clnt kernel: 2097136 pages RAM Jul 21 13:00:18 rhgs-clnt kernel: 81838 pages reserved Jul 21 13:00:18 rhgs-clnt kernel: 3251 pages shared Jul 21 13:00:18 rhgs-clnt kernel: 1984758 pages non-shared Jul 21 13:00:18 rhgs-clnt kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name Jul 21 13:00:18 rhgs-clnt kernel: [ 452] 0 452 2809 1 1 -17 -1000 udevd Jul 21 13:00:18 rhgs-clnt kernel: [ 1145] 0 1145 24346 41 1 -17 -1000 auditd Jul 21 13:00:18 rhgs-clnt kernel: [ 1147] 0 1147 20509 52 0 0 0 audispd Jul 21 13:00:18 rhgs-clnt kernel: [ 1168] 0 1168 113433 1 1 0 0 nslcd Jul 21 13:00:18 rhgs-clnt kernel: [ 1184] 0 1184 64047 788 0 0 0 rsyslogd Jul 21 13:00:18 rhgs-clnt kernel: [ 1219] 0 1219 4575 45 1 0 0 irqbalance Jul 21 13:00:18 rhgs-clnt kernel: [ 1237] 32 1237 4744 17 0 0 0 rpcbind Jul 21 13:00:18 rhgs-clnt kernel: [ 1259] 29 1259 5837 2 0 0 0 rpc.statd Jul 21 13:00:18 rhgs-clnt kernel: [ 1286] 81 1286 8472 2 1 0 0 dbus-daemon Jul 21 13:00:18 rhgs-clnt kernel: [ 1409] 0 1409 1020 1 1 0 0 acpid Jul 21 13:00:18 rhgs-clnt kernel: [ 1421] 68 1421 10533 113 1 0 0 hald Jul 21 13:00:18 rhgs-clnt kernel: [ 1422] 0 1422 5100 2 0 0 0 hald-runner Jul 21 13:00:18 rhgs-clnt kernel: [ 1454] 0 1454 5630 2 1 0 0 hald-addon-inpu Jul 21 13:00:18 rhgs-clnt kernel: [ 1464] 68 1464 4502 2 1 0 0 hald-addon-acpi Jul 21 13:00:18 rhgs-clnt kernel: [ 1479] 0 1479 53189 45 1 0 0 sssd Jul 21 13:00:18 rhgs-clnt kernel: [ 1480] 0 1480 62413 293 1 0 0 sssd_be Jul 21 13:00:18 rhgs-clnt kernel: [ 1481] 0 1481 54575 71 0 0 0 sssd_nss Jul 21 13:00:18 rhgs-clnt kernel: [ 1482] 0 1482 50939 75 1 0 0 sssd_pam Jul 21 13:00:18 rhgs-clnt kernel: [ 1483] 0 1483 52256 60 1 0 0 sssd_ssh Jul 21 13:00:18 rhgs-clnt kernel: [ 1484] 0 1484 50393 33 0 0 0 sssd_sudo Jul 21 13:00:18 rhgs-clnt kernel: [ 1530] 0 1530 26827 1 1 0 0 rpc.rquotad Jul 21 13:00:18 rhgs-clnt kernel: [ 1535] 0 1535 5417 1 1 0 0 rpc.mountd Jul 21 13:00:18 rhgs-clnt kernel: [ 1580] 0 1580 6292 2 0 0 0 rpc.idmapd Jul 21 13:00:18 rhgs-clnt kernel: [ 1869] 0 1869 15964 86 0 0 0 vmtoolsd Jul 21 13:00:18 rhgs-clnt kernel: [ 1909] 0 1909 16558 25 1 -17 -1000 sshd Jul 21 13:00:18 rhgs-clnt kernel: [ 1920] 0 1920 5429 18 0 0 0 xinetd Jul 21 13:00:18 rhgs-clnt kernel: [ 1944] 38 1944 8205 39 0 0 0 ntpd Jul 21 13:00:18 rhgs-clnt kernel: [ 1956] 0 1956 42102 39 1 0 0 vnetd Jul 21 13:00:18 rhgs-clnt kernel: [ 1960] 0 1960 51762 43 1 0 0 bpcd Jul 21 13:00:18 rhgs-clnt kernel: [ 1991] 5076 1991 10363 12 0 0 0 nrpe Jul 21 13:00:18 rhgs-clnt kernel: [ 2012] 0 2012 22737 63 0 0 0 sendmail Jul 21 13:00:18 rhgs-clnt kernel: [ 2022] 51 2022 20081 31 0 0 0 sendmail Jul 21 13:00:18 rhgs-clnt kernel: [ 2050] 0 2050 45255 2 0 0 0 abrtd Jul 21 13:00:18 rhgs-clnt kernel: [ 2060] 0 2060 44631 22 1 0 0 abrt-dump-oops Jul 21 13:00:18 rhgs-clnt kernel: [ 2071] 0 2071 49714 15 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [ 2091] 0 2091 51398 294 0 0 0 osad Jul 21 13:00:18 rhgs-clnt kernel: [ 2103] 0 2103 29216 28 0 0 0 crond Jul 21 13:00:18 rhgs-clnt kernel: [ 2117] 0 2117 25232 27 0 0 0 rhnsd Jul 21 13:00:18 rhgs-clnt kernel: [ 2128] 41 2128 54506 2 1 0 0 mailmanctl Jul 21 13:00:18 rhgs-clnt kernel: [ 2129] 41 2129 83976 15539 1 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2130] 41 2130 98478 12895 0 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2131] 41 2131 59263 1630 0 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2132] 41 2132 76469 14538 0 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2133] 41 2133 54565 615 0 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2134] 41 2134 78888 11628 1 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2135] 41 2135 80235 11255 0 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2136] 41 2136 54544 115 1 0 0 python Jul 21 13:00:18 rhgs-clnt kernel: [ 2148] 0 2148 6285 2 1 0 0 oddjobd Jul 21 13:00:18 rhgs-clnt kernel: [ 2163] 0 2163 1016 2 1 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2165] 0 2165 1016 2 1 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2167] 0 2167 1016 2 1 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2169] 0 2169 1016 2 0 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2171] 0 2171 3105 1 0 -17 -1000 udevd Jul 21 13:00:18 rhgs-clnt kernel: [ 2173] 0 2173 1016 2 1 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2175] 0 2175 1016 2 1 0 0 mingetty Jul 21 13:00:18 rhgs-clnt kernel: [ 2334] 0 2334 32831 2 0 0 0 screen Jul 21 13:00:18 rhgs-clnt kernel: [ 2335] 0 2335 27110 2 1 0 0 bash Jul 21 13:00:18 rhgs-clnt kernel: [ 7905] 0 7905 2808 1 0 -17 -1000 udevd Jul 21 13:00:18 rhgs-clnt kernel: [26396] 48 26396 50366 283 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26397] 48 26397 50308 262 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26398] 48 26398 50367 282 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26399] 48 26399 50303 266 0 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26400] 48 26400 50308 284 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26401] 48 26401 50366 264 0 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26402] 48 26402 50409 278 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [26403] 48 26403 50641 282 0 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [24104] 48 24104 50335 266 0 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [24121] 48 24121 50341 256 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [24268] 48 24268 50308 263 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [25299] 48 25299 50573 283 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [28054] 48 28054 50308 276 0 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [28055] 48 28055 50641 281 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [28056] 48 28056 50308 290 1 0 0 httpd Jul 21 13:00:18 rhgs-clnt kernel: [16587] 0 16587 2042236 1731318 0 0 0 glusterfs Jul 21 13:00:18 rhgs-clnt kernel: [10820] 0 10820 49787 809 1 0 0 bpbkar Jul 21 13:00:18 rhgs-clnt kernel: [ 6667] 41 6667 38651 108 1 0 0 crond Jul 21 13:00:18 rhgs-clnt kernel: [ 6668] 0 6668 38651 108 1 0 0 crond Jul 21 13:00:18 rhgs-clnt kernel: [ 6675] 0 6675 26524 35 1 0 0 sh Jul 21 13:00:18 rhgs-clnt kernel: [ 6677] 41 6677 73232 21481 0 0 0 gate_news Jul 21 13:00:18 rhgs-clnt kernel: [ 6678] 0 6678 90281 38494 0 0 0 genaliases Jul 21 13:00:18 rhgs-clnt kernel: [ 6681] 0 6681 27567 87 0 0 0 check_mk_agent Jul 21 13:00:18 rhgs-clnt kernel: Out of memory: Kill process 16587 (glusterfs) score 798 or sacrifice child Jul 21 13:00:18 rhgs-clnt kernel: Killed process 16587, UID 0, (glusterfs) total-vm:8168944kB, anon-rss:6925160kB, file-rss:124kB
Environment
Red Hat Enterprise Linux 7.2
Red Hat Gluster Storage 3.1.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
