GET 'IPv6 URL' causes 'squid crashes by segfault when it reboots' when IPv6 is disabled

Solution Verified - Updated -

Issue

(1) Category
Defect Report

(2) Abstract
[RHEL6] squid crashes by segfault when it reboots

(3) Symptom
When rebooting, squid crashes as follows.

[squid cachelog]
---
2011/12/19 08:38:52| Preparing for shutdown after 16537 requests
2011/12/19 08:38:52| Waiting 30 seconds for active connections to finish
<snip>
2011/12/19 08:39:23| Shutting down...
2011/12/19 08:39:23| AuthUserHashPointer::removeFromCache: entry in use - not freeing
2011/12/19 08:39:23| AuthUserHashPointer::removeFromCache: entry in use - not freeing
<snip>
2011/12/19 08:39:23| Open FD READ/WRITE  742 Waiting for next request
2011/12/19 08:39:23| Squid Cache (Version 3.1.10): Exiting normally.
FATAL: Received Segment Violation...dying.
2011/12/19 08:39:23| storeDirWriteCleanLogs: Starting...
2011/12/19 08:39:24| Starting Squid Cache version 3.1.10 for x86_64-redhat-linux-gnu...
2011/12/19 08:39:24| Process ID 32064
<snip>     
---

[messages]
---
Dec 19 08:39:23 proxy001 kernel: squid[31978]: segfault at f69 ip  00007f1c83cd3009 sp 00007fffd9b7a770 error 6 in  squid[7f1c83b18000+2ea000]
Dec 19 08:39:24 proxy001 squid[32062]: Squid Parent: child process 32064 started
Dec 19 08:39:25 proxy001 abrt[32056]: saved core dump of pid 31978  (/usr/sbin/squid) to /var/spool/abrt/ccpp-1324251563-31978.new/coredump  (101023744 bytes)
Dec 19 08:39:25 proxy001 abrtd: Directory 'ccpp-1324251563-31978' creation detected
Dec 19 08:39:25 proxy001 squid[5405]: Squid Parent: child process 31978 exited due to signal 11 with status 0
Dec 19 08:39:25 proxy001 abrtd: New crash /var/spool/abrt/ccpp-1324251563-31978, processing
---

The backtrace of the corefile is as follows. 

---
# gdb /usr/sbin/squid coredump
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-48.el6)
<snip>
Core was generated by `(squid) -f /etc/squid/squid_auth.conf'.
Program terminated with signal 11, Segmentation fault.
<snip>
(gdb) bt
#0  0x00007f1c83cd3009 in commSetCloseOnExec (fd=<value optimized out>) at comm.cc:1886
#1  0x00007f1c83c262a9 in file_open (path=0x7f1c85086f10  "/var/spool/squid_auth_cache/swap.state.clean", mode=<value optimized  out>)
at disk.cc:81
#2  0x00007f1c83d09617 in UFSSwapDir::writeCleanStart (this=0x7f1c84da7f70) at ufs/store_dir_ufs.cc:871
#3  0x00007f1c83cb3a8d in storeDirWriteCleanLogs (reopen=0) at store_dir.cc:450
#4  0x00007f1c83cc0961 in death (sig=<value optimized out>) at tools.cc:378
#5  <signal handler called>
#6  comm_remove_close_handler (fd=157, call=...) at comm.cc:1748
#7  0x00007f1c83cd454c in DeferredReadManager::popHead (deferredReads=...) at comm.cc:2604
#8  0x00007f1c83cd4673 in DeferredReadManager::flushReads (this=0x7f1c89f4b3d8) at comm.cc:2643
#9  0x00007f1c83cd472d in DeferredReadManager::~DeferredReadManager  (this=0x7f1c89f4b3d8, __in_chrg=<value optimized out>) at  comm.cc:2559
#10 0x00007f1c83c82de3 in MemObject::~MemObject (this=0x7f1c89f4b2f0, __in_chrg=<value optimized out>) at MemObject.cc:129
#11 0x00007f1c83caaeab in StoreEntry::destroyMemObject (this=0x7f1c87892690) at store.cc:376
#12 0x00007f1c83cab71e in destroyStoreEntry (data=<value optimized out>) at store.cc:389
#13 0x00007f1c83d67a0c in hashFreeItems (hid=<value optimized  out>, free_func=0x7f1c83cab6d0 <destroyStoreEntry(void*)>) at  hash.c:308
#14 0x00007f1c83cb13dd in StoreHashIndex::~StoreHashIndex (this=0x7f1c84da23f0, __in_chrg=<value optimized out>,
__vtt_parm=<value optimized out>) at store_dir.cc:723
#15 0x00007f1c83cb1b79 in StoreHashIndex::~StoreHashIndex (this=0x7f1c84da23f0, __in_chrg=<value optimized out>,
__vtt_parm=<value optimized out>) at store_dir.cc:727
#16 0x00007f1c83cb1a02 in dereference (this=0x7f1c87ecd968,  __in_chrg=<value optimized out>, \__vtt_parm=<value optimized  out>)
at ../include/RefCount.h:102
#17 ~RefCount (this=0x7f1c87ecd968, __in_chrg=<value optimized  out>, \__vtt_parm=<value optimized out>) at  ../include/RefCount.h:54
#18 StoreSearchHashIndex::~StoreSearchHashIndex  (this=0x7f1c87ecd968, __in_chrg=<value optimized out>,  \__vtt_parm=<value optimized out>)
at store_dir.cc:918
#19 0x00007f1c83cb1a99 in  StoreSearchHashIndex::~StoreSearchHashIndex (this=0x7f1c87ecd968,  __in_chrg=<value optimized out>,
__vtt_parm=<value optimized out>) at store_dir.cc:918
#20 0x00007f1c8140ee12 in exit () from /lib64/libc.so.6
#21 0x00007f1c83c7d3d6 in SquidShutdown () at main.cc:1828
#22 0x00007f1c83c7ea35 in SquidMain (argc=<value optimized out>, argv=<value optimized out>) at main.cc:1405
#23 0x00007f1c83c7eff6 in SquidMainSafe (argc=<value optimized out>, argv=<value optimized out>) at main.cc:1159
#24 main (argc=<value optimized out>, argv=<value optimized out>) at main.cc:1151
(gdb)
---

(4) Environment
[squid server]
OS : RHEL6(x86_64)
kernel : 2.6.32-131.21.1.el6.x86_64
squid : squid-3.1.10-1.el6_1.1

[ldap server]
OS : Windows 2003 (Active Directory)

(5) Recreation Steps
The custonmer could not reproduce it yet though we tried the reproduction test in our environment. However, the phenomenon occurs several times in the environment of the customer.

(6) Investigation
There is a report in the following sites. 

http://bugs.squid-cache.org/show_bug.cgi?id=1837

It might be the same phenomenon. 
The patch is posted but it is not applied to squid.

---
(gdb) bt
#0  0x080e904b in commSetCloseOnExec (fd=7) at comm.cc:1783
#1  0x080835cd in file_open (path=0x8231500
"/usr/local/squid/var/cache/swap.state.clean", mode=577) at disk.cc:83
#2  0x080f317d in UFSSwapDir::writeCleanStart (this=0x822c840) at
ufs/store_dir_ufs.cc:870
#3  0x080d6a86 in storeDirWriteCleanLogs (reopen=0) at store_dir.cc:439
#4  0x080dd606 in fatal (message=0x81c8060 "authparam basic program
/usr/local/bin/blah: (2) No such file or directory") at tools.cc:461
#5  0x080dda63 in fatalf (fmt=0x810a2d9 "%s %s: %s") at tools.cc:489
#6  0x08061ab1 in requirePathnameExists (name=0x812e73b "authparam basic
program", path=0x822d0c8 "/usr/local/bin/blah") at cache_cf.cc:3042
#7  0x08067fa7 in parse_line (buff=<value optimized out>) at cache_cf.cc:1308
#8  0x0806ab45 in parseConfigFile (file_name=0x8212788
"/usr/local/squid/etc/squid.conf", manager=@0x81a2640) at cache_cf.cc:307
#9  0x080b591d in main (argc=2, argv=0xbf8470a4) at main.cc:1219
----

(7) Related Documentation/Related Bugzilla #

http://bugs.squid-cache.org/show_bug.cgi?id=1837
https://bugzilla.redhat.com/show_bug.cgi?id=744355

(8) Attachments
data.tar.gz

sosreport
corefile : coredump_proxy001.tar.gz
squid.conf : squid_auth.conf,whitelist
squid cachelog : cache_auth.log-20111220

(9) Business Impacts

The SIGSEGV often occurs in the customer's system.
The customer has to check whether the system is normal when SIGSEGV occurs.
It is an unnecessary burden for the customer.

(9) Requests

To fix the issue and release errata.

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • kernel-2.6.32-131.21.1.el6
  • squid-3.1.10-1.el6_1.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.