sluggish performance when reaching high values of the ip_conntrack table

Issue

• System performance is slow when configured as firewall and is handling 1200-1500 simultaneous connections

• Following log message is observed and setting ip_conntrack_max to 65536 didn't helped.

ip_conntrack: table full, dropping packet

• System generates following call trace.

kernel: BUG: soft lockup - CPU#1 stuck for 10s! [ksoftirqd/1:6]
kernel: CPU 1:
kernel: Modules linked in: ip_nat_h323 ip_nat_irc ip_nat_snmp_basic ip_nat_sip ip_nat_amanda ip_nat_tftp ip_nat_pptp xt_conntrack ip_conntrack_pptp ip_conntrack_h323 ip_conntrack_sip ip_conntrack_netlink ip_conntrack_amanda ip_conntrack_proto_sctp ip_conntrack_tftp ip_conntrack_irc ip_nat_ftp ip_conntrack_ftp ip_conntrack_netbios_ns iptable_mangle xt_state iptable_filter iptable_nat ip_nat ip_conntrack nfnetlink ip_tables ebt_dnat ebt_snat ebtable_nat ebtables ts_kmp ip_vs i5k_amb coretemp(U) ipmi_devintf ipmi_si ipmi_msghandler hidp rfcomm l2cap bluetooth sunrpc ipt_owner ipt_LOG xt_multiport xt_tcpudp x_tables ipv6 xfrm_nalgo crypto_api dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec i2c_core dell_wmi wmi button battery asus_acpi acpi_memhotplug ac parport_pc lp parport joydev sr_mod cdrom igb 8021q i5000_edac edac_mc bnx2 serio_raw pcspkr sg dca dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata shpchp megaraid
kernel: sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
kernel: Pid: 6, comm: ksoftirqd/1 Tainted: G      2.6.18-194.el5 #1
kernel: RIP: 0010:[<ffffffff80063181>]  [<ffffffff80063181>] __write_lock_failed+0x9/0x20
kernel: RSP: 0018:ffff81010ef37f10  EFLAGS: 00000283
kernel: RAX: ffff81010ef2bfd8 RBX: ffff8102ea1111b0 RCX: ffff81010ef08ee8
kernel: RDX: ffff81010ef37f38 RSI: ffff8104357c59c0 RDI: ffffffff8843d640
kernel: RBP: ffff81010ef37e90 R08: ffffffff80236864 R09: 0000000027c4e60a
kernel: R10: 000000004f571a3d R11: 00000000000000c8 R12: ffffffff8005ec8e
kernel: R13: ffffffff8843d640 R14: ffffffff8007922b R15: ffff81010ef37e90
kernel: FS:  0000000000000000(0000) GS:ffff81010eef7840(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
kernel: CR2: 00000031644c5090 CR3: 0000000000201000 CR4: 00000000000006e0
kernel:
kernel: Call Trace:
kernel:  <IRQ>  [<ffffffff80065bce>] _write_lock_bh+0x1a/0x1c
kernel:  [<ffffffff884310c7>] :ip_conntrack:death_by_timeout+0x10/0x6c
kernel:  [<ffffffff80098c76>] run_timer_softirq+0x193/0x241
kernel:  [<ffffffff80012409>] __do_softirq+0x89/0x133
kernel:  [<ffffffff8005f2fc>] call_softirq+0x1c/0x28
kernel:  <EOI>  [<ffffffff8009609a>] ksoftirqd+0x0/0xbf
kernel:  [<ffffffff8006dba8>] do_softirq+0x2c/0x85
kernel:  [<ffffffff800960f9>] ksoftirqd+0x5f/0xbf
kernel:  [<ffffffff80032bdc>] kthread+0xfe/0x132
kernel:  [<ffffffff8005efb1>] child_rip+0xa/0x11
kernel:  [<ffffffff80032ade>] kthread+0x0/0x132
kernel:  [<ffffffff8005efa7>] child_rip+0x0/0x11