IPA server hangs indefinately after upgrading from ipa-4.2 to 4.4

Issue

IPA server hangs indefinately after upgrading from ipa-4.2 to 4.4

ipactl status commands hangs. ldapsearch commands just hangs as well and does not return any results.

pstack shows cos plugin cannot get the mutex lock that it needs.

#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f5345e13d68 in _L_lock_975 () from /lib64/libpthread.so.0
#2  0x00007f5345e13d11 in __GI___pthread_mutex_lock (mutex=0x7f5348ec8630) at pthread_mutex_lock.c:104
#3  0x00007f534646be39 in PR_Lock () from /lib64/libnspr4.so
#4  0x00007f53482b9f7a in slapi_lock_mutex (mutex=<optimized out>) at ldap/servers/slapd/slapi2nspr.c:72
#5  0x00007f533d74ff04 in cos_cache_change_notify (pb=pb@entry=0x7f52e401e770) at ldap/servers/plugins/cos/cos_cache.c:3372
#6  0x00007f533d74abfd in cos_post_op (pb=0x7f52e401e770) at ldap/servers/plugins/cos/cos.c:265
#7  0x00007f5348296d38 in plugin_call_func (list=0x7f5348b6d700, operation=operation@entry=522, pb=pb@entry=0x7f52e401e770, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#8  0x00007f5348296fc3 in plugin_call_list (pb=0x7f52e401e770, operation=522, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#9  plugin_call_plugins (pb=pb@entry=0x7f52e401e770, whichfunction=whichfunction@entry=522) at ldap/servers/slapd/plugin.c:445
#10 0x00007f5348286857 in op_shared_rename (pb=pb@entry=0x7f52e401e770, passin_args=0) at ldap/servers/slapd/modrdn.c:659
#11 0x00007f53482869ba in rename_internal_pb (pb=pb@entry=0x7f52e401e770) at ldap/servers/slapd/modrdn.c:367
#12 0x00007f534828713a in slapi_modrdn_internal_pb (pb=pb@entry=0x7f52e401e770) at ldap/servers/slapd/modrdn.c:299
#13 0x00007f5338773783 in urp_fixup_rename_entry (entry=entry@entry=0x7f52e400dd00, newrdn=0x7f52e401e630 "cn=Default Password Policy+nsuniqueid=443fd57e-f33c11e6-8c87ecdb-2fe05a6e", parentuniqueid=parentuniqueid@entry=0x0, opflags=opflags@entry=0) at ldap/servers/plugins/replication/urp.c:823
#14 0x00007f533877451f in urp_annotate_dn (sessionid=sessionid@entry=0x7f530aff9f70 "conn=101 op=6 csn=58a2803a000e00030000", entry=0x7f52e400dd00, opcsn=0x7f52e4005560, optype=optype@entry=0x7f53387a0e54 "ADD") at ldap/servers/plugins/replication/urp.c:1023
#15 0x00007f53387747fb in urp_add_operation (pb=pb@entry=0x7f530affca90) at ldap/servers/plugins/replication/urp.c:223
#16 0x00007f533875a2e8 in multimaster_bepreop_add (pb=0x7f530affca90) at ldap/servers/plugins/replication/repl5_plugins.c:726
#17 0x00007f5348296d38 in plugin_call_func (list=0x7f5348c76e70, operation=operation@entry=450, pb=pb@entry=0x7f530affca90, call_one=call_one@entry=0) at ldap/servers/slapd/plugin.c:2049
#18 0x00007f5348296fc3 in plugin_call_list (pb=0x7f530affca90, operation=450, list=<optimized out>) at ldap/servers/slapd/plugin.c:1993
#19 plugin_call_plugins (pb=pb@entry=0x7f530affca90, whichfunction=whichfunction@entry=450) at ldap/servers/slapd/plugin.c:445
#20 0x00007f5338a067b1 in ldbm_back_add (pb=0x7f530affca90) at ldap/servers/slapd/back-ldbm/ldbm_add.c:321
#21 0x00007f534823aee8 in op_shared_add (pb=pb@entry=0x7f530affca90) at ldap/servers/slapd/add.c:699
#22 0x00007f534823c1b0 in do_add (pb=pb@entry=0x7f530affca90) at ldap/servers/slapd/add.c:226
#23 0x00007f53487709b3 in connection_dispatch_operation (pb=0x7f530affca90, op=0x7f53491ea280, conn=0x7f5312538220) at ldap/servers/slapd/connection.c:612
#24 connection_threadmain () at ldap/servers/slapd/connection.c:1759
#25 0x00007f534647196b in _pt_root () from /lib64/libnspr4.so
#26 0x00007f5345e11df5 in start_thread (arg=0x7f530affd700) at pthread_create.c:308
#27 0x00007f5345b3f1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113