LDAP changes do not show up in EPP 5 immediately

Issue

In our current set up the groups/roles info is coming in from LDAP. If we are making any changes directly in LDAP, such as deleting a member from a group, the changes are reflected instantaneously in LDAP. However, in EPP 5 the user could still see the changes for a period of time before he gets to see the change made. Not sure why this is happening? In case this is cache related, is there any place where we can configure the cache settings.