Allow LDAP user login only if they are part of a local OS group

Is there way to restrict LDAP users to login only if they are part of a LOCAL OS group ? What are the steps ..

Currently i have an openldap client machine setup to talk to an external openldap server . sssd is configured with a domain to lookup in the external ldap server.

If I have a local OS group on my openldap client machine, say mygroup (gid=5000) .

I have added the following rule in /etc/security/access.conf
-:ALL EXCEPT root (mygroup):ALL

I dont want users from the external ldap servers to login via ssh to my machine if they are not explicitly added to 'mygroup' using gpasswd.

Problem I see that if external ldap server also has a group defined with gid=5000 , then users belonging to that LDAP group are allowed gett in via ssh even though those users are not yet added to the LOCAL os group 'mygroup'.

Thanks in advance