How can I use pam_access.so to allow virtual users access to FTP?

Issue

I've successfully configured vsftpd to restrict users by IP address. I've successfully configured vsftpd to use virtual users. While the two solutions works fine individually, they do not work when used together i.e. virtual users filtered by IP address. As soon as I enable pam_access, virtual users are not able to log into vsftpd anymore. That is even using the access rule "+ : ALL : ALL" to allow everyone.