If you run the vm-pop3d as a deamon, then it might not be possible. If you set it to run through xinetd, then it may be possible (for both pop and imap) to be setup for certain IPs. I'm not farmiliar with the xinetd.conf for doing that sort of thing, but found this page: