SSL doesn't work on my server https://mail.domain.com
As said, best is to not use kind of reserved names for your hostname.Can I use SMTP and POP?
Sorry, I don't understand what you mean by that. If you mean for your hostname, then I also always create an SSL certificate for my hostname, doesn't matter that this has server.domain.com as name.I always installed let's encrypt in apache2, so I could send without asking for a certificate.