Did you first add it as admin via Domain administration in admin level, not in user level? Do not create the subdomain in user level. A hostname is not a subdomain really.
After creating via DNS administration then check /etc/virtual because there is a bug and the hostname directory is probably not present. Create it yourself. mkdir server.domain.com
chown mail:mail server.domain.com
Wait until resolves, then create dkim record and request the hostname certificate manually via ssh.