I am a newbie and I try to understand dns records.
Sorry to say that, if this is the case, then you'd better read some official FAQs and DOCs on the subject. Directadmin forums are not the best place to learn basic things about types of DNS records.
See these links (found with Google):
http://needforair.com/blog/2012/04/22/dns-redirections/
http://www.d2digital.co.uk/blog/a-beginners-guide-to-dns-domain-name-system.aspx
What happens if I remove ftp, smtp and mail A records?
You won't be able to access FTP server by ftp.domain.com, only by (
www.)domain.com or IP, if you delete ftp.
You won't be able to access SMTP server by smtp.domain.com, only by (
www.)domain.com or IP, if you delete smtp.
You won't be able to access SMTP/POP3/IMAP server by mail.domain.com, only by (
www.)domain.com or IP
You won't be able to receive emails at your domain, if you delete mail, as it used for MX record.
Why I point to server's ip?
To avoid a situation, when your site opens by ftp.domains.com, smtp.domain.com, etc.
For example can it be cause to drop my mails to junk?
No
Also I wonder about only am I suffer from this situation or is it a general directadmin situation?
I still don't see anything wrong with the settings and records, and they are good enough for the most cases.
I've never seen a question like this on the web. Am I doing something wrong?
Nobody knows but you. If you need an answer on this question, somebody should check your server and directadmin installation.