By default all IPs can connect to your server, including SSH port. The idea of limiting an access to SSH of the server is as the following.
- You close 22 port with a firewall for all IPs.
- You add your IP as trusted to a firewall and allow connections to 22 port from your own IP.
This can be easily done with a help of CSF/LFD which might be pre-installed. If you don't have yet installed, then you should go and install it.
Thus you don't need to add/remove IPs from DirectAdmin on IP management page. Instead of it you should work with a firewall on CSF/LFD page.
If your home/office IP changes and you don't have a fixed one, then as suggested by
@Active8, you might think of starting using a VPN. This way you will have a chance to have a fixed IP for your internet connection. Or you might purchase a fixed IP from your internet provider. A fixed IP will allow you to connect to your server from a known and trusted IP, which won't change during your internet sessions and/or after breaks.
And again a fixed IP is here only about IP from which you connect to internet, this is not about DirectAdmin. Servers in internets are supposed to have a fixed IP by default.
If your connection to internet from PC, laptop, phone has already a fixed IP, then you might ignore the sections about VPN and go straight to CSF/LFD page of your Directadmin and add it as allowed/trusted, and then remove SSH port 22 from a list of opened page, it is
TCP_IN in CSF settings.