Make sure that the IP addresses that you want removed from Apache to have a "free" status in the IP Manager section in DirectAdmin (admin-level login).
Once they have a "free" status, then you may go ahead and delete them from IP Manager.
Deleting them from IP Manager will remove their entry in the Apache configuration files, and the IP addresses will be free for you to do whatever you want with them...