Sure you can turn of SMTP. Of course if you do, your clients will not onlly not be able to send mail through your server; they won't be able to receive it either.
At the SMTP protocol level there's no difference between mail arriving on the server for local delivery and mail arriving from your clients for relay off the server.
You can do what you want with custom changes to exim.conf.
Jeff