You can do that on your firewall, which is the most effective or even on windows firewall.
create a rule that allows rdp (3389) from only a specific IP Address
if you have other users connecting to this box then you would have to create rules for everyone, this would basically means that only the listed ip's will be allowed.
doing it on your firewall is preferred as it doesn't disturbs your server's firewall configuration.
