If you have a time stamp for one of the connection refuse errors, I would look in the system and application event logs to see if there is a corresponding event around that same timeframe that my indicate the reason for the refusal. Of course, if it is not actually causing any issues, I may be tempted to chalk it up to normal operations.