Sorry for the delay in responding. What actually caused the issue was having the network cable pass through the surge protection ports of a Belkin Battery Backup. Once I plugged the network cable directly into the NIC on the server the latency disappeared.
Guess I'll never use the network surge protection ports on a battery backup again...