Have you tried turning the echo cancellation off on your Digium PRI card? It may seem like a daft thing to do, but it is best if echo cancellation is only operating at one end of the path and chances are that your telco is already providing it. There is a setting in /etc/asterisk/chan_dahdi.conf that can be used to disable echo cancellation.
Another thing to try is adjusting the setting for LBO in /etc/dahdi/system.conf:
# span=<span num>,<timing source>,<line build out (LBO)>,<framing>,<coding>[,yellow]
# The line build-out (or LBO) is an integer, from the following table:
# 0: 0 db (CSU) / 0-133 feet (DSX-1)
# 1: 133-266 feet (DSX-1)
# 2: 266-399 feet (DSX-1)
# 3: 399-533 feet (DSX-1)
# 4: 533-655 feet (DSX-1)
# 5: -7.5db (CSU)
# 6: -15db (CSU)
# 7: -22.5db (CSU)
Try different values and see if it makes a difference.