Gotcha!
It could be an issue with the primary switch having (from what you've described) so many requests and it just bogs down, not to mention the VPN requests that switch is addressing as well. You may want to look into putting a router (with the necessary equipment to connect to the Adtran) in between your Adtran and your primary switch. Is that primary switch a managed switch? If so, you may want to setup those different 'mini' networks into their own VLANS. The router suggestion will help with the packet management and if you every need to pull in another network with a different subnet and IP address scheme.
Hope this helps some. ^_^