First off, upgrade Adobe Reader to latest version if you haven't done so already.
Second, close IE 7 and re-open it to clear the plugin from memory. Then, browse to another web site (any will do) which has PDFs and try opening one there and see if it takes just as long.
This will narrow it down to either a plugin issue or an issue with your web site.
Let me know how this works out before we proceed.