There are so many factors. I actually doubt that the server specs are the problem. One way to isolate that is to run some of your heavier views using SSMS. If they are fast that way, then the server is doing fine. It's more likely the latency, throughput and method of linking that's the problem.
The most likely problem is your (seemingly) exclusive use of linked tables. They have their place, but they are the slowest way to use SQL Server data. Other options are passthrough queries, binding to ADO recordsets, etc.
Please see my PowerPoint presentation on techniques for using Access as a client-server front-end to SQL Server databases. It's called "Best of Both Worlds" at our free J Street Downloads page:
http://ow.ly/M2WI. It includes some thoughts on when to use SQL Server, performance and security considerations, concurrency approaches, and techniques to help everything run smoothly.
Cheers,
Armen