I have seen Server 2008 (SBS and otherwise) be *far* more picky about hardware than any other version of windows. Even Server 2008 R2 is less picky. If thsi is a homebrew rig, as it sounds, I'm guessing you just have one of thsoe bad combinations and there really is no way around it. Try a different motherboard...the usual culprit.
The fact that you see no free space during the initial configuration...even if it *works*...is a big sign that sindows is already coughing on the hardware and not properly detecting your drive configuration.