hi,
few reasons that might cause this.....
while creating a new VM you have not chosen freebsd as the OS which might have added unsupported devices.
Create a new VM and before you start the install remove all other devices like USB,floppy & sound card
then perform the install.
One more thing is to try with the stable install as 5266 is a experimental release.
post me back..as i speak i am trying out the stable install on vmwks 7 and will post back.
regards
bhanu