Firefox is a web browser, not an email client (unless you're using some kind of plugin). When you click a mailto link, it tries to open an email client--not a web browser.
You can change this by going to control panel>default programs>Set program access and computer defaults>Custom. There you can select a different email client under 'Choose a default email program' (for example, Thunderbird) and click OK.