There has been a lot of recent activity among security blogs covering serious new vulnerabilities in Mozilla's URI protocol handling. This is a significant issue because it is easily exploitable and yields remote command execution on a victim's PC: an attacker only needs to lead the user to a maliciously crafted link.
The root of this flaw lies in how Mozilla handles URIs that are opened by other applications (such as mailto://, aim://, etc.). These links are designed to pass specific parameters to the external application - for example, a mailto: link typically launches your e-mail client, and composes a new message with a specified "To: address. Mozilla does not sufficiently parse these URIs; as a result, it is possible to inject characters and change the execution path.
To make matters worse, the browser does not prompt the user before handling URIs for “mailto”, “news”, “nntp” and “snews”, so this attack can be launched without any interaction. Yesterday, Billy Rios posted a simple exploit that executes commands through these URIs without any user warning. However, it only works if the victim has no applications registered to the URIs, which is fairly uncommon (Outlook Express handles them by default, and is installed on most Windows systems).
The latest and most comprehensive vulnerability report was released today by Thor Larholm, and includes several proof-of-concept demos for Firefox, Thunderbird, and the mailto: protocol. He goes into a lot of detail - it's a great read and a slick hack.
So an important moral to this story is that registering a URI handler really exposes an application to significant risk of attacks originating from the browser. This is nothing new: for example, there have been numerous buffer overflows affecting AOL Instant Messenger that can be triggered by crafted URIs. However, when you combine them with widespread cross-site scripting vulnerabilities and something like this URI handling flaw, you can see that a lot of interesting attack vectors are possible. It is likely that we'll see variations on these theme affecting multiple browser platforms and used in a variety of exploit scenarios.
One last thing - a good post on ush.it details how you can disable unnecessary external protocol handlers to harden your browser.
Firefox 184.108.40.206 and Thunderbird 220.127.116.11 are vulnerable; this will be patched in the soon-to-be-released Firefox 18.104.22.168.