Note to the Internet:

The Transmission bittorrent client supports the magnet bittorrent protocol, but only after Transmission has run once. On it’s initial run it registers itself as the provider of the magnet: protocol. If you haven’t started Transmission (at least, a version after 1.80) and restarted (say) FireFox, the magnet: protocol won’t seem to work.

You need to use the age-old solution of closing everything and starting it back up again.

This is not documented anywhere, but if you search hard enough through the closed bug reports for Transmission you might figure it out eventually.