Silverlight is a breeze in Safari for me. It doesn't eat all of my CPU and the fans keep it down. When I watch a flash movie however on my Mac the fans kick in and it sounds like it's gonna take off or something.
Click2Flash let's me load the m4v equivalent on youtube but that's unfortunately only for youtube.
I've noticed that either using Snow Leopard or using the "Play fullscreen in QuickTime" option cause better framerates. I think it has to do with support for the <video> tag.