Penguin.SWF

The third beta of Adobe Flash Player 10.1 (designated 10.1.51.95) is available for download in labs, including 32-bit Linux.

Posted by Mike Melanson at 9:39 AM | Permalink | Comments (8)

Pursuant to some of my technical posts regarding Linux, some people have asked for similar discussions covering other platforms. Coworker Tinic has just published a detailed blog post describing the drawing improvements being made for Mac platform. Read it and learn about all the possible graphic output paths depending on OS version and browser brand.

I'll tell you one thing-- learning what my Mac colleagues have to put up with when further developing the Flash Player plugin for OS X is almost enough to make me stop complaining about the API jungles on Linux... almost.

Posted by Mike Melanson at 11:21 AM | Permalink | Comments (13)

The most common apples 'n oranges-style tech comparison I hear these days takes place between the Adobe Flash Player and pick-your-favorite-dedicated-video-player program. "Why does Flash Player take more CPU to play a video than my favorite video player?" It's a reasonable question and the answer can be distilled quite simply as, "The Flash Player solves a different problem than your favorite video player."

This is the core thesis of coworker Tinic's "On Performance" blog post from some time ago which addressed complaints comparing the performance of AJAX, SVG, and even static HTML to Flash. I've wanted to write this post for awhile but always hesitated since previous attempts at technical exposition largely fell on deaf ears. However, with the recent developments that minority browsers have made with supporting the HTML5 video tag, coupled with the recent announcements of such video sites as YouTube and Vimeo offering streams via these technologies, more people are recognizing that the issues surrounding video playback in the browser are in no way unique to the Flash Player. For those who wish to learn, here goes (for those who don't, feel free to skip straight to the comment form and get your flame on).

Video flow

Here is the general flow that a dedicated media player has to worry about:

The compressed video data traverses through the video decoder which outputs YUV data to be splatted on the screen.

Here is what the Flash Player needs to do in order to play back video:

The Flash Player has to do a little bit more. In addition to decoding the data, it has to convert YUV data to the RGB colorspace and combine the image with other Flash elements. Then it has to cooperate with another application (web browser) to present the video to the user.

So the dedicated media player solves a problem: Generally, it plays linear media files from start to finish while allowing user interaction in the form of random seeking along the timeline. That's the most basic, trained monkey-type of labor in video playback. At most, the player might handle DVD menus.

Flash Player solves a different problem: It plays linear media files from start to finish while combining the video with a wide array of graphical and interactive elements (buttons, bitmaps, vector graphics, filters), as well as providing network, webcam, and microphone facilities, all programmable via a full-featured scripting language, and all easily accessible via a web browser using a plugin that most of the browsing population already has installed.

"But if I download the media file and play it in my dedicated media player, it takes less CPU."

Perhaps, but that's solving a different problem (playing a file using a dedicated, single-purpose application). The Flash Player works to solve the problem of making video accessible via the web browsing environment.

"But minority browsers don't use as much CPU while playing HTML5 video tag data."

Are you sure about that? As of this writing, browsers that support the video tag are likely to use a tremendous amount of CPU in order to play video. Why? It's likely because they have to deal with the exact same performance penalties that Flash does; specifically, that YUV video data needs to be transformed into RGB data to be plopped in the browser window. This is why I thought this might be a good time to write this article-- I have been seeing more and more comments in online forums observing that the video tag seems quite likely to peg CPUs.

"But at least with HTML5 video tag data, I can download the video and play it offline using a dedicated player."

Try to keep up here. That's not the problem that Flash Player is chartered to solve (same goes for HTML5 video). I know you're smart enough to download videos and watch them offline, but try to think on a macro level. Try to think of non-tech-savvy people of your acquaintence. (Please tell me you know people who aren't as tech savvy as you. If you don't, get to know some in order to maintain proper perspective.) Have you ever tried to tutor such an individual (remotely) in downloading and installing a separate video player? I have and it rarely goes well.

But I digress...

What can be done to improve performance?

Flash Player has an impressive software H.264 decoder, competitive with anything else you're likely to find out there. But there's going to be that YUV->RGB conversion penalty, plus blending the other SWF elements on the video, and possibly scaling to a much larger rectangle. We're pushing the software algorithms as hard as they will go. So what are the hardware options?

Dedicated decoding coprocessor

We're beginning to see video decoding coprocessors packaged in computers, particularly low-end netbooks. In this situation, the green box labeled [Video decoder] in the diagrams above is replaced by a piece of hardware. Everything else about those diagrams remains the same (including Flash's need to convert YUV to RGB). Flash Player 10.1 has support for Broadcom's video decoding coprocessors (according to the release notes, the BCM70012 "LINK" chip). This is currently supported in Windows. Support for these chips in Mac and Linux would be nice but the drivers aren't ready yet.

Video card acceleration

Assorted video cards from Intel, ATI and nVidia are also able to assist in video decoding as of Flash Player 10.1. Again, see the release notes for more detail on which models and drivers are required. Again, this is presently only available in Windows.

"Why aren't these card supported in Linux? My free software media player support VA-API, VDPAU, and XvBA (the various hardware acceleration APIs)."

Again, say it with me: Flash Player solves a different problem.

In the above depiction of Flash Player's workflow, the video card stands in for the green [Video decoder] box. The key point here is that the decoded video frames need to be accessible by the Player which needs to do its thing before the data can be presented to the user. As of this writing, none of these drivers in Linux allow retrieval of the decoded video data. Their counterpart Windows drivers do allow this which is why this feature is supported in Windows.

That's for Linux. What about Mac? I'm not sure but my Mac colleagues have mentioned something about Apple not making their hardware decoding APIs available to applications (if the APIs exist at all, which I'm not sure they do).

Video card scaling

After Flash Player has decoded a video frame, overlaid everything on top and is ready to present it in fullscreen mode, the hardware can assist in doing the scaling to fullscreen. This is a feature that Flash Player has had for a few releases now. Having to scale an RGB bitmap from 400x300 -> 1920x1200 is something that can easily be pawned off on hardware assuming that the right APIs are present.

"Why doesn't the Linux Flash Player use Xv for fullscreen scaling?"

We're not really going through this again, are we? I hope I have cogently presented the reasons between this post and this other one.

Summary

In case it's unclear, let me re-iterate: Flash Player is not the same as your standalone multimedia program. Flash Player solves a different set of problems than your standalone multimedia program. You might claim that your favorite multimedia program solves video playback problems to your satisfaction. However, the internet at large has unanimously declared that it appreciates the Flash Player's solution to the video problem.

Regarding the myriad video acceleration APIs in Linux, we are watching developments closely and look forward to implementing support when the drivers support the features conducive to Flash Player's operational model.

Posted by Mike Melanson at 12:30 PM | Permalink | Comments (108)

It seems that my Welcome to the Jungle post illustrating the mess of audio APIs in Linux has shaped up to be among the most popular on this blog. A picture is worth a thousand rants and all that. I sometimes consider updating that graph to reflect the current (surely messier) state of things. But I think it drives home its point nicely as is.

Another topic that has arisen is hardware-accelerated video decoding, H.264 video in particular since that is the standard video format. There aren't nearly as many ways to do this as there are there are to, e.g., push time domain samples out to a kernel-abstracted DAC, but it still amounts to at least a thicket.

Unfortunately, all of these options have problems right now which make them unsuitable for use in the Linux Flash Player. I plan to expound on this fact in a later post.

Posted by Mike Melanson at 5:00 PM | Permalink | Comments (41)

There's a new version of Flash Player 10 available, designated version 10,0,42,34, which primarily addresses security issues. This update includes x86_32 Linux and even a refresh of the x86_64 version. In the interest of managing expectations, the new x86_64 version is not significantly different from the previous x86_64 version. It is different in that it contains the relevant security fixes and perhaps a few other low-risk fixes.

Check the sidebar ("Adobe Flash Player For Linux") for download locations.

This is the new current version of Flash Player 10, which is not to be confused with the 10.1 beta which was recently released and of which there is presently no 64-bit version. Sometimes I think I need to draw a picture to keep all these versions straight (but where to find software that can enable such creativity?).

Posted by Mike Melanson at 12:17 PM | Permalink | Comments (18)