Like my “Top 13 things not to do” page, this page is mainly intended to vent my irritation towards things that bother me on the internet. The topic of today is (multi)media formats and the lack of knowledge thereabout by the average person. Many people do not know the difference between a container format and a codec. Other people have some — mostly unjustified — hatred against certain media formats and try to justify this hatred with pseudo-knowledge when bashing those formats on forums and newsgroups. This hatred is often rooted in ignorance about the subject in question. Instead of descending to the same level as those people and insulting them on the same forums and newsgroups, I will now try to give a shot at re-education.
This article was written around the year 2006 and has since been updated only sporadically. The introductions for each chapter are general and will always remain relevant. Unless you are looking for historical information however, you'll probably want to skip the parts where specific containers, codecs and media players are being described. There are newer codecs now, the latest WMV codecs are not as closed and proprietary as the ones at the time this article was written, and using Flash to play videos is a thing of the past as Flash is (as good as) dead. I gave up on keeping those sections of the page up-to-date with information about the latest new codecs, therefore you may want to look elsewhere for that specific kind of information.
Because this text is quite long, here's a summary.
A media file like a .MOV, .AVI or .WMV, is a container file. If you're talking about a ‘format’, you are actually talking about the container format. A container file specifies how the data streams inside it are organized, but doesn't say anything about how the actual data is represented. This is the task of the codecs. A codec describes how video or audio data is to be compressed and decompressed. Theoretically, you can use any codec inside any container format. However, codecs are traditionally licensed exclusively to a certain format. For instance, the Sorenson codec is only found in QuickTime files and WMV video is only used in Windows Media files.
Many people don't know the difference between formats and codecs, hence think that all QuickTime files use the infamous Sorenson codec and are a pain in the ass to play. Or, they think that AVI files will always be good quality because they would use the DIVX codec.
There is no ‘ideal’ format for all situations. Some formats are easier to use on certain platforms, some are nearly impossible to use. If you need to embed movies in your website, the best way to deal with this problem is to give the user a choice between several different formats.
The most common mistake people make, is confusing container formats with codecs. Many people hate QuickTime, but in fact what they're actually hating is the Sorenson codec, even though they don't even know what it is. Other people think AVI is a great format because it delivers good quality. I can make you an AVI that will make you puke. And other people wonder why one AVI or MOV plays fine on their computer while another one gives no image or sound.
When you download an AVI, MOV or WMV movie, you're downloading a container file. As the name says, it is a file which contains something else. In the specifications for MOV or AVI files, you will find nothing about how to store, compress or decompress actual video or audio data. The only thing these files do, is providing a wrapper around data streams. When talking about a ‘format’, you're talking about the structure of a container file. So saying that
the AVI format has good video compression is nonsense, because the format has nothing to do with how the video is compressed.
Inside those container files, there are one or more data streams. The most common files have one video stream and one audio stream, but in more advanced formats like QuickTime, there can be any amount and type of streams. Any container that is designed for streaming (either from a sequential medium like a DVD, or the internet) will interleave the streams in chunks. This ensures that data that belongs together temporally, e.g. audio chunks that belong with certain video frames, can also be read from approximately the same location inside the file. Again, the format specifications only say how these streams are to be included in the file, not what's in the streams themselves. This is where the codecs come into play.
In almost all cases, video and audio data will be compressed, which means it will be stored in a way that takes less space than if the data would simply be represented in raw form. The compression scheme will typically be specified by the codec and is often an integral part of it. A simple codec could take a raw stream of data and send it through a compression scheme like ZIP, but most practical codecs will try to do something more advanced. Compression schemes are divided into two categories: a codec that preserves the original data with perfect accuracy is called a lossless codec, a codec that is unable to perfectly reconstruct the original data is called a lossy codec. Lossy codecs are designed such that they degrade the signal in a way that is minimally visible or audible. They do this by throwing away data that is the least noticeable according to a so-called psychoacoustic or psychovisual model.
A short list of the most common container formats, the file extensions they use, and a bit of history:
Remember that a container file is nothing more than a wrapper around a bunch of data streams. The structure of the data inside those streams depends on which codec was used to encode it. Codec stands for coder/decoder. Theoretically, one could just take raw video and dump it into a file. This would make the file monstrously huge however, hence the need for encoding. The same goes for audio, although uncompressed audio is still somewhat tractable and was actually used in the early days (which is why those movies were very short).
The reason why people often confuse codecs with formats is that traditionally codecs are linked with formats. The Sorenson codec used to be licensed exclusively to Apple, hence Sorenson was only to be found in MOV files. The original DivX codec was a hacked Microsoft MPEG-4 codec used in the first pirated AVI movies, so DivX is often confused with AVI. And with the name correction of the non-standard AVI into DIVX, it is clear that a .divx file is only intended to use the DivX codec.
However, MOV supports a myriad of other codecs next to Sorenson, and the same goes for AVI and DIVX. Technically spoken there's no reason why you couldn't use a WMV stream inside an AVI or MOV file, or a Sorenson stream in a WMV file. The only barriers against this are potential lack of support in the format for advanced features of the codec, and legal barriers. MOV files actually exist with DivX video streams inside them. Having an MP3 audio track for a MOV has become quite common, even though the standard QuickTime software does not give you the option to use this codec for audio. Even though the original AVI specification didn't allow to use the DivX or MP3 codec, the amount of AVI files with these codecs is now to be counted in the millions.
The tendency towards hard-linking a format with a (set of) codec(s) is both good and bad. The good is that you immediately know what codec such a file uses, the bad is that you're stuck with that codec. It is less confusing for the average user, but more frustrating for the more professional users. That's why QuickTime is a popular format for editing video: it allows any codec to be used. A single MOV file can even contain multiple streams with different codecs!
A list of popular video codecs and the formats they are/were used in:
what happened to MPEG-3?Well, it never existed. The MPEG numbering scheme is a story on its own, for instance after MPEG-4 comes MPEG-7. Anyhow, MPEG-4 is not really a codec but rather a specification for a group of codecs. This means that one can make multiple different MPEG-4 compliant codecs. The nice thing is that if a codec is MPEG-4 compliant, it can in theory be played with any MPEG-4 compliant player. The MPEG-4 standard consists of multiple parts and Part 2 describes the first MPEG-4 video codec, which is therefore the one that is often simply called
the MPEG-4 codec.The basis is the same as MPEG-1 but with advanced features which allow another leap in compression performance. For instance, instead of simply linking image parts to different positions in previous frames, small differences between those parts can be modeled and encoded as well if not too large. This allows much more efficient modeling of colour/intensity changes or minor differences between frames. The standard also allows predicting a frame from multiple reference frames, and multiple consecutive B-frames instead of only one. Microsoft was one of the first to make a so-called MPEG-4 codec, but as usual they managed to make it non-standard in some way.
A list of popular audio codecs and the formats they are/were used in:
The second most common mistake is to confuse a media format with a media player. Traditionally, a media format was only playable in its accompanying player, like the QuickTime Player for MOV, Windows Media Player for AVI and WMV, or RealPlayer for RA and RM. Back in those days this made sense, because in the beginning there was no need for other players: there was only QT on Mac OS, and AVI on Windows. However, there is no technical reason why today a certain player couldn't open and play a different format than its native format. Again, the only barriers against this are legal. But, many of these barriers are starting to break due to the proliferation of media formats on the internet. For instance after a few versions QuickTime Player could also play standard AVI files, and with the correct plug-in, the Mac OS version can even play WMV files. Most players can play MPG files and virtually any player can play WAV files. MPlayer and VLC are open-source players which intend to support as many formats as possible. They can play MOV, AVI, WMV, RM and many other formats, and support a large number of codecs.
There are only two obstacles that can prevent a player from playing a certain file: the codecs or DRM. Codecs are the reason why one AVI or MOV file plays fine in your copy of mplayer while other files with the same extensions don't: if your player doesn't have the codec used in the file, it won't play. As you know, WMV is a proprietary codec, which is why playing WMV files in anything else than Windows Media Player is often a pain in the ass. Detailed specifications required to implement your own complete WMV codec are not publicly available. Any players different from WMP that can play WMV either pay a license fee to Microsoft, are based on reverse engineering, or (ab)use copied Windows Media software libraries. This means that the variants of WMV that have not been reverse engineered yet, cannot be played on a different platform than the ones for which WM libraries are available.
The same goes for QuickTime's Sorenson codec. The most recent variants of it have not been reverse engineered yet, so to play them you either need to use QuickTime or hack into the QuickTime libraries.
DRM (Digital Rights or Restrictions Management, depending on whether you're a lawyer or user) is the other reason why you can only play certain files in a certain player. As far as I know, Windows Media is the only media format that is engineered for DRM. The creator of a WMV or WMA file can decide to only let the user play a movie on a single PC after having paid for a license code. Because WM DRM is only implemented in Windows, it's impossible to play such files on any other platform. The AAC audio files from the iTunes music store also had a layer of DRM added to them until the introduction of iTunes Plus, but the restrictions were not that strict. It is clear that from a user's point of view, DRM is totally undesirable.
On the Windows platform and also to a lesser degree on Linux, QuickTime has become rather unpopular, due to different reasons. Of course one of the reasons is that Windows Media is bundled with Windows, and QuickTime isn't. That's not something Apple can do much about. However, Apple itself have also been making a few stupid mistakes. First of all, QuickTime on Windows has become notorious for bad stability. Of course it would have been easy for Microsoft to make QuickTime crash on random occasions(1b). But even if we put all paranoia aside and look at how stable current versions are, there are still some major issues.
Since version 3 of QuickTime, there are two variants: standard QuickTime and QuickTime Pro. The first only allows to play movies, and you need to pay for the latter. However, Apple was stupid enough to include a nagging screen in the standard version which always popped up when opening the QT Player, until you upgraded to QuickTime Pro. That's one hell of a way to annoy people! Luckily the pop-ups were removed in newer versions, but they are still burned into peoples' collective memory. Worse however, is that for a very long time full screen playback was restricted to the Pro version only. Whoever came up with that idea should be fired for obviously having no clue about how annoying the lack of full screen is. All other free players have it! Luckily someone saw the light and since version 7.2, full screen is again a feature of the standard QuickTime player. Still lacking is a full screen option in the browser plug-in, so if the person who was behind restoring full screen in QT 7.2 reads this: there's some work left to do.
The grudge of Linux users against QuickTime stems — next to the fact that there has never been an official QuickTime for Linux — mostly from the proprietary codecs like Sorenson and the QDesign Music codec. These were used in many movies created with QT 3 up to QT 6. Actually that's also the only valid argument against QuickTime, because movies with non-proprietary codecs can be played fine. The specification of the QuickTime format itself can be found on the web, and there are open source projects with libraries for handling the format. Plus, the use of proprietary codecs is decreasing in favour of the more standard MPEG-4 and AAC.
Of course on the Mac and Linux platforms the roles are reversed, and the WMV format is pretty unpopular there to say the least. Despite what the average Windows user may think, WMV is a lot worse than QuickTime once it leaves its native environment. With all essential documentation publicly available, one can write a working QuickTime player from scratch. Of course it will only be able to play movies with non-proprietary codecs, but it will be able to play many movies. Writing a WMV player from scratch won't even get you anywhere, because both the format and all codecs are propietary. The result is that for a very long time, there hasn't been any decent way to play WMV movies in Mac OS or Linux. When Microsoft finally released the first Windows Media Player for Mac OS it was so bad that it almost seemed intentional(1c). Things got a lot better when Flip4Mac introduced a plug-in that enables QuickTime to import, play and even convert WMV files as if they're QuickTime movies. However, movies with “advanced” (read: annoying) features like DRM are still impossible to play at the time of this writing.
In Linux running on an x86 machine, it is possible to tap into the Windows libraries to decode WMV. So on that platform it isn't too bad, legal issues put aside. However, with the growing popularity of 64-bit machines and the lack of a 64-bit WMV library, it again became apparent that this was just a lucky coincidence. On those machines only the variants of WMV that have been reverse engineered could be played until the advent of a 64-bit WMV implementation. Of course playing files with DRM is — and probably will remain — impossible on any Linux machine until someone hacks it.
This is a rather nasty question. You will have to live with the fact that there is almost no single format, container-wise and codec-wise, that will fit all your needs and will please your entire audience at the same time. My recommendation is to provide at least two formats for every movie, chosen such that people who don't like or are unable to play the first format can use the second one.
The popularity of YouTube, Google Video and related sites is partly due to the ease of playing Flash-based videos if a working Flash browser plug-in is installed. True, the performance of Flash-based video is not that great on older computers, probably due to the overhead of routing the video through Flash. But still I believe that at this time, it's a good option if you want to include embedded video content on your site.
Nevertheless, there are platforms where Flash is not available or a major pain to get working. Moreover it is often hard or impossible to download the FLV file to play it in an external media player or on another machine. So you should provide at least one alternative to cater for the case where Flash is unusable. As you have read in the previous sections, any hatred against QuickTime is mostly unfunded, provided that you don't use any proprietary codecs. My advice: never, ever, use any of those codecs again. The MPEG-4 and H264 video and AAC audio codecs are better, and are ISO standards. A QuickTime file with these codecs, be it a .MOV or .MP4, will play fine in Linux, although the format's history will still make some hardcore users whine. Especially H.264 performs very well in low-bitrate situations, but it is quite heavy on the processor. So a downloadable QuickTime or (better) MP4 file is a good option.
Considering the fact that Windows still has its rather undeserved huge market share and WMV is pre-installed on it, a download to a WMV file is also acceptable. If you have enough space, providing both a MOV/MP4 and WMV download is a sure way to enable almost anyone to view your videos. Many Linux machines are able to play WMV, so those whiners that don't like QT/MP4 can probably use the WMV, even though this format is IMHO the farthest away from ‘Linux ideals’.
Theoretically, you could also use the AVI format with the DivX or XviD codec and mp3 for audio. Practically, this will require many users to install these codecs, because although they both are MPEG-4 compliant, their use within an AVI file is not compliant with the official VfW standard. Using this format for movies that are supposed to be viewed by computer illiterates is a no-no. But if your users are known to be ‘tech-savvy’ and you require high quality, it is a good choice.
As for codecs, if you use WMV then there is nothing to choose. You might have the option to choose between versions of the codec but I doubt it. If you can, not choosing the latest version will greatly increase the chance that people will be able to view the movie. Enabling any kind of DRM will greatly decrease this chance.
For QuickTime, I'll repeat: never, ever use any of the proprietary codecs! Use MPEG-4 or H.264 for video, and MP3 or AAC for audio. H.264 is really good but is also processor hungry, so for movies that are supposed to be viewed by anyone including people with older computers, MPEG-4 may be a better choice. Of course, for certain movies like simple line art animations, codecs like the good old ‘Animation’ might yield better quality for the same file size. Other codecs may be more suited for other situations, but only use them if you really know what you're doing. Also, you should ‘flatten’ MOV files intended for end-users, meaning that their essential content should consist of a single video and audio stream. The advanced features of QT are nice during development, but many non-QuickTime players are unable to render them.
The upcoming HTML5 standard includes a video tag and the goal is that this tag is linked to a fixed container format with fixed codecs, such that any web browser that is HTML5 compliant is guaranteed to be able to play any video embedded with the video tag in any HTML5 website. Therefore this looks very promising, but at this time HTML5 video is still problematic because no consensus has yet been achieved over the container format, and HTML5 is not yet supported everywhere. There are two contenders: H.264 (in an MPEG-4 container) and WebM, and it is unclear which of the two formats will eventually emerge as the single true HTML5 video standard. Of course you could simply provide your videos in both formats if processing and storage are not an issue. A solution that has a very good chance to be playable pretty much everywhere, is to use HTML5 video with both formats, plus a fallback to a Flash video player in case HTML5 is not fully supported. This player can use the same H.264 file as was prepared for HTML5 video, or a third custom FLV file.