MOV, WMV, AVI, MPEG, MP3, OGG, WMA, … Media Formats Explained

Like my “Top 13 things not to do” page, this page is mainly intended to vent my irritation towards things that bother me on the internet. The topic of today is (multi)media formats and the lack of knowledge thereabout by the average person. Many people do not know the difference between a container format and a codec. Other people have some — mostly unjustified — hatred against certain media formats and try to justify this hatred with pseudo-knowledge when bashing those formats on forums and newsgroups. This hatred is often rooted in ignorance about the subject in question. Instead of descending to the same level as those people and insulting them on the same forums and newsgroups, I will now try to give a shot at re-education.

This article was written around the year 2006 and has since been updated only sporadically. Although the first parts are general and will always be applicable, the information about specific containers, codecs and media players is starting to become dated (e.g. there are newer codecs, the latest WMV codecs are not as closed and proprietary as the ones at the time this article was written, etc.) I gave up on keeping the page updated with information about the latest new codecs, therefore you may want to look elsewhere if you want to have the most up-to-date information.

Summary

Because this text is quite long, here's a summary.
A media file like a .MOV, .AVI or .WMV, is a container file. If you're talking about a ‘format’, you are actually talking about the container format. A container file specifies how the data streams inside it are organized, but doesn't say anything about how the actual data is represented. This is the task of the codecs. A codec describes how video or audio data is to be compressed and decompressed. Theoretically, you can use any codec inside any container format. However, codecs are traditionally licensed exclusively to a certain format. For instance, the Sorenson codec is only found in QuickTime files and WMV video is only used in Windows Media files.
Many people don't know the difference between formats and codecs, hence think that all QuickTime files use the infamous Sorenson codec and are a pain in the ass to play. Or, they think that AVI files will always be good quality because they would use the DIVX codec.
There is no ‘ideal’ format for all situations. Some formats are easier to use on certain platforms, some are nearly impossible to use. If you need to embed movies in your website, the best way to deal with this problem is to give the user a choice between several different formats.

Container formats vs. codecs

Container formats, or formats in short

Media container file exampleThe most common mistake people make, is confusing container formats with codecs. Many people hate QuickTime, but in fact what they're actually hating is the Sorenson codec, even though they don't even know what it is. Other people think AVI is a great format because it delivers good quality. I can make you an AVI that will make you puke. And other people wonder why one AVI or MOV plays fine on their computer while another one gives no image or sound.

When you download an AVI, MOV or WMV movie, you're downloading a container file. As the name says, it is a file which contains something else. In the specifications for MOV or AVI files, you will find nothing about how to store, compress or decompress actual video or audio data. The only thing these files do, is providing a wrapper around data streams. When talking about a ‘format’, you're talking about the structure of a container file. So saying that “the AVI format has good video compression” is nonsense, because the format has nothing to do with how the video is compressed.

Internal structure of container fileInside those container files, there are one or more data streams. The most common files have one video stream and one audio stream, but in more advanced formats like QuickTime, there can be any amount and type of streams. Any container that is designed for streaming (either from a sequential medium like a DVD, or the internet) will interleave the streams in chunks. This ensures that data that belongs together temporally, e.g. audio chunks that belong with certain video frames, can also be read from approximately the same location inside the file. Again, the format specifications only say how these streams are to be included in the file, not what's in the streams themselves. This is where the codecs come into play.

A short list of the most common container formats, the file extensions they use, and a bit of history:

  • QuickTime (.MOV, .QT): this was the first real successful multimedia format, introduced by Apple Computer, and it dates back to 1991. The first versions were only available for Mac OS but a Windows version soon followed. From the start on, QuickTime was quite advanced, with support for any amount of streams and multi-segmented files. Editing a movie is simple and intuitive. Later versions even allowed scripting and interactive features. I remember viewing (albeit very short) movies of Apollo launches and a heap of camel bones fusing into a skeleton, while on a PC the most exciting thing one could do was watching an animated GIF.
  • Video for Windows (.AVI): of course Microsoft had to make a clone(1a) of QuickTime as soon as possible, and they managed to do it within one year. Maybe this is why the original VfW specification was such a braindead format. It didn't have most of the advanced features QuickTime has. The format was extended by the OpenDML group to tackle some limitations of the original specification. Later on, MS dropped support for AVI in favour of their new WMV format. One reason was that VfW was not a big success. The main reason however was probably because AVI had become the format of choice for pirated movies — maybe due to the link between the simplicity of the format and the simplicity of the average teenage scriptkiddie pirate. Arrrr. Technically however, those movies are not AVI files because Microsoft's VfW specification doesn't allow the use of the typical pirate codecs.
  • DivX (.DIVX): at the time of this writing, this format is totally identical to AVI, at least the kind of unofficial AVI with the non-standard codecs. This is also the reason why the company which produces the (now legal) DivX codec has changed the file extension. Although Microsoft abandoned AVI, they still don't like it to be associated with pirated movies. Mind that due to this name change there's now a DivX format and a DivX codec.
  • RealVideo (.RM): this was one of the first more succesful competitors for QuickTime, although it never really became very popular either, probably due to the fact that initially it could only be played in the extremely proprietary RealPlayer. There is also an audio-only variant called RealAudio (.RA, .RAM).
  • Windows Media (.WMV, .WMA, .ASF, .ASX): Microsoft's second attempt to dethrone QuickTime was more succesful. It was just a matter of timing and abusing their 95% market share for the umpteenth time. Contrary to the time when VfW was introduced, now the average people actually started to use video on their PC. Windows Media being bundled with Windows, it really didn't matter how good or bad it was. Being preinstalled gave it a huge edge over QuickTime, which had to be installed by the user. The MS monopoly did the rest.
  • MPEG-1 (.MPG, .MPEG, .MPE): this is a format designed by the ISO, or International Organization for Standardization. It is about as braindead as AVI, but was much more popular due to the fact that it is an ISO standard, and was inherently linked to a compression scheme which was really good at that time. The format was, and is still used in Video CD's (VCD). The nice thing about this format is that it's an open standard. This means that playback is supported pretty much everywhere, although you need to pay a license to make an encoder.
  • MPEG-2 (.VOB): this is an update of the MPEG-1 format, and is also an ISO standard. It is the format used for DVD movies. It is also inherently linked with certain compression schemes which allow better quality than the ones from MPEG-1.
  • MPEG-4 (.MP4): another installment in the MPEG series, offering much advanced features over the previous. Because it was supposed to offer similar features as QuickTime, MPEG-4 is in fact a fusion of QuickTime and the older MPEG standards. Apple has decoupled QuickTime from the .MOV format and made .MP4 the default format for the QuickTime 7 platform.
  • Audio Interchange Format File (.AIFF, .AIF) and Windows WAVE audio (.WAV): as the names say, these formats can only handle sound. I take them together because they're pretty much equivalent and are subsets of the more general IFF format. AIFF is popular on Mac OS and some flavours of UNIX, WAV in Windows. Traditionally, an AIF or WAV holds simple uncompressed audio. But the format specification actually allows to use any codec, so you can e.g. have a WAV or AIF with MP3 audio.
  • Ogg (.ogg): this is an open-source format, which is only supposed to contain Vorbis or FLAC audio and Theora video streams. Vorbis was designed as a royalty-free alternative to the MP3 and WMA codecs which require a license to be used. On top of that, it offers a superior quality compared to those codecs. Theora is the same for video.
  • Matroska (.mkv): an open-source container format that intends to provide more flexibility than the aging AVI format. For instance, multiple sound and subtitle tracks can be contained in the same file. Matroska is popular within certain circles like underground releases of Anime series and HD-DVD/BluRay rips, although there is no specific reason why it would be better for that type of content.
  • Flash Video (.flv): like the name says, it is the container format used by the Adobe Flash plug-in. When you view videos in YouTube and other Flash-based video websites, your browser actually downloads an .flv file in the background and feeds it to the Flash player. This format is gaining increasing support by other standalone players. The container format itself is an open standard, but some of the codecs contained inside the files are proprietary.
  • WebM (.webm): this is actually a restricted variant of the Matroska container, only allowing the VP8 video and Vorbis audio codecs. The format is intended to become the standard for HTML5 video. However, at this time there is a battle between proponents of this format (Google amidst others) and proponents of H.264 (in an MPEG-4 container). Either one of these formats will need to emerge as the winner in order for HTML5 video to become a truly practical standard.

Codecs

Remember that a container file is nothing more than a wrapper around a bunch of data streams. The structure of the data inside those streams depends on which codec was used to encode it. Codec stands for coder/decoder. Theoretically, one could just take raw video and dump it into a file. This would make the file monstrously huge however, hence the need for encoding. The same goes for audio, although uncompressed audio is still somewhat tractable and was actually used in the early days (which is why those movies were very short).

The reason why people often confuse codecs with formats is that traditionally codecs are linked with formats. The Sorenson codec used to be licensed exclusively to Apple, hence Sorenson was only to be found in MOV files. The original DivX codec was a hacked Microsoft MPEG-4 codec used in the first pirated AVI movies, so DivX is often confused with AVI. And with the name correction of the non-standard AVI into DIVX, it is clear that a .divx file is only intended to use the DivX codec.

However, MOV supports a myriad of other codecs next to Sorenson, and the same goes for AVI and DIVX. Technically spoken there's no reason why you couldn't use a WMV stream inside an AVI or MOV file, or a Sorenson stream in a WMV file. The only barriers against this are potential lack of support in the format for advanced features of the codec, and legal barriers. MOV files actually exist with DivX video streams inside them. Having an MP3 audio track for a MOV has become quite common, even though the standard QuickTime software does not give you the option to use this codec for audio. Even though the original AVI specification didn't allow to use the DivX or MP3 codec, the amount of AVI files with these codecs is now to be counted in the millions.
The tendency towards hard-linking a format with a (set of) codec(s) is both good and bad. The good is that you immediately know what codec such a file uses, the bad is that you're stuck with that codec. It is less confusing for the average user, but more frustrating for the more professional users. That's why QuickTime is a popular format for editing video: it allows any codec to be used. A single MOV file can even contain multiple streams with different codecs!

A list of popular video codecs and the formats they are/were used in:

  • Cinepak: now utterly obsolete, but in the early days of internet video this was the most popular codec. It is a standard codec in both QuickTime and VfW. The codec worked by splitting up each video frame in a grid of small squares. If in the next frame a square was almost identical to the same square in the previous frame, it was omitted. Otherwise a new square was stored, which was compressed on its own. Compared to raw video, this gave a huge reduction in size.
  • Motion-JPEG: this is an adaptation for video of the JPEG compression scheme (used for still photographs). Actually MJPEG just encodes each frame with a variation of JPEG that is optimized for speed, without trying to exploit relations between similar frames. The advantage is that each frame can be accessed without having to reconstruct previous or future frames, for this reason this codec is still used when straightforward editing is desirable, e.g. on portable cameras.
  • MPEG-1: this codec is actually a group of codecs, meant to be used in MPEG-1 media files. In the case of MPEG-1, it is justified to interchange the format and the codec, because they are mostly unseparable. This codec builds upon the same idea as Cinepak, but adds the concept of motion estimation. This means that each of the tiny blocks can now also move around between consecutive frames. So-called P-frames can be described, or predicted, by referring to parts of the previous (or same) frame at any position (in practice, the range of movement is limited). It is also possible to use both preceding and future frames for prediction (so-called B-frames, for ‘bidirectional’). This enables an even huger reduction in file size compared to Cinepak.
  • MPEG-2: technically, there is not much difference with MPEG-1, except for some small improvements. While MPEG-1 was mostly meant to be the digital equivalent of VHS video, MPEG-2 is intended for higher quality digital video, in casu DVD. The specification allows larger video resolutions and higher bitrates. This codec requires licensing for commercial applications.
  • MPEG-4: you may wonder “what happened to MPEG-3?” Well, it never existed. The MPEG numbering scheme is a story on its own, for instance after MPEG-4 comes MPEG-7. Anyhow, MPEG-4 is not really a codec, but more like a specification for a group of codecs. This means that one can make multiple different MPEG-4 compliant codecs. The nice thing is that if a codec is MPEG-4 compliant, it can in theory be played with any MPEG-4 compliant player. The MPEG-4 standard consists of multiple parts and Part 2 describes the first MPEG-4 video codec, which is therefore the one that is often simply called “the MPEG-4 codec”. The basis is the same as MPEG-1, but with advanced features which allow another leap in compression performance. For instance, instead of simply linking image parts to different positions in previous frames, small differences between those parts can be modeled and encoded as well if not too large. This allows much more efficient modeling of colour/intensity changes or minor differences between frames. The standard also allows predicting a frame from multiple reference frames, and multiple consecutive B-frames instead of only one. Microsoft was one of the first to make a so-called MPEG-4 codec, but as usual they managed to make it non-standard in some way.
  • H.264: this codec was developed by the same groups as MPEG-4 and therefore has been included in the standard (more specifically, it is part 10 of the standard). It has been designed to give excellent performance even under low bitrate situations, and it achieves this goal. Some of the features that enable this are an in-loop deblocking filter and a much larger flexibility in what data can be used to predict other data. At the time of this writing it is one of the best performing codecs. Unfortunately this performance comes at the price of increased processing power requirements. Also, even though this is a standardized codec, licensing is required for commercial applications. The popularity of the open-source library called x264 has caused many people to incorrectly think there is an “x264 codec”, while x264 is just an implementation of the H.264 codec.
  • Intel Indeo Video: this is a proprietary codec from Intel. It is a bit like a predecessor of Sorenson, with the difference that it isn't licensed exclusively to a single format: Intel made the codec available for both VfW and QuickTime. Unfortunately, the QuickTime codecs were not ported to the new OS X version of QuickTime. On the technical side, I cannot tell how Indeo works because it is proprietary. I assume it has similarities with MPEG-1. There were multiple versions with increasing performance.
  • Sorenson Video: this is the infamous codec I've been talking about. It was originally licensed exclusively to Apple, and later on also to Adobe, so you'll only find it in QuickTime and Flash movies. The upside of this codec is that it had quite good performance. There were several versions, with each version improving upon the previous. During a long time, this codec was used for all the movie trailers available on the Apple site. (Luckily, these trailers now use the standard MPEG-4 codec or H.264.) Again, due to it being proprietary, I can't tell you anything about how it works. Judging from the compression quality and artefacts on low-bitrate video, it seems to be related to MPEG-1, with later versions moving towards MPEG-4.
  • WMV: like the name says, this is the codec which accompanies the Windows Media format. It is as proprietary as it can be, so don't ask me how it works. Because it came shortly after Microsoft's non-standard MPEG-4, I assume they started from the same basis. There are several versions with slight improvements in each update. However, the main reason why MS is pushing this format, is that they have absolute control over it. It features DRM, which officially means Digital Rights Management, but from a user's point of view it might as well mean Digital Restrictions Management.
  • DivX: the first version of DivX (DivX 3) actually was nothing more than a hacked version of the MS MPEG-4 codec. This codec started the proliferation of pirated movies, because most movies could now fit onto one CDR with reasonable image quality. The pirates hijacked the AVI format for this purpose. For audio, another hacked codec was used: either MP3 or WMA. Some of the early pirated AVI movies even used WMV for video. With version 4, the developers of DivX re-wrote the codec from scratch to make it legal and improve it. More versions with better performance followed, and from version 6 on, DivX decided to rename the ‘pirated’ AVI format to DIVX.
  • XviD: yes, it is simply DivX spelled backwards. This codec was an attempt by the open source community to create a legal clone of the DivX 3 codec. It was developed independently from the DivX 4 codec, but since both codecs are MPEG-4 compliant, the resulting video streams are mostly compatible. After a few versions, the XviD codec equalled or even surpassed the quality of the DivX codec.
  • VC-1: a codec primarily developed by Microsoft and originally tightly tied to WMV, but it has become more open and is one of the codecs supported by BluRay. It is often considered an alternative to H.264, it has a similar licensing scheme and offers comparable performance, with an advantage in low-latency situations.

A list of popular audio codecs and the formats they are/were used in:

  • PCM (Pulse-code modulation): this is not really worthy of the name ‘codec’, because it is in fact uncompressed audio. It was the only option in the early days, and the only way to save space was to use lower sampling rates or fewer bits per sample, resulting in a dull or noisy sound.
  • A-law and Mu-law: these are not really codecs either, but alternative ways to sample audio than the regular volume intervals used by PCM. A logarithmic scale is used, which effectively gives more dynamic range. In PCM, more silent sounds have more noise, while with A- or Mu-law the noise is more equally distributed over a large range of volumes.
  • IMA/ADPCM: this was one of the first codecs that delivered a 1:4 compression ratio for 16-bit audio, with less quality loss than simple downsampling. It was used both in QuickTime and VfW, called IMA in the former and ADPCM in the latter. The codec was still primitive, though, and an audible amount of noise was introduced in the sound.
  • QDesign Music and Qualcomm PureVoice: these are two proprietary QuickTime codecs which were quite popular in the days they were introduced. The QDesign codec has similarities to MP3 but is more primitive, and the Qualcomm codec is designed to compress speech.
  • MPEG-1 audio (MP1, MP2, MP3): more correctly, the three flavours of this codec are called MPEG-1 Layer I, II and III. Complexity and low-bitrate performance increases with increasing layer number. Layer I is a simplified version of II, Layer III is quite different.
    The Layer III codec is by far the most popular (not just of these three, of all codecs) and is more commonly known as MP3. Like DivX/MPEG-4, this popularity finds its roots in piracy: MP3 made widespread copying of music over the internet possible. Strictly spoken, anyone who makes an MP3 encoder needs to pay a license fee to the Fraunhofer institute. Worse, using MP3 for a commercial streaming application also requires a license.
    These codecs work so well, because they use a so-called psychoacoustic model, a model of the human auditory system, to throw away inaudible information. The compression ratio can be chosen; a popular bitrate is 128kbps which results in a ratio of 1:11. It has long been a myth that it results in sound quality indistinguishable from a CD, and many people still believe in this myth. The truth is that on any decent sound system and with well-functioning ears, 128kbps MP3 sounds like manure. A bitrate of 192kbps or higher is more realistic for near-CD quality. (In case you were wondering: transcoding a 128kbps file into 192 kbps will not improve its quality, on the contrary.) Moreover, there is a complication: the standard only describes the MP3 decoding in detail, not the encoding. This means that there are infinite ways to implement an MP3 encoder, which is why quality between MP3 files of the same bitrate can greatly differ. There are some very good MP3 encoders out there, like LAME, and also some truly crappy ones. Mind that despite its higher complexity and better performance at low bitrates, MP3 has some inherent limitations that prevent it from accurately reproducing certain sounds even at its highest possible bitrate. Starting at 256kbps, Layer II actually beats MP3 in this regard.
  • AC3: this is one of the codecs used for audio on DVD and BluRay. It is designed to support multiple channels, which makes it suitable to encode surround sound. It is related to MP3 because it also uses frequency information to discard inaudible information, but is more primitive. It is typically used at bitrates of 384 or 448kbit on DVD, and 640kbit on BluRay.
  • DTS: another codec that is specifically designed for surround sound on DVDs and BluRays. It uses a less complicated compression scheme than AC3 and therefore requires a much higher bitrate: about 1.5Mbit (similar to uncompressed stereo CD audio). At this bitrate DTS is generally regarded as being indistinguishable from the uncompressed source. It is possible to encode 5.1 DTS at 768kbps but in my experience this results in worse quality than 384kbps AC3.
  • AAC: this is intended to be the successor to both MP3 and AC3 and is the standard audio codec for MPEG-4 media (to be precise, it belongs with Part 3 of the MPEG-4 standard). Quality is slightly better than MP3. It is also the codec used in the iTunes music store, initially augmented with a DRM scheme until Apple decided to switch the entire store to the DRM-free “iTunes Plus” in 2009. Although a license is required to manufacture an AAC codec, unlike MP3 no license is required to use the codec for streaming of any kind.
  • WMA: the audio codec of Windows Media. Judging from sound quality, it is very strongly related to MP3, although newer versions are slightly better. How much better is to be guessed because it is proprietary. This format is often used in other music stores, and of course will use DRM in such cases.
  • Ogg Vorbis: this is an open source alternative to MP3, developed by Xiph. It is designed to be completely patent- and royalty-free (although the companies behind MP3 and AAC want you to believe otherwise). Moreover, the performance is really good, from low to high bitrates. This codec focuses on quality-based encoding, which means the bitrate is automatically adjusted to ensure constant quality. For most applications which do not involve real-time streaming, this makes a lot more sense than the fixed-bitrate that is the default for most other codecs. Of course this codec also supports a fixed-bitrate mode.

Media formats vs. players

The second most common mistake is to confuse a media format with a media player. Traditionally, a media format was only playable in its accompanying player, like the QuickTime Player for MOV, Windows Media Player for AVI and WMV, or RealPlayer for RA and RM. This once made sense, because in the beginning there was no need for other players: there was only QT on Mac OS, and AVI on Windows. However, there is no technical reason why today a certain player couldn't open and play a different format than its native format. Again, the only barriers against this are legal. But, many of these barriers are starting to break due to the proliferation of media formats on the internet. For instance, after a few versions QuickTime Player could also play standard AVI files. And with the correct plug-in, the Mac OS version can even play WMV files. Most players can play MPG files and virtually any player can play WAV files. MPlayer and VLC are open-source players which intend to support as many formats as possible. They can play MOV, AVI, WMV, RM and many other formats, and support a large number of codecs.

There are only two obstacles that can prevent a player from playing a certain file: the codecs or DRM. Codecs are the reason why one AVI or MOV file plays fine in your copy of mplayer while other files with the same extensions don't: if your player doesn't have the codec used in the file, it won't play. As you know, WMV is a proprietary codec, which is why playing WMV files in anything else than Windows Media Player is often a pain in the ass. Detailed specifications required to implement your own complete WMV codec are not publicly available. Any players different from WMP that can play WMV either pay a license fee to Microsoft, are based on reverse engineering, or (ab)use copied Windows Media software libraries. This means that the variants of WMV that have not been reverse engineered yet, cannot be played on a different platform than the ones for which WM libraries are available.
The same goes for QuickTime's Sorenson codec. The most recent variants of it have not been reverse engineered yet, so to play them you either need to use QuickTime or hack into the QuickTime libraries.
DRM (Digital Rights or Restrictions Management, depending on whether you're a lawyer or user) is the other reason why you can only play certain files in a certain player. As far as I know, Windows Media is the only media format that is engineered for DRM. The creator of a WMV or WMA file can decide to only let the user play a movie on a single PC after having paid for a license code. Because WM DRM is only implemented in Windows, it's impossible to play such files on any other platform. The AAC audio files from the iTunes music store also had a layer of DRM added to them until the introduction of iTunes Plus, but the restrictions were not that strict. It is clear that from a user's point of view, DRM is totally undesirable.

Why are certain formats hated by certain people?

QuickTime

On the Windows platform, and also to a lesser degree on Linux, QuickTime has become rather unpopular, due to different reasons. Of course, one of the reasons is that Windows Media is bundled with Windows, and QuickTime isn't. That's not something Apple can do much about. However, Apple itself have also been making a few stupid mistakes. First of all, QuickTime on Windows has become notorious for bad stability. Of course it would have been easy for Microsoft to make QuickTime crash on random occasions(1b). But even if we put all paranoia aside and look at how stable current versions are, there are still some major issues.
Since version 3 of QuickTime, there are two variants: standard QuickTime and QuickTime Pro. The first only allows to play movies, and you need to pay for the latter. However, Apple was stupid enough to include a nagging screen in the standard version which always popped up when opening the QT Player, until you upgraded to QuickTime Pro. That's one hell of a way to annoy people! Luckily the pop-ups were removed in newer versions, but they are still burned into peoples' collective memory. Worse, however, is that for a very long time full screen playback was restricted to the Pro version only. Whoever came up with that idea should be fired for obviously having no clue about how annoying the lack of full screen is. All other free players have it! Luckily, someone saw the light and since version 7.2, full screen is again a feature of the standard QuickTime player. Still lacking is a full screen option in the browser plug-in, so if the person who was behind restoring full screen in QT 7.2 reads this: there's some work left to do.
The grudge of Linux users against QuickTime stems — next to the fact that there has never been an official QuickTime for Linux — mostly from the proprietary codecs like Sorenson and the QDesign Music codec. These were used in many movies created with QT 3 up to QT 6. Actually that's also the only valid argument against QuickTime, because movies with non-proprietary codecs can be played fine. The specification of the QuickTime format itself can be found on the web, and there are open source projects with libraries for handling the format. Plus, the use of proprietary codecs is decreasing in favour of the more standard MPEG-4 and AAC.

Windows Media

Of course, on the Mac and Linux platforms the roles are reversed, and the WMV format is pretty unpopular there, to say the least. Despite what the average Windows user may think, WMV is a lot worse than QuickTime once it leaves its native environment. With all essential documentation publicly available, one can write a working QuickTime player from scratch. Of course it will only be able to play movies with non-proprietary codecs, but it will be able to play many movies. Writing a WMV player from scratch won't even get you anywhere, because both the format and all codecs are propietary. The result is that for a very long time, there hasn't been any decent way to play WMV movies in Mac OS or Linux. When Microsoft finally released the first Windows Media Player for Mac OS it was so bad that it almost seemed intentional(1c). Things got a lot better when Flip4Mac introduced a plug-in that enables QuickTime to import, play and even convert WMV files as if they're QuickTime movies. However, movies with “advanced” (read: annoying) features like DRM are still impossible to play at the time of this writing.
In Linux running on an x86 machine, it is possible to tap into the Windows libraries to decode WMV. So on that platform it isn't too bad, legal issues put aside. However, with the growing popularity of 64-bit machines and the lack of a 64-bit WMV library, it again became apparent that this was just a lucky coincidence. On those machines only the variants of WMV that have been reverse engineered could be played until the advent of a 64-bit WMV implementation. Of course playing files with DRM is — and probably will remain — impossible on any Linux machine until someone hacks it.

Which format/codec should I use for my web movies?

This is a rather nasty question. You will have to live with the fact that there is almost no single format, container-wise and codec-wise, that will fit all your needs and will please your entire audience at the same time. My recommendation is to provide at least two formats for every movie, chosen such that people who don't like or are unable to play the first format can use the second one.

The popularity of YouTube, Google Video and related sites is partly due to the ease of playing Flash-based videos if a working Flash browser plug-in is installed. True, the performance of Flash-based video is not that great on older computers, probably due to the overhead of routing the video through Flash. But still I believe that at this time, it's a good option if you want to include embedded video content on your site.

Nevertheless, there are platforms where Flash is not available, or a major pain to get working. Moreover, it is often hard or impossible to download the FLV file to play it in an external media player or on another machine. So you should provide at least one alternative to cater for the case where Flash is unusable. As you have read in the previous sections, any hatred against QuickTime is mostly unfunded, provided that you don't use any proprietary codecs. My advice: never, ever, use any of those codecs again. The MPEG-4 and H264 video and AAC audio codecs are better, and are ISO standards. A QuickTime file with these codecs, be it a .MOV or .MP4, will play fine in Linux, although the format's history will still make some hardcore users whine. Especially H.264 performs very well in low-bitrate situations, but it is quite heavy on the processor. So, a downloadable QuickTime or (better) MP4 file is a good option.
Considering the fact that Windows still has its rather undeserved huge market share and WMV is pre-installed on it, a download to a WMV file is also acceptable. If you have enough space, providing both a MOV/MP4 and WMV download is a sure way to enable almost anyone to view your videos. Many Linux machines are able to play WMV, so those whiners that don't like QT/MP4 can probably use the WMV, even though this format is IMHO the farthest away from ‘Linux ideals’.

Theoretically, you could also use the AVI format with the DivX or XviD codec and mp3 for audio. Practically, this will require many users to install these codecs, because although they both are MPEG-4 compliant, their use within an AVI file is not compliant with the official VfW standard. Using this format for movies that are supposed to be viewed by computer illiterates is a no-no. But if your users are known to be ‘tech-savvy’ and you require high quality, it is a good choice.

As for codecs, if you use WMV then there is nothing to choose. You might have the option to choose between versions of the codec but I doubt it. If you can, not choosing the latest version will greatly increase the chance that people will be able to view the movie. Enabling any kind of DRM will greatly decrease this chance.
For QuickTime, I'll repeat: never, ever use any of the proprietary codecs! Use MPEG-4 or H.264 for video, and MP3 or AAC for audio. H.264 is really good but is also processor hungry, so for movies that are supposed to be viewed by anyone including people with older computers, MPEG-4 may be a better choice. Of course, for certain movies like simple line art animations, codecs like the good old ‘Animation’ might yield better quality for the same file size. Other codecs may be more suited for other situations, but only use them if you really know what you're doing. Also, you should ‘flatten’ MOV files intended for end-users, meaning that their essential content should consist of a single video and audio stream. The advanced features of QT are nice during development, but many non-QuickTime players are unable to render them.

The upcoming HTML5 standard includes a video tag and the goal is that this tag is linked to a fixed container format with fixed codecs, such that any web browser that is HTML5 compliant is guaranteed to be able to play any video embedded with the video tag in any HTML5 website. Therefore this looks very promising, but at this time HTML5 video is still problematic because no consensus has yet been achieved over the container format, and HTML5 is not yet supported everywhere. There are two contenders: H.264 (in an MPEG-4 container) and WebM, and it is unclear which of the two formats will eventually emerge as the single true HTML5 video standard. Of course, you could simply provide your videos in both formats if processing and storage are not an issue. A solution that has a very good chance to be playable pretty much everywhere, is to use HTML5 video with both formats, plus a fallback to a Flash video player in case HTML5 is not fully supported. This player can use the same H.264 file as was prepared for HTML5 video, or a third custom FLV file.

(1a, 1b, 1c): If you want to know more about the entire complicated media format war between Microsoft and its competitors, this article explains it in detail.

©2006/9-2012/12 Alexander Thomas