I spent the last month completely taking the libavg video decoding module apart and putting it together again. I’m finally convinced that the code is well-designed and readable – and it’s fast. It turns out that getting good video decoding is not as easy as it sounds, so I’ve written up a complete description of the insides for anyone that’s interested: VideoDecoding.

The weird thing is that from the outside, it looks like a solved problem, so every time I start telling someone about this, I get the same reaction. There’s libraries for that, right? You just plug in libav or ffmpeg or gstreamer or proprietary things like QuickTime or DirectShow. All these libraries have existed for years, so they’re stable and easy to use, right?

Well, yes and no. If you don’t need advanced features, high-level libraries like gstreamer might do what you want. But we want frame-accurate seeking and a low-latency mode, as well as color space conversion using shaders. Opening and closing video files shouldn’t cause any interface stutters, and so on. Also, libavg can’t work with proprietary libs – we need something that works cross-plattform. That leaves libav/ffmpeg, and this library exposes a pretty low-level interface. It does support every codec but the kitchen sink (pardon the wording) and gives you control over every knob that all of these codecs have to tune things. That’s really great, because you wanted control, right? Anyway, you can get everything done with libav/ffmpeg, but suddenly things get complicated. For starters, you’re suddenly juggling five threads: demuxer, video decoder, audio decoder, display and audio mixer. libav/ffmpeg leaves all the threading to the user, so you’re dealing with a pretty complicated real-time system where lots of things happen at the same time. Dranger’s tutorial helps, but it’s dated.

To make things worse, the interface of libav/ffmpeg changes with minor revision numbers, so to support a few years of operating systems, you find yourself adding a generous amount of #ifdefs to the code. I couldn’t find documentation that describes which changes happened in which minor revision, so you need to guess appropriate version numbers for the #ifdefs based on tests with multiple systems. Oh, and there’s actually several constituent libraries that each have their own version number. Of course, you need to query the correct one. All of that takes time; the resulting code is hard to read and test. In addition, since ffmpeg forked and the developers aren’t on speaking terms (see this and this if you really want to know more), you need to test with libav (the fork) and ffmpeg (the original) if you want maximum compatibility.

All of this is really a pity, because I think the libav/ffmpeg developers are insanely smart guys and the library does do a really admirable job of de- and encoding everything you can throw at it. Also, if I’m honest, most of the time spent was figuring out how to organize the different threads well – and that’s something I really can’t blame libav/ffmpeg for.

Anyway, we’re now ready to add Raspberry Pi (read: OpenMAX IL) and VA-API hardware decoding, seamless audio loops and other cool things to libavg.