Colors

Posted on May 29, 2015 by Ulrich von Zadow

libavg has always supported specifying colors in html-like fashion: You pass a string of six hex digits, two each for the red, green and blue components of the color. That’s nice if you’re copying the code from Gimp or Photoshop, but it was quite limiting if you wanted to actually calculate colors or maybe interpolate between two colors.

So… there’s a shiny new Color class that can be constructed from a tuple and passed whereever color strings were expected before (of course, strings still work). Colors can be mixed and used in animations. Last but not least, I did some research on smart ways to mix colors. It turns out that a simple weighed average of the RGB components doesn’t look so good – for instance, mixing 50% red with 50% yellow results gives you gray. So libavg now mixes in CIE L’ch color space, which more-or-less preserves the saturation and looks pretty good.

As a result, you can now do this:

startColor = avg.Color("0000FF")
endColor = avg.Color("FFFF00")
for i in xrange(11):
    color = avg.Color.mix(startColor, endColor, 1-(i/10.))
    avg.RectNode(pos=(i*20,0), size=(20,20), fillcolor=color, 
            fillopacity=1.0, parent=rootNode)

to produce:

More infos on color mixing are in this blog post.

Raspberry Pi Update

Posted on April 25, 2015 by Ulrich von Zadow

libavg has supported the Raspberry Pi for a while now. In the last few weeks, I set up a cross-compile toolchain: You compile libavg for the Raspberry Pi on a separate Linux machine. This means compiling is done in a few minutes (as opposed to an hour or more if you compile directly) – here are build instructions. Also, after a bugfix, we have full libavg functionality on the Pi. Video decoding, sadly, is still done in software, since the first two people that have tried implementing it have given up – I’ll see what I can do on that front.

Speed

Posted on January 28, 2015 by Ulrich von Zadow

libavg just got another major optimization.

I implemented an image registry and cache for libavg. ImageNodes that reference the same image file now reference the same bitmap in CPU memory and the same texture in GPU memory. This is completely hidden from the app developer, who just specifies the file location for all instances. The obvious benefit is that this saves a lot of memory if an application re-uses lots of bitmaps. The less obvious benefit is that it speeds things up as well: avg_checkspeed, which tests with thousands of identical ImageNodes, can now handle around 15000 Nodes at 60 FPS on my old i7 (still Core i7 920 Bloomfield, 2.66 MHz, NVidia GF260 like in the old benchmarks). This is twice as many as before.

A Tutorial

Posted on December 2, 2014 by Ulrich von Zadow

Finally, libavg has a tutorial. Over the last few weekends, I’ve put together a short but thorough tutorial on libavg. It covers the important concepts – app structure, scene graph, update loop, event handling and deriving your own node classes – and it does this in the context of a short and very nice 500-line program that exercises all of these concepts.

The firebirds sample that’s been included in libavg for a while is the basis for this tutorial. In fact, when Scotty wrote the sample two or three years ago, I promised him that I’d write a tutorial based on it – Scotty, thanks for the sample and sorry for taking so long!

Multithreading, Realtime Graphics and Process Affinity Masks

Posted on October 11, 2014 by Ulrich von Zadow

In libavg, we try to make it as easy as possible to have a consistent framerate that matches the screen refresh rate. For almost all current systems (and ignoring new developments such as NVidia G-Sync), that means delivering a new frame every 16.67 milliseconds.

To make this possible, libavg is designed as a multi-threaded system and long-running tasks are moved to separate threads. So, for instance, the BitmapManager class loads image files in one or more background threads (the number of background threads is configurable using BitmapManager.setNumThreads()), the VideoWriter uses a background thread to encode and write video files, and all videos are decoded in background threads as well. Besides enabling quick screen updates in the main thread, this also allows libavg-based programs to utilize more than one core in a multi-core computer. The threads are distributed among the cores by the operating system according to the load, and in general, this works pretty well.

However, the operating system has no way of knowing that one of the libavg threads is special and should be able to churn out frames at 60 fps. So, if the background threads cause too much load, some of them will run on the same core that the main thread is running on, and framerate can become irregular.

Happily, there’s a cure for the issue: We lock the screen update thread to one specific core and forbid all other threads from using this core using thread affinity functions.

Version 1.8.2 Released

Posted on July 16, 2014 by Ulrich von Zadow

Release 1.8.1 broke audio on some linux machines and all macs, so we’ve released version 1.8.2 that just fixes this bug.

Switch to github & Version 1.8.1 Released

Posted on June 21, 2014 by Ulrich von Zadow

We’ve moved the libavg sourcecode from our own svn repository to github. Here’s the project in it’s full glory: https://github.com/libavg/libavg. Obviously, that gives us the full power of github, including much better issue and branch/merge tracking, easy forking, etc.. If you look at the libavg network graph, you can see that we’re busy using all these new capabilities .

Most of the work needed – including moving all branches, issue tracking, adapting the continuous build, fixing web links and instructions etc. – was done by Richy, with help by OXullo and Benedikt Seidl. Thanks!

Also, we just released version 1.8.1, a bugfix release. Of course, we happily used github support for this.

Supporting Twelve Screens at Once

Posted on March 5, 2014 by Ulrich von Zadow

Our latest and biggest (not to mention coolest) toy at the Interactive Media Lab Dresden is a ten square-meter interactive wall that’s fully touch-sensitive and supports markers and pens as well. It consists of twelve Full HD monitors hooked up to two Radeon 7970 graphics cards in a single dual-Xeon workstation. Since it’s driven by a single workstation, we can drive the complete wall with a single application, which is really cool and sets it apart from most similar setups. However, the dual-graphics-card setup causes issues: Under Linux, we have two separate desktops, and under Windows, applications that span the graphics card boundary are extremely slow.

To get full-screen rendering at interactive speeds, you basically have to open two borderless windows – each spanning 6 screens and pinned to one of the GPUs. Then you render the same scene with different viewports in each of the windows. That means that all context-specific data – textures, vertex buffers, shaders, framebuffer objects, and even caching of shader parameters – needs to be replicated across both contexts. Also, we can’t switch contexts too often, because that would make things slow.

libavg renders in two passes: The first (implemented in the Node.preRender() functions) prepares textures and vertex data. It also renders FX nodes. The second pass (implemented in Node.render()) actually sends render commands to the graphics card. The multi-context code changes a few things: While preRender() is still executed only once, render() is executed once per GPU. Uploads of data as well as effects that need to be rendered are scheduled in preRender and actually executed at the beginning of the render. In total, refactoring everything accordingly was (obviously) a lot of work that impacts lots of code all over the graphics engine, but the result is good rendering performance with 24 megapixels of resolution.

The code is still on a branch (The svn repository is at https://www.libavg.de/svn/branches/experiments/libavg_multicontext/), but it passes all tests, and I’ll merge it to trunk after we’ve used it a bit.

Version 1.8 Released

Posted on February 15, 2014 by Ulrich von Zadow

I’ve just released version 1.8 of libavg. You can download it at the usual place.

This means that finally, all of the cool features that have been in the development version for a while are available in an easy-to-install package. The release includes a skinnable widget library, a unified event handling framework (see Cleaning up Messaging), and a much-improved App class by OXullo. Scotty added a very nice sample program called firebirds that showcases libavg development in a compact form, and Richy implemented a new logging framework. Rendering is much faster (see Speeding up Rendering), and I completely rewrote the video decoding subsystem (see Video Decoding using libav and ffmpeg). Lots of other things have been improved as well – see the NEWS file for details.

Lots of thanks to OXullo, Richy and Scotty for testing the release!

Video Decoding using libav and ffmpeg

Posted on February 12, 2013 by Ulrich von Zadow

I spent the last month completely taking the libavg video decoding module apart and putting it together again. I’m finally convinced that the code is well-designed and readable – and it’s fast. It turns out that getting good video decoding is not as easy as it sounds, so I’ve written up a complete description of the insides for anyone that’s interested: VideoDecoding.

The weird thing is that from the outside, it looks like a solved problem, so every time I start telling someone about this, I get the same reaction. There’s libraries for that, right? You just plug in libav or ffmpeg or gstreamer or proprietary things like QuickTime or DirectShow. All these libraries have existed for years, so they’re stable and easy to use, right?

Well, yes and no. If you don’t need advanced features, high-level libraries like gstreamer might do what you want. But we want frame-accurate seeking and a low-latency mode, as well as color space conversion using shaders. Opening and closing video files shouldn’t cause any interface stutters, and so on. Also, libavg can’t work with proprietary libs – we need something that works cross-plattform. That leaves libav/ffmpeg, and this library exposes a pretty low-level interface. It does support every codec but the kitchen sink (pardon the wording) and gives you control over every knob that all of these codecs have to tune things. That’s really great, because you wanted control, right? Anyway, you can get everything done with libav/ffmpeg, but suddenly things get complicated. For starters, you’re suddenly juggling five threads: demuxer, video decoder, audio decoder, display and audio mixer. libav/ffmpeg leaves all the threading to the user, so you’re dealing with a pretty complicated real-time system where lots of things happen at the same time. Dranger’s tutorial helps, but it’s dated.

To make things worse, the interface of libav/ffmpeg changes with minor revision numbers, so to support a few years of operating systems, you find yourself adding a generous amount of #ifdefs to the code. I couldn’t find documentation that describes which changes happened in which minor revision, so you need to guess appropriate version numbers for the #ifdefs based on tests with multiple systems. Oh, and there’s actually several constituent libraries that each have their own version number. Of course, you need to query the correct one. All of that takes time; the resulting code is hard to read and test. In addition, since ffmpeg forked and the developers aren’t on speaking terms (see this and this if you really want to know more), you need to test with libav (the fork) and ffmpeg (the original) if you want maximum compatibility.

All of this is really a pity, because I think the libav/ffmpeg developers are insanely smart guys and the library does do a really admirable job of de- and encoding everything you can throw at it. Also, if I’m honest, most of the time spent was figuring out how to organize the different threads well – and that’s something I really can’t blame libav/ffmpeg for.

Anyway, we’re now ready to add Raspberry Pi (read: OpenMAX IL) and VA-API hardware decoding, seamless audio loops and other cool things to libavg.