Motion Vector Experiments
The use of motion vectors for motion compensation in video compression is ingenious - another testament to how amazing compression algorithms are. I thought it would be an interesting experiment to get into the guts of video decoder and attempt to distort the decoded motion vectors before they are actually used to move the macro-blocks (i.e. before they affect the final output frame). My motivation - more creative in nature - was to see what kinds of images would result from different types of mathematical distortion. The process would also help me better understand the lowest levels of video coding.
In the end I discovered the most unusual effects could be produced by reversing the motion vectors (multiplying their x & y components by -1). The stills and videos shown here were created using this technique. Another test I performed was forcing them all to zero (effectively turning off the motion compensation) but the images were not as 'compelling'.
Here are some stills with Neo and The Oracle conversing from The Matrix: Reloaded after the video's motion vectors have been distorted by my hacked version of libavcodec:
Mashed up: The Matrix - Reloaded
The following snippet of the Burley Brawl from The Matrix: Reloaded has been passed through my hacked version of libavcodec to reverse the direction of each motion vector:
Reversing My Motion Vectors
To more carefully study the effects of reversing motion vector directions, I created a 'control' video of me making particular motions at different speeds. You can witness the results:
libavcodec & ffdshow
To perform these experiments, I needed to be able to tweak the source code of a video decompressor. Since I do most of my development on the Windows platform using Visual Studio and I wanted to support as many codecs that use motion compensation, my options became limited. libavcodec in ffmpeg is the library to encode and decode any of a host of video and audio codecs. There is one catch however: it is written in C using C99 features and VC does not compile C99. Therefore I (back-)ported libavcodec to the Windows platform so I could use a GUI debugger and quickly learn the structure of the code. This would allow me to make quick changes and evaluate the results.
ffdshow is a DirectShow filter that, among others, uses libavcodec to handle a variety of formats in the context of a DirectShow filter graph - very handy stuff. I also modified this to be compatible with my version of libavcodec and to stream a video's motion vectors to an external application via UDP. This enabled me to visualise the motion vectors in my 2D fluid dynamics simulation.
I imagine a Windows version of libavcodec (albeit not in its entirety) would be useful to many people, so I will shortly attach it to this post along with a patch that can be applied to the SVN trunk.