spench.net

I wrote the following major system components:

1) A multi-threaded, load balancing MPEG-2 video decoder engine, featuring:
• Automatic memory management & caching
• Level-Of-Detail support and seamless transitions
• Continuous playback or shot looping (given cut information)
• Asynchronous loading and destruction

[Initial tests indicate it can play over 500 videos simultaneously on one computer (with 2 HT CPUs and 1GB of RAM at the lowest LOD). TVisionarium is capable of displaying a couple of hundred videos without any significant degradation in performance, but there's so much still to optimise that I would be surprised if it couldn't handle in excess of 1000.]

With my latest optimisations, TVisionarium is able to play back 1000 shots simultaneously!
While profiling the system, total CPU usage averages around 90-95% on a quad-core render node!
This indicates that those optimisations have drastically minimised lock contention and support far more fluid rendering.
Have a look at TVis in the following video:




This is an in-development 'video tube' test of the video engine:
(Watch it on youtube.com to leave comments/rate it if you like.)

T_Visionarium was officially launched on 08/01/2006 as part of the 2008 Sydney Festival. Please read my blog post about it. Here are some pictures:

The festival banner:

Crowd before the speeches:

The digital maestros (Matt McGinity & I):

09/01/2007 - SBS World News:

August 2006 - Channel Nine News:

This series of pages summarises the contribution I made to TVisionarium Mk II, an immersive eye-popping stereo 3D interactive 360-degree experience where a user can search through a vast database of television shows and rearrange their shots in the virtual space that surrounds them to explore intuitively their semantic similarities and differences.

It is a research project undertaken by iCinema, The iCinema Centre for Interactive Cinema Research at the University of New South Wales (my former uni) directed by Professor Jeffrey Shaw and Dr Dennis Del Favero. More information about the project itself, Mk I and the infrastructure used, is available online.

I was contracted by iCinema to develop several core system components during an intense one month period before the launch in September of 2006. My responsibilities included writing the distributed MPEG-2 video streaming engine that enables efficient clustered playback of the shots, a distributed communications library, the spatial layout algorithm that positions the shots on the 360-degree screen and various other video processing utilities. The most complex component was the video engine, which I engineered from scratch to meet very demanding requirements (more details are available on the next page).

Luckily I had the pleasure of working alongside some wonderfully talented people: in particular Matt McGinity (3D graphics/VR guru), as well as Jared Berghold, Ardrian Hardjono and Tim Kreger.

I fixed the lighting calculations and thought I would use a built-in texture:

Here is a fly-through of the standard tornado simulation with some pretty filaments:

Shortly after the presentation day, I ripped out the original physics code that someone (who shall not be mentioned!) had written in the minutes prior to the presentation and replaced it with more 'physically correct' code:

A little something I made in my spare time:
(More details coming later...)

This started with my desire to build a Woktenna.
Of course you can't very well put a PCMCIA wireless card at the focal point of a cooking wok!

So the alternative is using a USB WiFi adapter that can hang on the end of a USB extension cable and
introduce minimal analog signal loss and USB is digital!

Despite the fact is says "Linux compatible" on the box, it wasn't immediately possible to do what I wanted to do,
which is: monitor mode!
With monitor mode, I'd be able to point the woktenna around and pick up the beacon frames of distant APs.

I found two drivers available for this device:
one over at BerliOS,
and the other at SourceForge.

Thanks to the generosity of Aras Vaichas, I came into possesion of an old (1992) 60x8 dual-colour LED display. As it was just the display itself (no manual, instructions, software, etc) I set about reverse engineering the board. Using my multimeter I re-created the schematic for the board and found all the relevant datasheets online. Having figured out how to talk to the display, I interfaced it via the parallel port and wrote some control software for it. Once I could display various test patterns (multi-colours sine waves), I 'net-enabled' the software so that the display could be controlled over a network via UDP packets - the resolution is so low that the entire LED configuration fits into a single packet! Finally, I wrote a plugin for Winamp that streams the frequency analysis of the playing song to the display, which produces results like this:

This is a GUI frontend to the genetic programming assignment given in this subject. The aim is to evolve a wall-following robot. The program provides multiple visualisations of the process. It was written with Janice Leung - many thanks for the beautiful widgets! Developed on (but not for) Linux using Python and its bindings & add-ons: PyQt, PyOpenGL, PIL and psyco. README available. It contains more information about the code used to render the robot & world.

My friend Rafal Kolanski and I decided one night to put our gigabit network cards to the test and measure how quickly a user-space program could generate packets and how many would be discarded before the destination's NIC would receive them. Therefore we wrote a small Linux program to complement my Broadcast Flooder, which I had previously written to test other aspects of my network.

Although there exists a plethora of programs that count lines of code, I thought I would write my own. It is designed to analyse C/C++ code and ignore whitespace, // and /**/ comments (both the single and multi-line sort). It also counts the number of FIXME's one has left in their code. Other languages (eg: Javascript, assembly) that also use such commenting conventions are compatible too.

Using my modified version of ffdshow, which sends a video's motion vectors via UDP to an external application, I visualised the motion vectors from The Matrix: Reloaded inside my fluid simulation. The grid resolution is set based upon the macro-block resolution in the video sequence and each type-16x16 motion vector controls one spatially-matching point on the velocity grid. The following visualisation is taken from the scene where they are discussing the threat to Zion while inside the Matrix before Neo senses that agents are coming (followed by Smith) and tells the ships' crews to retreat.

This is the new-and-improved fluid simulation in action. I'm perturbing the 'blue milk' with my mouse. Watch for the darker region form and expand behind the point of perturbation. Due to finer resolution of the velocity grid, the linear artifacts apparent in the earlier version have disappeared and it now looks smooth in all directions.

Velocity-grid-based 2D fluid simulation with effects that interestingly enough resemble Navier-Stokes simulations.

To more carefully study the effects of reversing motion vector directions, I created a 'control' video of me making particular motions at different speeds. You can witness the results:

The following snippet of the Burley Brawl from The Matrix: Reloaded has been passed through my hacked version of libavcodec to reverse the direction of each motion vector:

To perform these experiments, I needed to be able to tweak the source code of a video decompressor. Since I do most of my development on the Windows platform using Visual Studio and I wanted to support as many codecs that use motion compensation, my options became limited. libavcodec in ffmpeg is the library to encode and decode any of a host of video and audio codecs. There is one catch however: it is written in C using C99 features and VC does not compile C99. Therefore I (back-)ported libavcodec to the Windows platform so I could use a GUI debugger and quickly learn the structure of the code. This would allow me to make quick changes and evaluate the results.

The use of motion vectors for motion compensation in video compression is ingenious - another testament to how amazing compression algorithms are. I thought it would be an interesting experiment to get into the guts of video decoder and attempt to distort the decoded motion vectors before they are actually used to move the macro-blocks (i.e. before they affect the final output frame). My motivation - more creative in nature - was to see what kinds of images would result from different types of mathematical distortion. The process would also help me better understand the lowest levels of video coding.

In the end I discovered the most unusual effects could be produced by reversing the motion vectors (multiplying their x & y components by -1). The stills and videos shown here were created using this technique. Another test I performed was forcing them all to zero (effectively turning off the motion compensation) but the images were not as 'compelling'.

Here are some stills with Neo and The Oracle conversing from The Matrix: Reloaded after the video's motion vectors have been distorted by my hacked version of libavcodec:

This is "THE Game Mk II"! concocted with my 'partner in crime' Xianhang "Hang" Zhang. There's ~20k lines of code after I coded for about 2 weeks straight.

For the tutorials of the second-session-first-year C course, I joined the advanced tutorial where we decided our end-of-term goal would be to create a little networked game. hat high hopes we had... (And I still haven't taken down the message board.)

This is the presentation we gave to demonstrate Teh Engine during the final Computer Graphics lecture in front of a full lecture hall of ~300 students.

(Thanks Ashley "Mac-man" Butterworth for operating the camera.) Please excuse my 'ah's and 'um's - it happens when I'm really tired. Was up for >48 hours.

It should be noted that I fixed the physics so the car behaves properly. Please read the previous page for more detail about the engine.

Teh Engine is a graphics engine I wrote in C++ using OpenGL. I boasts many features but can also be considered as re-inventing the wheel. However I believe writing such a complex piece of software is a 'rite of passage' for anyone seriously interested in computer graphics and creating well designed code.

Building on the foundations of the particle simulation, my friend Dom De Re and I decided to dable in the creation of cloth. The parameters are highly tunable and make for interesting results and 'explosions'.

Having come to rest under gravity (the orange particles at the bottom indicate a collision with the floor):

Although I like the basic particle-line aesthetic presented in the previous pictures and videos, I felt it time to add a bit of colour, texture and lighting to the simulation.

Here you can see me tearing my face up in a lit environment:

I re-enabled the skybox in Teh Engine and used an image of the full-saturation hue wheel to give the particles a 'random' colour:

(I think this is reminiscent of the Sony Bravia LCD TV ad!)

Here is a newer video of tearing the cloth and enabling the tornado simulation after reducing the cloth to small fragments:

To make the simulation even more fun I made it possible to tear the cloth by shooting a red bullet at the grid. The red sphere is pulled downward under the influence of gravity and breaks any constraints within its radius.

The early videos follow - click to download them. The later (and much better) videos can be found throughout the following pages.

Tearing the cloth	Colouring the cloth
Running the tornado with connected particles	Blowing in the wind
External view of tearing the cloth	Zoom from view of the tornado to another