Wednesday, June 8, 2011

Overlaying text and images into the video stream

Having tinkered around with the toolchain for a while I have turned my attention to one of the core tasks of the project - that is overlaying text and images in the video stream.

The video capture, encoding and motion detection are all handled by the plmedia.o module; fortunately the source code for this module is provided in the toolchain, although there is no documentation.

After a fair bit of scratching my head it is apparent that the video capture hardware writes directly to memory on demand (by the module) and then a further request is sent to the encoder hardware which encodes/compresses it directly to another buffer. This is the buffer which is used by the device when stremaing video. There is further hardware which performs motion detection, using the video capture buffer.

I decided to inject into the video stream directly before the encoding takes place in order to be sure (with as much certainty as possible) that the frame has been captured in full.

plmedia.o source is made up of 3 core components: plgrabber, plencoder and plmd. plencoder is the component responsible for managing the encoding process and this is where I have added my code.

The video is captured in YCbCr format with 4.1.1 ratio between the luminence and the chroma channels - that is, the Y channel is at 640x480 resolution with 8 bpp and the Cb & Cr channels are at 320x240 resolution with 8 bpp.

Each channel is captured simultaniously in its own buffer, I added an IOCTL command to pass a structure which contains the dimensions of the image to be added, a TTL (so images can expire) and pointers to buffers which contain appropriate images. The definitions for the IOCTL command and the buffer can be found in the source below in include/video/plmedia.h.

The heavy lifting is done in the plencoder.c source; look for a function called OverlayImages which iterates through the image buffers before each encode - it is interesting to note that I was having huge issues with image corruption which took some real head scratching to solve - the images would appear to be corrupt with some elements of the video image showing through - it turns out that the processor has a little cache which needs to be flushed otherwise it doesn't write to the buffer before the encoder kicks in.

In EncIOCTL you will find the new function to add an image to the buffer list - there is functionality to replace an existing buffer or add a new one - the replace function is useful for adding animation or moving images to the buffer (such as a time stamp) without needlessly wasting resources.

Finally in InitEncoder and CleanupEncode is addional code to create and release the buffers.

There is an arbitary limit of upto 6 independent overlays and this can be increased by modifying the IMAGEBUFFERS constant in the plencoder.h

The image buffers must be passed in the right format - and this is where it can get tricky - images need to be converted to YCbCr format and at the right resolution for each channel - the co-ordinates and dimensions refer to the Y channel and are appropriatly converted for the CbCr channels.

I have also created an applet to add text to the video stream as part of the babycam project; the code is attached below. It uses a pre-created BMP consisting of a matrix of characters white on black. The code extracts the relevant characters, builds a buffer, converts it and then sends it to the module. The same code can be used to send a BMP although it needs to be in 24bpp uncompressed format (regular windows .bmp's are fine).

The sample below shows a screen grab of the the module working with 2 overlays added by the displaytext applet - although they are in black and white there is full colour support.

And here is the same scene being streamed to an iPad:

The source and cross compiled code for the module can be downloaded here

The source and cross compiled code for the text overlay applet can be downloaded here


  1. The source code links seem to be missing. Also, do you have details on reproducing the toolchain?

  2. Sorry, I got ahead of myself, source code links now work.

  3. I will post another entry shortly with steps to get the toolchain working - its quite a painful process...