Processing video frames with SciPy / NumPy

How to enable this feature

Please see this page for the pre-requisites and pip install options needed to enable this feature.

Example with explanation

In this section we define a GaussianSmootherBGRA Python class that implements a custom image processing capability. Namely, GaussianSmootherBGRA:

• gets a video frame whose data is formatted in the BGRA colour space,
• smoothes this video frame with a Gaussian kernel,
• and finally passes it further down the video processing pipeline.

The GaussianSmootherBGRA class achieves this by implementing the observer and observable interfaces simultaneously. To this end, it extends the IObservableObserver class exposed by GIFT-Grab:

from pygiftgrab import IObservableObserver
import scipy.ndimage as ndimage

class GaussianSmootherBGRA(IObservableObserver):

    def __init__(self):
        super(GaussianSmootherBGRA, self).__init__()

    def update(self, frame):
        data_np = frame.data(True)
        ndimage.gaussian_filter(input=data_np,
                                sigma=(5, 15, 0),
                                order=0, output=data_np)

The update() method above implements the corresponding method of GIFT-Grab’s observer. This is a crucial component needed for creating custom image processing capabilities using GIFT-Grab. In this particular case, in the update() method implementation we obtain a reference to the actual data of the video frame in the form of a NumPy array by calling the data method. We then apply Gaussian filtering in-place on this NumPy array using the corresponding method of SciPy.

A GaussianSmootherBGRA instance can be attached to an observable video source. This causes that video source object to pass each new video frame to the GaussianSmootherBGRA instance. The GaussianSmootherBGRA instance in return processes that video frame according to the update() method above. It then passes it to any observer object attached to itself.

As a concrete example, consider attaching a GaussianSmootherBGRA object to the file_reader created in the Reading video files section. There is a subtlety, though: we need to initialise this file_reader using the BGRA colour space instead of I420 as in the Reading video files example:

file_reader = sfac.create_file_reader(
    '/tmp/myinput.mp4', ColourSpace.BGRA )

Note that we use ColourSpace.BGRA instead of ColourSpace.I420, which is the only difference to the corresponding line in the Reading video files section. This is because our GaussianSmootherBGRA class expects the video frames to be encoded in the BGRA colour space.

Let’s create an instance of our class and attach it to the file_reader:

gauss = GaussianSmootherBGRA()
file_reader.attach( gauss )

Now the file_reader has started feeding video frames to gauss. To save the Gaussian-smoothed frames to a file, we can attach the file_writer defined in the Encoding video streams in real time section to gauss:

gauss.attach( file_writer )

At this point gauss has started feeding the Gaussian-smoothed video frames to the file_writer. In other words, each video frame read from our video file is Gaussian-smoothed and subsequently encoded to a new video file. This pipeline keeps operating until we detach the observers from the observables:

file_reader.detach( gauss )
gauss.detach( file_writer )

Full source code

Below is the full source code of the example explained above. Please note that depending on the resolution of the video file used and the computational power of your platform, this application might end up processing less frames than are available during its runtime. In other words, the resulting /tmp/myoutput.mp4 file might be shorter than 20 sec.

#!/usr/bin/env python2

from pygiftgrab import IObservableObserver
import scipy.ndimage as ndimage
from pygiftgrab import VideoSourceFactory
from pygiftgrab import ColourSpace
from time import sleep
from pygiftgrab import VideoTargetFactory
from pygiftgrab import Codec


class GaussianSmootherBGRA(IObservableObserver):

    def __init__(self):
        super(GaussianSmootherBGRA, self).__init__()

    def update(self, frame):
        data_np = frame.data(True)
        ndimage.gaussian_filter(input=data_np,
                                sigma=(5, 15, 0),
                                order=0, output=data_np)


if __name__ == '__main__':
    sfac = VideoSourceFactory.get_instance()
    file_reader = sfac.create_file_reader(
        '/tmp/myinput.mp4', ColourSpace.BGRA )

    gauss = GaussianSmootherBGRA()

    tfac = VideoTargetFactory.get_instance()
    frame_rate = file_reader.get_frame_rate()
    file_writer = tfac.create_file_writer(
        Codec.HEVC, '/tmp/myoutput.mp4', frame_rate )

    file_reader.attach( gauss )
    gauss.attach( file_writer )

    sleep( 20 )  # operate pipeline for 20 sec

    file_reader.detach( gauss )
    gauss.detach( file_writer )