Creating scenarios for gst-validate

[This article is a previous rendition of the copy-edited version published at the Samsung OSG blog]

The gst-validate utility, part of the developers' tools offered by GStreamer through the gst-devtols module, allows for the detection of known issues in GStreamer pipelines. In this brief guide, I will show you how to create playback scenarios to test your pipeline's reaction to a new set of controlling actions.

There are a couple of common and not-so-common scenarios already included with the GStreamer validate suite. This set, allows for the identification of known error conditions on a running pipeline subjected to the actions expressed by the provided scenarios. Now, what if you are after inspecting the reaction of your pipeline to a new set of actions? Then you need a new scenario that describes it.

Scenarios are built from serialized actions on a .scenario file. These actions and their parameters are expressed using a GstStructure-based text format. The current list of actions includes the following core ones: seek, pause, switch-track, wait, dot-pipeline, set-feature-rank, set-state, set-property, set-debug-threshold, emit-signal, disable-plugin; and a few others implemented by gst validate plugins, like: validatefaultinjection: corrupt-socket-recv: validategtk: gtk-put-event: validate-launcher: set-subtitle. For this introductory article we will be dealing only with a few of them so you get an idea of how everything fits together.

Let's say we want to test the resilience of a pipeline to a flush seek to 10s after the pipeline is paused at playback-time 1s for 5s. Lets also say that we want to stop playback at 20s. A preliminary version of our scenario file will look like:

description, seek=true, need-clock-sync=true
pause, name=Our-pause, playback-time=1.0, duration=5.0
seek, name=Our-forward-seek, playback-time=6.0, start=10.0, flags=accurate+flush
stop, playback-time=20.0

Whenever you run into a new action name (first member on each line), use the following command to get a full description of what the action does and what parameters you can use to control it:

gst-validate-1.0 -t

For example, the "description" line above talks about two important aspects of our scenario:

seek=true: This scenario performs seek operations
need-clock-sync=true: The scenario execution needs to be synchronized to the pipeline's clock

Pretty straightforward isn't it? Now, we do need to add some boilerplate for this scenario to be fully operational. For example, this one will likely not provide the results we want if our media has a duration that doesn't allow a proper forward seek at 5s to 10s. For the sake of example then, let's inform the scenario runner the test media should have a minimum duration of 30 seconds. We do this using the following "description" line:

description, seek=true, need-clock-sync=true, min-media-duration=30s

The scenario parsing system assumes each line on the scenario file represents an action but we can use backslashes to enter multi-line ones. Here's how our scenario file looks with a multi-line *description* action (sic) with a long "summary":

description, seek=true, need-clock-sync=true, min-media-duration=30s, \
summary="This example scenario seeks forward to 10s after pausing playback for 5. \
Then, it stops the execution at 20s"
pause, name=Our-pause, playback-time=1.0, duration=5.0
seek, name=Our-forward-seek, playback-time=6.0, start=10.0, flags=accurate+flush
stop, playback-time=20.0

By the way, we have been carrying this one over for a while, but "description" is not really an action. Sorry to break it to you but please consider this an implementation detail you will have to live with.

Now that we know what to write, we need to save this scenario script to a file named after the scenario. In this particular example, we will save the above content as test-seek.scenario and pass this file name (without extension) directly to gst-validate for testing a pipeline:

gst-validate-1.0 --set-scenario test-seek filesrc location=test20sVP9.mkv ! matroskademux ! vp9dec ! autovideosink

Beyond passing the full PATH to the scenario file to --set-scenario, you can also put this file somewhere else and make the GST_VALIDATE_SCENARIOS_PATH environment variable point to that location. These are not even the only options but both should get you going with the above example.

I assume you might be using your own pipeline to try this procedure out but if you want to follow the above example to the letter, you will need to download the test20sVP9.mkv sample file. Additionally if the syntax used to express the pipeline is not something you are familiar with you can find our best attempt at an explanation on the gst-launch documentation page.

And that's it. gst-validate will timely subject the pipeline to the actions we scripted via our scenario file, and will provide a report on whether any of the built-on gst-validate tests failed. Beyond a simple statement of problem, sometimes gst-validate will even provide hints towards a full diagnostic.

In this article we briefly discuss the scenario-writing aspect of gst-validate. While the suite of tools goes far beyond, a good understanding of the steps required to express action scenarios, provides a great foray into this GStreamer testing system. Moving forward, If you want additional examples, take a look at the real-life ones being included with gst-validate and do consider contributing your own by submitting a patch to the gst-devtools component on the project's buzilla.

Good luck!