Basic concepts

Architecture

Manifest Edit is a command line utility to apply a custom transformation to a dash manifest.

It can be useful to show a basic architectural drawing to illustrate some concepts related to how manifest transformations are implemented and applied.

The first thing to be noticed is that Manifest Edit applies transformations to dash manifests by means of a Pipeline Engine.

The second thing to be noticed is that Manifest Edit relies on two external components ("Plugins Library" and "Pipeline configuration file") that are totally separated from the Manifest Edit executable.

A "Plugin" is a tiny bit of manifest editing logic (technically, a python module), usually implementing a self-contained basic manifest manipulation (i.e. move, add or remove an element).

The Pipeline Engine has two responsibilities:

• to create the complete desired manifest editing logic by assembling, at startup, one or more plugins into a complete processing pipeline. This "assembly" is done by following instructions from a provided Pipeline Configuration File.
• to apply the logic implemented by the processing pipeline on every input manifest to produce an output manifest.

The key concept here, enabling the mentioned "flexibility", is that the editing logic effectively applied to manifests is not "coded" into Manifest Edit: it is instead assembled at every manifest_edit launch by loading from disk its elementary components. In other words, the manifest_edit executable is just an empty engine and the kind of manifest manipulations that can be achieved are entirely dependent on the Plugins capabilities and on the ways they can be assembled into a Processing Pipeline.

Thus, as a user, you are only concerned in two things:

• knowing the set of manifest editing capabilities provided by each Plugin present in the Plugins Library.
• knowing the syntax of Pipeline configuration files, so that you can compose your custom manifest editing logic.

Both topics are addressed in the following chapter of this manual.

Integration with the Origin

Manifest Edit comes in the form of a command-line executable. Integration with the Origin is performed via Apache's mod_ext_filter module, which allows to have the generated manifest be handled by an external manifest-edit process via stdin/stdout. You can refer to the Apache mod_ext_filter documentation to understand how Apache can be configured to associate specific HTTP requests to the desired processing pipeline. Examples of possible configurations are also available in the Enabling Manifest Edit chapter.

Note

The initial implementation uses mod_ext_filter which as mentioned in the Apache documentation doesn't necessarily have the best performance, but allowed us to prove the concept and functionality. We are currently investigating options for a more performant implementation, for example writing a native Apache filter module. Even though every effort will be done to avoid it, it is possible that future versions of Manifest Edit will introduce changes that will break backward compatibility with your pipeline or apache configuration files.

HTTP Return codes

Due to the nature of mod_ext_filter, which does not check the return code of the external process, Unified Origin will always return a 200 OK http response code for requests associated to a pipeline, even if the pipeline processing was not successful (e.g. because the syntax of the pipeline configuration file is not correct). In such cases, failures in the Manifest Edit pipeline can only be detected by the empty response body.

Troubleshooting

If you need to Troubleshoot your setup (e.g. the Origin response has an empty body), the first place to look at is your Origin's apache logs. Manifest Edit in facts integrates in the logs and adds meaningful warning and error message to help you troubleshoot your configuration.

This is an example of log for a common syntax error in the .yaml pipeline configuration file:

E0.088 ERROR during manifest_edit process phase: Exception: Invalid yaml configuration \
            in /usr/share/manifest-edit/utc_remove.yaml:  mapping values are not allowed here
E0.088   in "<unicode string>", line 2, column 37:
E0.088       - manifest_edit.plugins.mpd.utc_remove:
E0.088                                         ^

Additional warning messages can be present to notify events that you should be aware of, but that do not cause a failure. An example of such warnings is trying to remove a non-existent element from a manifest or trying to introduce a duplicate one.

Once you have solved all the errors, you should be able to examine the returned manifest. In order to confirm that Manifest Edit processing has happened and is the one you expect, the edited manifest includes an extended XML comment field, reporting the exact .yaml configuration applied. Here is an example of how such a comment looks:

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Unified Streaming Platform (version=1.10.27-devel) -->
<!-- Edited by Unified manifest-edit, using the following pipeline:
  mpd:
  - manifest_edit.plugins.mpd.adaptation_sets_switching:
      periods:
      - '*': .*
        adaptationSets:
        - contentType: video
          destination: essential_property_add
          switching_set: '1'
        - contentType: audio
          destination: supplemental_property_add
          switching_set: '2'
  - manifest_edit.plugins.mpd.supplemental_property_add: null
  - manifest_edit.plugins.mpd.essential_property_add: null
-->
<MPD>
  Omitted manifest body
</MPD>

Such a comment in a manifest is a confirmation that the mentioned pipeline has been correctly applied.

Command-line usage

If you need to, you can directly use the command-line version of Manifest Edit to apply your use cases to existing .mpd files or to dynamically packaged ones.

Manifest Edit requires two input files:

• a Pipeline Configuration file (.yaml format)
• an mpd manifest file or URL

and produces one output

• an edited mpd manifest file (or stdout)

You will learn in the rest of this documentation how to create your own .yaml Pipeline Configuration file. For the moment, you can obtain a sample configuration file to run an example by just copy-pasting the following to an ~/example.yaml file in your home folder.

mpd:
 - manifest_edit.plugins.mpd.utc_add:
     http-iso:
       https://time.akamai.com/?iso

Once you have the .yaml file ready, you can use it with the manifest_edit executable to retrieve a demo .mpd manifest, apply a simple transformation to it and dump the results to stdout. Try and run this command from the same folder where you have saved your example.yaml file:

user@ubuntu:~$ manifest_edit --license_key=/etc/usp-license.key \
  --python_pipeline_config=example.yaml \
  https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel.mp4/.mpd

If you see a manifest dumped to stdout, you have successfully run manifest_edit. Otherwise, check your command line options or your .yaml file. You can now compare the edited manifest to the original one and figure out what specific transformation this pipeline has applied:

user@ubuntu:~$ curl https://demo.unified-streaming.com/video/tears-of-steel/tears-of-steel.mp4/.mpd

For your reference, the complete command line usage is the following:

user@server:~$ manifest_edit --h
Usage: manifest_edit [options] --python_pipeline_config=<filename> <input>

--python_pipeline_config=<filename>  manifest edit pipeline config file.
<input>                              .mpd manifest input file or URL
[-o outfile]                         output file.
[-v level]                           0=quiet 1=error 2=warning 3=info 4=debug
[--python_path path]                 manifest edit python modules search path.
                                     If not provided, defaults to the current
                                     system PYTHONPATH plus the current dir.

Please note that the input file specification may need quoting when extra
parameters are appended in URL style.

This program is part of USP (Unified Streaming Platform). For more information
and its license, please visit http://www.unified-streaming.com

where:

• <input> can be either a path to a file on disk or an Origin URL. If you wish to, you can also have manifest_edit capture stdin by using the special value stdin: (useful for integration with other scripts).
• <outfile> can be either a path to a file on disk or the special value stdout: to have the manifest rendered to stdout (useful for integration with other scripts).
• <python_pipeline_config> must point to the .yaml file describing the pipeline you want to run.
• <level> can be used to increase or decreasing log verbosity
• <path> can be used to load plugins from an alternative folder. You should never need to modify this to a custom value except for specific purposes (test, troubleshooting) that are not covered in this manual.

Note

On Windows systems, the executable name of Manifest Edit is manifest_edit.exe