Basic concepts

Architecture

Manifest Edit is an Origin feature that can be used to apply custom transformations to dash and HLS manifests.

It can be useful to show a basic architectural drawing to illustrate some concepts related to how manifest transformations are implemented and applied.

The first thing to be noticed is that Manifest Edit applies transformations to manifests by means of a Pipeline Engine.

The second thing to be noticed is that Manifest Edit relies on two external components ("Plugins Library" and "Pipeline configuration file").

A "Plugin" is a tiny bit of manifest editing logic (technically, a python module), usually implementing a self-contained basic manifest manipulation (i.e. move, add or remove an element).

The Pipeline Engine has two responsibilities:

• to create the complete desired manifest editing logic by assembling, at startup, one or more plugins into a complete processing pipeline. This "assembly" is done by following instructions from a provided Pipeline Configuration File.
• to apply the logic implemented by the processing pipeline on every input manifest to produce an output manifest.

The key concept here, enabling the mentioned "flexibility", is that the editing logic effectively applied to manifests is not "coded" into Manifest Edit: it is instead assembled at every manifest_edit launch by loading from disk its elementary components. In other words, manifest_edit is just an empty engine and the kind of manifest transformations that can be achieved are entirely dependent on the Plugins capabilities and on the ways they can be assembled into a Processing Pipeline.

Thus, as a user, you are only concerned in two things:

• knowing the set of manifest editing capabilities provided by each Plugin present in the Plugins Library.
• knowing the syntax of Pipeline configuration files, so that you can compose your custom manifest editing logic.

Both topics are addressed in the following chapter of this manual.

Integration with the Origin

Starting from version 1.11.13, Manifest Edit comes pre-installed with Unified Origin and configuration is more straightforward. Its details are available in our installation docs, but it is useful to remind here the basic steps you need to enable your use case, assuming you are using the configuration suggested in Manifest Edit-related configuration

• write your own my-use-case.yaml Pipeline configuration file (examples available at /usr/share/manifest-edit) and drop it in the /etc/manifest-edit folder
• if your manifest is available at i.e. http://my-video.ism/.mpd, then you can enable your use case and get an edited manifest by accessing the URL http://my-video.ism/.mpd?python_pipeline_config=my-use-case

Tip

When using a ?python_pipeline_config query parameter, the resulting manifest will include media segments URLs that include the same query parameter. This is usually harmless, as the additional query parameter is just ignored. However, you can prevent this by using the option --suppress_query_parameters in your server manifest or using the http://my-video.ism/.mpd?python_pipeline_config=my-use-case&suppress_query_parameters URL.

Tip

If you have rewrite rules in place, make sure they include the QSA flag so that the python_pipeline_config query parameter is not replaced but combined with your rule.

Troubleshooting

If you need to Troubleshoot your setup (e.g. the Origin response has an empty body), the first place to look at is your Origin's apache logs. Manifest Edit in facts integrates in the logs and adds meaningful warning and error message to help you troubleshoot your configuration.

This is an example of log for a common syntax error in the .yaml pipeline configuration file:

E0.088 ERROR during manifest_edit process phase: Exception: Invalid yaml configuration \
            in /etc/manifest-edit/utc_remove.yaml:  mapping values are not allowed here
E0.088   in "<unicode string>", line 2, column 37:
E0.088       - manifest_edit.plugins.mpd.utc_remove:
E0.088                                         ^

Additional warning messages can be present to notify events that you should be aware of, but that do not cause a failure. An example of such warnings is trying to remove a non-existent element from a manifest or trying to introduce a duplicate one.

Once you have solved all the errors, you should be able to examine the returned manifest. In order to confirm that Manifest Edit processing has happened and is the one you expect, the edited manifest includes an extended XML comment field, reporting the exact .yaml configuration applied. Here is an example of how such a comment may look:

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Unified Streaming Platform (version=1.10.27-devel) -->
<!-- Edited by Unified manifest-edit, using the following pipeline:
  mpd:
  - manifest_edit.plugins.mpd.adaptation_sets_switching:
      periods:
      - '*': .*
        adaptationSets:
        - contentType: video
          plugin_config:
            destination: essential_property
            switching_set: '1'
        - contentType: audio
          plugin_config:
            destination: supplemental_property
            switching_set: '2'
  - manifest_edit.plugins.mpd.descriptor_add: null
-->
<MPD>
  Omitted manifest body
</MPD>

Such a comment in a manifest is a confirmation that the mentioned pipeline has been correctly applied.

Command-line usage

If you need to, you can directly use the command-line version of Manifest Edit to apply your use cases to existing manifest files or to dynamically packaged ones.

Manifest Edit requires two input files:

• a Pipeline Configuration file (.yaml format)
• an mpd or m3u8 "main" manifest file or URL

and produces one output

• an edited manifest file

You will learn in the rest of this documentation how to create your own .yaml Pipeline Configuration file. For the moment, you can obtain a sample configuration file to run an example by just copy-pasting the following to an ~/example.yaml file in your home folder.

mpd:
 - manifest_edit.plugins.mpd.utc_add:
     http-iso:
       https://time.akamai.com/?iso

Once you have the .yaml file ready, you can use it with the manifest_edit python script to retrieve a demo .mpd manifest, apply a simple transformation to it and dump the results to stdout. Try and run this command from the same folder where you have saved your example.yaml file:

user@ubuntu:~$ manifest_edit --license_key=/etc/usp-license.key \
  --python_pipeline_config=example.yaml \
  https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.mp4/.mpd

If you see a manifest dumped to stdout, you have successfully run manifest_edit. Otherwise, check your command line options or your .yaml file. You can now compare the edited manifest to the original one and figure out what specific transformation this pipeline has applied:

user@ubuntu:~$ curl https://demo.unified-streaming.com/k8s/features/stable/video/tears-of-steel/tears-of-steel.mp4/.mpd

For your reference, the complete command line usage is the following:

user@server:~$ manifest_edit --h
Usage: manifest_edit [options] [--python_pipeline_config=<filename>] <input>

 --python_pipeline_path=<dirname>     manifest edit pipeline config path.
 --python_pipeline_config=<filename>  manifest edit pipeline config file.
<input>                              manifest input file or URL
[-o outfile]                         output file.
[-v level]                           0=quiet 1=error 2=warning 3=info 4=debug
[--python_path path]                 manifest edit python modules search path.
                                     If not provided, defaults to the current
                                     system PYTHONPATH plus the current dir.

Please note that the input file specification may need quoting when extra
parameters are appended in URL style.

This program is part of USP (Unified Streaming Platform). For more information
and its license, please visit http://www.unified-streaming.com

where:

• <input> can be either a path to a file on disk or an Origin URL. If you wish to, you can also have manifest_edit capture stdin by using the special value stdin: (useful for integration with other scripts).
• <outfile> can be either a path to a file on disk or the special value stdout: to have the manifest rendered to stdout (useful for integration with other scripts).
• <python_pipeline_config> must point to the .yaml file describing the pipeline you want to run.
• optionally, <python_pipeline_path> points to the directory containing the .yaml files. The value of the <python_pipeline_config> parameter will be appended to this path.
• <level> can be used to increase or decreaseg log verbosity.
• <path> can be used to load plugins from an alternative folder. You should never need to modify this to a custom value except for specific purposes (test, troubleshooting) that are not covered in this manual.

Note

On Windows systems, the Manifest Edit python script should be started via the provided manifest_edit.exe included in your installation.