Troubleshooting LIVE Streaming¶
Table of Contents
If you experience a problem with your stream, there are a few steps you can take to determine what the exact issue is that you are experiencing. Sometimes, going through these steps will be enough to solve your problem. Or it might become clear that your issue if one of the more common problems that are described further down below, along with their solutions. And otherwise, these steps will give you the necessary info to open a well-defined ticket in our support portal, so that we can help you as quickly as possible:
- Check license key
- Check publishing point’s state (Live only)
- Check server’s log files
- Check encoder settings
- Check release notes
- Check your setup
All six steps are described in more detail below.
Publishing point’s state¶
This step applies to streaming Live only. To check whether a publishing point is
properly set up, you can check its state. A working publishing point will return
the following from a
/state REST call, with the actual state differing based
on whether a stream has been ingested or not ((see Retrieving additional publishing point information):
<?xml version="1.0" encoding="utf-8"?> <!-- Created with Unified Streaming Platform(version=1.x.x) --> <smil xmlns="http://www.w3.org/2001/SMIL20/Language"> <head> <meta name="state" content="idle"> </meta> </head> </smil>
If a publishing point doesn’t return a state, it might be either that it has not been set up at all, or that there are other problems with your setup (such as access rights). If the state returned by the publishing point is not as expected, the most obvious cause would be a problem with your ingest, either because of problems with the content itself (i.e. encoder settings) or because the webserver isn’t correctly set up to handle the ingest.
The webserver’s log files are the most valuable source for determining the exact nature of the problem that you are experiencing with your stream. These logs allow you to determine whether content is properly ingested (in the case of Live) and whether requests for content are handled correctly (both for Live and VOD). For Apache on Ubuntu, three log files are of importance:
- The general error log (/var/log/apache2/error.log)
- The virtual host access log (configured in the virtual host config)
- The virtual host error log (configured in the virtual host config)
In the virtual host’s access log, you can check how the ingest and requests for content are being handled. For every problem with either the ingest or a request, an error is written to the virtual host’s error log.
Adaptive bitrate video streaming is a delicate process that requires well formed content that has to adhere to many requirements. If those requirements are not met, a wide variety of issues can occur. Check Recommended LIVE Settings and Recommended VOD Settings to determine whether you have configured your encoder correctly. Do note that even with the right settings, there might still be problems with the output of your encoder. Determining whether this is or is not the case requires a detailed inspection of the content itself, with specialized tools (e.g., ffprobe). For streaming Live, make sure that your encoder is supported by checking the overview in our Factsheet.
If you are currently not using the latest version of our software, please check the Release notes to determine whether we introduced a fix for your problem in a newer version.
This error is returned when you make a request for the server manifest (.ism(l)), instead of a playout format. Direct requests for the server manifest are prohibited for security reasons. You should add a virtual path to the URL of the server manifest, specifying the playout format. For more information about the URLs of the different playout formats, please consult Player URLs. .. end
This error typically indicates that your license key is not being loaded correctly. If you are using Apache, you can check if this is the case in the general error log of the webserver, by looking for the following error:
failed to initialize! (Invalid base64 character)
Common causes for this error are the addition of a newline or an erroneous character that has crept in when copying and pasting the license key.
If the error log does not indicate any problems with the license key, your license might not cover the use case that you are trying to achieve.
This error looks like this:
(70007)The timeout specified has expired: Error reading chunk
It means that Unified Origin did not receive any data for a specified amount of time.
This amount of time is equal to the ‘timeout’ value that can be set using Apache’s
mod_reqtimeout module. Note that its default setting might be considered quite
aggressive (10 seconds).
Note also that setting the RequestReadTimeout in a virtual host to override the global settings, does not work. You have to set the global RequestReadTimeout to a larger timeout value, or disable it completely with:
RequestReadTimeout header=0 body=0
Make sure to include the following in your (virtual host) configuration:
The default behaviour of Nginx is to write the POST content to a tmp file in ‘/var/cache/nginx/client_temp’.
For Live streaming this behaviour is wrong as for instance a 24/7 live stream will use all available diskspace until the disk if full.
Nginx provides a few options (the ‘client_body’ options in the core module), but there is no option to turn this off.
Therefore, the nginx module does not hand the POSted data to the next handler and is the sole handler of a POST.
The ‘Stream Closed’ error occurs when the timestamps sent by the encoder are not increasing.
The webserver log will show entries like the following:
[Mon Mar 09 19:19:32 2015] [error] [client 192.168.12.100] pubpoint_push_input_stream(Streams(Video1)) returned: 409 Stream is closed, cannot restart (track_name=video stream.time=41863154666 fragment.time=41598557000)
As a basic requirement, the timestamps of a livestream must increase.
As an additional requirement, which will allow the encoder to start, stop and
restart (when the
restart-on-encoder-reconnect option is enabled for the
publishing point, see Options for LIVE ingest), the encoder needs
to be configured to use UTC timing. Please refer to the encoder manual on how to
configure this (or see Encoder Settings).
The StreamIndex element in the MSS client manifest contains the timestamps, next to the media type (for instance video) or the QualityLevel element indicating the bitrate.
<StreamIndex> <QualityLevel /> <c t="0" d="20000000" /> <c d="20000000" /> <c d="20000000" /> </StreamIndex>
Sometimes you might notice a
@t attribute in the client manifest, which
indicates a gap in the stream. In most cases, this is caused by an interruption of
the ingest of data on the server, but it can be caused by improper encoder
implementations as well.
<c t="0" d="20000000" /> <c d="20000000" /> <c t="60000000" d="20000000" />
In the above example, the length of a full segment is missing after the first two. More information can be found on the IIS blog.
Discontinuities can also be found by looking at the archive of the publishing
point, which you can access using the
/archive REST call, as described in
the Publishing Point API. Ideally, every track should have it start and end
time displayed on one line. For every discontinuity, there will be a new line.
On the origin we see a discontinuity when the timestamp on a next fragment doesn’t equal the timestamp of the previous fragment + its duration. Even if it’s only off by 1/10th of a microsecond.
Although the difference may be very small, this triggers a discontinuity and is noticed in playlist generation and eventually by a player implementation that triggers a decoder reset (introducing ticks/clicks/pops) whenever it sees a timestamp discontinuity. This is especially true for HLS streams that use MPEG-TS.
Therefore, it’s important that the timestamp and duration that are signaled in the ‘tfxd’ box (the one that the encoder is inserting for every fragment) is accurate.
In a more mathematical way:
tfxd[fragment N].timestamp = tfxd[fragment N - 1].timestamp + tfxd[fragment N - 1].duration.
Also worth noting is that the ‘tfxd[fragment N].duration’ should be equal to the sum of the sample durations specified in the ‘trun’ box.