APIs for media synchronization

Support in HbbTV

Available since: HbbTV 2.0.1 (ETSI TS 102 796 V1.4.1)

Introduction (informative)

This clause defines the API for multi-stream synchronization as defined in clause 10.2.8 and the API for the inter-device synchronization with a media presentation on companion screens.

As the architecture for multi-stream and inter-device synchronization is basically the same, the MediaSynchroniser embedded object defined in clause 8.2.3.2 is the main object for both synchronization features. The sequence diagrams in clause 13.8 provide further indication how the API interacts with the architecture entities.

Example usage for multi-stream synchronization:

Example usage for inter-device synchronization:

JavaScript snippets on HbbTV® terminal:

The MediaSynchroniser embedded object

General

The terminal shall support a non-visual embedded object of type “application/hbbtvMediaSynchroniser” with the following JavaScript API. The terminal shall support one operable MediaSynchroniser embedded object. If an application creates multiple instances only the last one created needs to be operable. A MediaSynchroniser can be used for multi-stream synchronization and for inter-device synchronization.

Properties

function onError(Number lastError, Object lastErrorSource)

Description

The function that gets called when an error occurs for this MediaSynchroniser object. The terminal shall pass two arguments in the call.

The first argument shall be the error code as defined in clause 8.2.3.2.4.

The second argument shall be the media object that was the cause of the error. If the error was not caused by a media object, then this shall be null.

function onSynchroniserInitialised()

Description

The function that gets called when the initialization of this MediaSynchroniser object is successfully completed – specifically after the timeline is being monitored and the currentTime property has started to be updated.

function onMediaObjectAdded(Object mediaObject)

Description

The function that gets called when a call to the addMediaObject method is successfully completed. Specifically, component selection has completed and either:

· the set of selected components will not change; or

· the changed set of components have started to be presented synchronized (timeupdate events are being generated due to “the usual monotonic increase of the current playback position during normal playback” as defined in HTML5 [54]).

The terminal shall pass one argument in the call, the media object that was passed as an argument to addMediaObject and that has been successfully added.

function onSyncNowAchievable (Object mediaObject)

Description

The function shall be called by the terminal if it was previously not possible for the terminal to time presentation of this media to synchronize it to the master media but now it has become possible and the terminal has started to do so.

The argument shall be the media object for which synchronized presentation timing can now be achieved.

This function is only called (with a given media object passed as argument) after an earlier transient error of the MediaSynchroniser with error code 1 or 11 as appropriate (relating to the same media object).

readonly Number lastError

Description

If no error has yet occurred for this MediaSynchroniser object then the value of this property shall be null, otherwise the value shall be the code of the last error that occurred for this MediaSynchroniser object as defined in clause 8.2.3.2.4.

readonly Object lastErrorSource

Description

Shall be the media object that was the cause of the last error. If no error has yet occurred for this MediaSynchroniser object, or if the error was not caused by a media object, then this shall be null.

readonly Number nrOfSlaves

Description

If this MediaSynchroniser is being used for inter-device synchronization, it shall be the number of current web socket connections on the CSS-CII service endpoint provided by the terminal. Otherwise it shall be null.

readonly Boolean interDeviceSyncEnabled

Description

Shall be true if and only if the terminal is currently a master terminal for inter-device synchronization.

readonly Number maxBroadbandStreamsWithBroadcast

Description

The number of broadband streams the terminal supports for multi-stream synchronization if one stream in the multi-stream synchronization is a broadcast.

readonly Number maxBroadbandStreamsNoBroadcast

Description

The number of broadband streams the terminal supports for multi-stream synchronization if there is no broadcast stream in the multi-stream synchronization.

readonly Number currentTime

Description

The value of this property shall be the current playback position of the master media, as defined in clause 13.11.3.

If the MediaSynchroniser is not initialized or if the MediaSynchroniser has transitioned to the permanent error state, the value of this property shall be NaN

When the value is not NaN, it shall be expressed in units of seconds and fractions of a second and shall meet the precision requirements defined in clause 13.11.3.

String contentIdOverride

Description

This value overrides the content ID that would normally be reported to Companion Screen Applications during inter-device synchronization.

When the terminal is a master terminal and inter-device synchronization functionality is enabled and the value of this property is a string then the content ID that the terminal uses for the CSS-CII service endpoint and the CSS-TS service endpoint is overridden and the value of this property is used instead.

If the value of this property is null or undefined then there is no override.

This behaviour is defined in clauses 13.6.2 and 13.8.2.

The value of this property shall initially be null.

Methods

void initMediaSynchroniser() (Object mediaObject, String timelineSelector)

Description

Initializes a MediaSynchroniser for multi-stream synchronization and for inter-device synchronization.

The media object becomes the master media (see clause 13.2.4) and the timeline selected for it defines the timeline used for the MediaSynchroniser API when performing multi-stream synchronization and inter-device synchronization as explained in clause 13.4.3.

The media object specified as the parameter to this method shall be automatically added to this MediaSynchroniser and therefore cannot subsequently be explicitly added using the addMediaObject() method.

If the MediaSynchroniser has already been initialized (including if it is in a permanent error state) then this call shall fail and an error event shall be triggered with error code 13 or 17 (according to the definition of the error codes).

Initialization is immediate, however if at that time, or subsequently, if the media stream for the media object is determined to be not available or if the selected timeline is determined to be not available then this shall result in a permanent error of the MediaSynchroniser and an error event shall be triggered with error code 15 or 16 (according to the definition of the error codes).

If this method completes successfully without an error event being triggered then the MediaSynchroniser shall be considered initialized and a SynchroniserInitialised event shall be triggered after the currentTime property starts being updated.

In some early implementations where SynchroniserInitialised event is not supported,there is no explicit event indicating a successful call to this method. Applications may detect success by polling the currentTime property and detecting when it changes from NaN to a numeric value.

When this MediaSynchroniser is initialized, if there is an existing MediaSynchroniser that has already been initialized and is not in permament error state already then this shall result in a permanent error of the existing MediaSynchroniser and it shall trigger an error event with error code 18.

Arguments

mediaObject

The media object (video/broadcast object or HTML5 media object) that carries the timeline that will be used by the MediaSynchroniser API.

timelineSelector

Type and location of the timeline to be used by the MediaSynchroniser API. Please refer to clause 13.4.

NOTE: The availability of a timeline can sometimes only be determined some time after the terminal has begun presentation of the stream. This error can therefore occur some time after the MediaSynchroniser is initialized. See clause 9.7.3.

void addMediaObject (Object mediaObject,

String timelineSelector,

CorrelationTimestamp correlationTimestamp,

Number tolerance,

Boolean multiDecoderMode)

Description

Adds a media object, i.e. video/broadcast object or HTML5 media object, to the MediaSynchroniser. If the MediaSynchroniser was initialized with the initMediaSynchroniser() method, or if inter-device synchronization has been enabled, then the terminal shall start to synchronize the media object to other media objects associated to this MediaSynchroniser as a result of this method call. The behaviour of the media objects when this method is called is defined in clause 9.7.1.

If the MediaSynchroniser is not initialized, or is in a permanent error state, then this call shall be ignored and an error event dispatched with error code 7 or 13 (according to the definition of the error codes).

If the media object has already been added to the MediaSynchroniser (either by passing it to addMediaObject() or initMediaSynchroniser() methods), then this call shall be ignored and an error event dispatched with error code 4.

If adding the media object would result in multi-stream synchronization using a combination of streams that is unsupported by the terminal, then this call shall be ignored and a transient error of the MediaSynchroniser shall be generated with error code 20.

The terminal may be required to buffer one or more of the media objects. If the terminal has insufficient buffer space or cannot present the media sufficiently early then the media object shall be added to the MediaSynchroniser but a transient error of the MediaSynchroniser shall be generated with error code 1 or 11.

The terminal shall select the components from the media object to be presented in accordance with the value of the multiDecoderMode parameter and the definitions in clause 10.2.7.

If the terminal fails to access a media item or its timeline, e.g. the resource is not available, then adding the media object shall fail and the MediaSynchroniser shall dispatch an error event with error code 2 or 3 (according to the definition of the error codes).

If the correlation timestamp correlationTimestamp is undefined a correlation timestamp where the value of both properties is 0 shall be assumed. If the correlation timestamp is null or has an invalid format, adding the media object shall fail and the terminal dispatch an error event with error code 5.

The method is asynchronous. The process of determining any changes in the set of media components to be presented might not be completed before the method call returns. The actual changes in the content being presented will probably be delayed while the terminal de-initializes and re-initializes media decoders and aligns the master media object and the other media object(s) to achieve synchronized presentation in accordance with the correlation timestamps.

If the method completes successfully without an error event being dispatched then a MediaObjectAdded event shall be dispatched.

In some early implementations where MediaObjectAdded is not supported,there is no explicit event indicating a successful call to this method. In these implementations, if the set of selected components changes to include ones in the added media object, applications may detect success implicitly by registering for events on the media object being added. For an HTML5 media element that is paused:

· Firing of a playing event without any subsequent error will indicate that a component will be presented soon (e.g. within 250 ms – 2 s).

· Firing of timeupdate events due to usual monotonic increase of the current playback position during normal playback (see the “time marches on steps” in HTML5 [54]) will indicate that a component is being presented. Applications need to be careful to distinguish these timeupdate events from timeupdate events fired for other reasons.

In these early implementations, there is no implicit event enabling applications to detect success if the set of selected components does not change to include ones in the added media object.

The rules stated in clause 10.2.8.3 shall apply.

Arguments

mediaObject

video/broadcast object or an HTML5 media object

timelineSelector

Type and location of the timeline used for the media object’s timeline. Please refer to clause 13.4.

correlationTimestamp

An optional initial correlation timestamp that relates the media objects timeline to the synchronization timeline.

tolerance

An optional synchronization tolerance in milliseconds. The tolerance, if provided, is expressed as a positive value including 0. If the application passes a negative value or does not provide this argument then the terminal shall assume a tolerance of 0.

See clause 9.7.2 for details on how a terminal uses this value.

This argument does not define the accuracy of synchronization to be achieved by the terminal, such as frame accuracy or lip‑sync accurate.

multiDecoderMode

An optional parameter that defines whether component selection for this media object is performed separately (as defined in clause 10.2.7.3) or collectively with other media objects on this MediaSynchroniser (as defined in clause 10.2.7.4).

If the value does not equal true, the terminal shall follow the rules for component selection in clause 10.2.7.3.

If the value is true and the terminal does not support multiple decoders or currently does not have sufficient resources it shall ignore the call and dispatch an error event with the error code 2. Otherwise the terminal shall follow the rules in clause 10.2.7.4.

Applications should check the extraSDVideoDecodes and extraHDVideoDecodes properties before using this parameter set to true.

void removeMediaObject (Object mediaObject)

Description

Removes an object from this MediaSynchroniser.

Using this method to remove all media objects added to a MediaSynchroniser using addMediaObject shall terminate multi-stream synchronization but shall leave the timeline specified when the MediaSynchroniser was initialized being monitored. Applications shall be able to continue to read the value of the timeline from the currentTime property.

Using this method to remove the master media object (specified in the call to initMediaSynchroniser) shall stop timeline monitoring and release all scarce resources associated with this MediaSynchroniser. The MediaSynchroniser object shall enter the permanent error state and en error event shall be triggered with error code 18.

The terminal shall not stop the presentation of media objects purely as a result of this method call. However, if there are insufficient decoders available to continue to present this media object, the presentation of the media object may cease due to insufficient resources.

If the media object has not already been added to the MediaSynchroniser and is not the master media object then this call shall be ignored and an error event dispatched with error code 8.

If the MediaSynchroniser is not initialized, or is in a permanent error state, then this call shall be ignored and an error event dispatched with error code 7 or 13 (according to the definition of the error codes).

Arguments

mediaObject

The media object to be removed.

void updateCorrelationTimestamp (Object mediaObject,

CorrelationTimestamp correlationTimestamp)

Description

Updates the correlation timestamp for the media object, which relates the media object’s timeline to the timeline used by the MediaSynchroniser API.

A correlation timestamp should be updated as often as necessary if the correlation of the timelines changes, e.g. due to drift or discontinuity event of one timeline.

If the correlation timestamp is null or has an invalid format then this call shall be ignored and an error shall be dispatched with error code 5.

When an updated correlation timestamp is set for a media object which would change the temporal relative presentation of the media objects, the terminal shall adapt to this. Example mechanisms for achieving this gracefully can be found in clause C.3 of ETSI TS 103 286‑2 [47].

If the change to the correlation timestamp would cause the terminal to run out of buffer space or be unable to present the media sufficiently early then an error shall be dispatched with error code 1 or 11 as appropriate.

If the change to the correlation timestamp would cause the terminal to be able to achieve synchronized presentation of this media object when previously it could not, then an onSyncNowAchievable event shall be generated for the MediaSynchroniser object (see clause 8.2.3.2.3).

If the media object either is not already added to the MediaSynchroniser or is the master media object, then this call shall be ignored and an error event dispatched with error code 8.

If the MediaSynchroniser is not initialized, or is in a permanent error state, then this call shall be ignored and an error event dispatched with error code 7 or 13 (according to the definition of the error codes).

Arguments

mediaObject

The media object for which the correlation timestamp shall be set.

correlationTimestamp

The correlation timestamp to be used with this media object.

void enableInterDeviceSync(function callback)

Description

Enables inter device synchronization . If it is already enabled then this call shall be ignored.

On calling this method, the terminal become a master terminal as defined in clause 13.3.3.

The callback method shall be called when the endpoints are operable.

The nrOfSlaves property can be used to poll for the number of connected companion screen applications.

If the MediaSynchroniser is not initialized, or is in a permanent error state, then this call shall be ignored and an error event dispatched with error code 7 or 13 (according to the definition of the error codes).

Arguments

callback

Optional callback function.

void disableInterDeviceSync(function callback)

Description

Disables the inter device synchronization.

If the terminal is a master terminal it shall cease to be a master terminal as defined in clause 13.3.4. Once the terminal is no longer a master terminal then the callback function shall be called.

If the MediaSynchroniser is initialized but the terminal is currently not a master terminal then the callback function shall be immediately called.

If the MediaSynchroniser is not initialized, or is in a permanent error state, then this call shall be ignored and an error event dispatched (see clause 13.3.8) with error code 7 or 13 (according to the definition of the error codes).

Arguments

callback

Optional callback function.

DOM2 events

When an error occurs for the MediaSynchroniser object then the intrinsic event onError shall be generated and a corresponding DOM level 2 event shall be generated in the following manner:

Intrinsic eventCorresponding DOM 2 eventDOM 2 Event properties
onErrorErrorBubbles: No Cancelable: No Context Info: lastError, lastErrorSource

For the intrinsic event onSyncNowAchievable , a corresponding DOM level 2 event shall be generated, in the following manner:

Intrinsic eventCorresponding DOM 2 eventDOM 2 Event properties
onSyncNowAchievableSyncNowAchievableBubbles: No Cancelable: No Context Info: mediaObject

For the intrinsic event onSynchroniserInitialised, a corresponding DOM level 2 event shall be generated, in the following manner:

Intrinsic eventCorresponding DOM 2 eventDOM 2 Event properties
onSynchroniserInitialisedSynchroniserInitialisedBubbles: No Cancelable: No Context Info: None

For the intrinsic event on onMediaObjectAdded, a corresponding DOM level 2 event shall be generated, in the following manner:

Intrinsic eventCorresponding DOM 2 eventDOM 2 Event properties
onMediaObjectAddedMediaObjectAddedBubbles: No Cancelable: No Context Info: mediaObject

NOTE: These DOM 2 events are directly dispatched to the event target, and will not bubble nor capture. Applications SHOULD NOT rely on receiving these events during the bubbling or the capturing phase. Applications that use DOM 2 event handlers have to call the addEventListener() method on the MediaSynchroniser embedded object. The third parameter of addEventListener(), i.e. “useCapture”, will be ignored.

Error codes

The values of the error property are defined in this clause.

Value

Description

Permanent or Transient

1

Synchronization is unachievable because the terminal could not delay presentation of content (represented by a media object added using the addMediaObject() method) sufficiently to synchronize it with the master media. For example: the buffer size for media synchronization is not sufficient.

Transient

2

The presentation of media object(that was added using the addMediaObject() method) failed. The specific reason is given by the error handler of that media object.

Transient

3

The media or the selected timeline for the media could not be found or the media timeline is no longer present (for media represented by a media object that was added using the addMediaObject() method).

Transient

4

Media object is already associated with the MediaSynchroniser.

Transient

5

The correlation timestamp set for a media object is null or has an invalid format.

Transient

6

Not used.

7

The call failed because the MediaSynchroniser is not yet initialized.

Transient

8

The media object has not been added to the MediaSynchroniser using the addMediaObject() method.

Transient

9

The media object (that was passed using the addMediaObject() method) is not in a suitable state to participate in synchronization. See clause 9.7.1.

Transient

10

Not used.

11

Synchronization is unachievable because the terminal could not present the content (represented by a media object added using the addMediaObject() method) sufficiently early to synchronize it with the master media.

Transient

13

The method call failed because the MediaSynchroniser is in a permanent error state or because it has been replaced by a newer initialized MediaSynchroniser.

Transient (see note)

14

The presentation of the master media (that was specified as an argument when the initMediaSynchroniser() method was called) failed. The specific reason is given by the error handler of that media object.

Permanent

15

Either the master media object or the selected timeline for the master media object (that were specified as arguments when the initMediaSynchroniser() method was called) could not be found or the media timeline is no longer present.

Permanent

16

The master media object is not in a suitable state to participate in synchronization. See clause 9.7.1.

Permanent

17

The method call failed because the MediaSynchroniser is already initialized.

Transient

18

This MediaSynchroniser has been replaced by a new MediaSynchroniser being initialized.

Permanent

19

Not used.

20

The combination of streams requested for multi-stream synchronization (by a call to the addMediaObject() method) is unsupported.

Transient

NOTE: The MediaSynchroniser will already be in a permanent error state. If this error occurs, the MediaSynchroniser remains in the permanent error state.

Behaviour of the MediaSynchroniser object depending on whether the error is transient or permanent is defined in clauses 13.3.7 and 13.3.8.

The CorrelationTimestamp class

General

Applications shall construct objects that conform to this class definition to deliver correlation timestamps to the terminal. Any object conforms to this class definition if it includes the named properties described in this class definition.

Properties

Number tlvMaster

Description

A value on the timeline used by the MediaSynchroniser API.

Number tlvOther

Description

A value on the timeline of a media object that correlates to the value on the timeline used by the MediaSynchroniser API.

The significance of the properties of this class for both multi-stream and inter-device synchronization are discussed in more detail in clause 13.4.