HbbTV application testing and verification

Just a Web app, right?

HbbTV apps are essentially Web apps. This enables application developers to leverage Web applications development environments in order to build, test and deploy HbbTV applications. Testing and prototyping process is accelerated by different browser plugins and emulators which (partially) implement HbbTV specifics on the PC. See Resources section for a list of useful tools.

Navigating the specifics of HbbTV

However, there are specific challenges developers need to overcome in order to ensure a good user experience of their HbbTV apps, namely:

Varying hardware and software environments across terminals

HbbTV applications run on a multitude of terminal models, each with varying hardware performance and capabilities, as well as quality of software implementations. This mixture of runtime environments can lead to discrepancies in how a HbbTV application behaves across different devices. 

A key difference between terminals is the hardware performance. The differences observed can be most significant. It is therefore wise to keep one’s apps as simple as possible and with minimum memory footprint. If fast CPU or lots of memory are required to run the envisioned application it can be a good strategy to perform some kind of whitelisting of the terminals that are known to run the application in a satisfactory fashion and abort execution on other terminals in order to avoid terminal crashes and unresponsiveness.

It’s very important to resist the temptation to simply acquire the newest models from the top two manufacturers in your market and develop on those. They will not be representative of the older models from those manufacturers let alone models from other manufacturers. You will be creating a problem for yourself later on when you start testing on a wider set of terminals.

If running on older terminal models is important (see later), specific issues for older products include graphics performance and amount of available memory and, for much older TV sets, versions of TLS.

Identifying terminal manufacturers and models

Since there are many HbbTV terminal manufacturers and models, it is wise to prioritize testing on a number of the most popular or widely used terminal models to cover a significant portion of the user base. Every market is specific in that regard, so it is important to gather information on the installed HbbTV terminal base. This is accomplished by analyzing User-Agent header information sent by the terminals. All outgoing HTTP requests made on behalf of an HbbTV application running on a terminal include a User-Agent header using the following syntax:

The version parameter indicates the HbbTV standard version supported by the terminal. Since there are multiple versions of the standard, with significant differences in capabilities, any application should adopt one of the two following strategies:

• Require a minimum version and abort execution on older version terminals
• Dynamically adapt to different versions of the standard supported by the terminal

The capabilities parameter signals optional features implemented on the terminal (e.g. DRM support).

The optional vendorName, modelName, softwareVersion, hardwareVersion parameters which are not always present but commonly, together with full User-Agent string, contain enough information to ascertain the terminal model family. 

Access to a variety of TVs

Developers will need access to a representative terminal base. For many developers, this will be a mixture of having some models in-house combined with use of an external so-called “receiver zoo”. There are a few of these around and at least one of these has some kind of remote access. If you need to test on older products then an external receiver zoo may be the only way of doing this unless you have some available receivers from previous HbbTV or SmartTV projects.

In order to define what would be needed for a representative terminal base, following questions should be answered:

• Which markets are targeted for the app?
• Which TV manufacturers have the largest market share in those markets and which product families do they have? (If you are developing the app under contract to a broadcaster then they may be able to help with these. If no data is available, you can measure the data supplied within the User-Agent header as explained above).
• Which TVs from the leading manufacturers do you already have? How much can you afford to buy more TVs? What are the gaps in what you already have?
• What percentage of the market does your customer want your app to reach? The higher the proportion of the market, the more you will need to look at TV manufacturers with a smaller market share and/or older products from main manufacturers.
• If you are developing the app under contract to a broadcaster, what evidence of verification will they want from you? Agreeing the amount of detail and the format required in advance may save both of you time and money later. Do they want a list of the models the app has been successfully tested on?

Generally speaking, in order to run a HbbTV application on a terminal, one should deploy the application in question on a Web server and tune the terminal to a broadcast signal that contains a (TV) service with AIT signaling holding the URL of the deployed application. Easy, right? A lab setup accomplishing the task looks something like this:

Anybody who has developed and deployed a Web application knows how to deploy the application on the Web server. Creating and sending the broadcast signal that contains correct AIT signaling information may prove more difficult to most developers. In order to send the signal to the HbbTV terminal’s RF interface, one needs to modulate it to a modulation scheme supported by the terminal, e.g. DVB-T2. There are plenty of lab modulators out there. Some of them require a source signal in the form of a content file, typically in MPEG2-TS format. The European Broadcasting Union provides a free, open-source tool to generate MPEG2-TS files containing TV services that embed HbbTV AIT signaling information. The tool is called hbbtv-dvbstream and is listed with other useful tools in the Resources section.

Once you’ve worked out what TV sets you want to test the app on, you need to develop a test script or journey – a reproducible sequence of steps through the app that exercise what you believe are the key features and the key steps of the user journey. User journey maps clarify complicated user journeys and ensure that the process is smooth and seamless. They are designed to help us figure out where the pain points are and improve the user experience of the provided service. They make testing reproducible and make it easier to spot regressions. If the app is developed under contract to a broadcaster they may want to review this with you or indeed, you may want them to review it with you.

RC navigation and video interaction

One thing a HbbTV application developer should always keep in mind is that users use the terminal’s remote controller for navigation. Therefore it is quintessential to keep this in mind when designing UX and perform extensive testing on terminals (instead of emulators) using their respective remote controllers. 

Typically, most applications will have some form of video interaction, either with broadcast A/V object or the player object. It is very important to test this interaction on the HbbTV terminals most common on the target market, since same actions or API calls may manifest differently on different terminals. Support for different media formats and protocols can also differ significantly on different terminal models, so it is very wise to test media playback across all available terminals. 

Some applications include interactive functionalities synchronized with the video playback. Due to reasons mentioned above, it is wise to test these functionalities extensively on the HbbTV terminals most common on the target market.

Security

A TV set typically has a much longer lifespan compared to a mobile phone or a PC. More crucially, once it is installed in a customer’s home it seldomly receives software updates. More common than not, a HbbTV terminal will run on the same software stack as delivered from the factory throughout its lifespan. 

This has many unwanted consequences, such as terminals trusting revoked certificates, not supporting new certificate authorities and root certificates etc. As a Web app developer, one can usually expect customers to run up-to-date browsers and put less effort and energy into supporting older versions. A HbbTV developer typically does not have this luxury, as the browser version may not change throughout the lifespan of the terminal.

There are also many other cybersecurity concerns developers should know about before putting HbbTV services in production. It is therefore important for HbbTV developers to get familiar with the relevant material specific for cybersecurity in the HbbTV domain, such as EBU presentation on HbbTV security and EBU recommendations on cybersecurity for connected televisions and services.

Conclusion

Despite the fact that HbbTV applications are essentially Web applications, HbbTV application testing and verification presents a unique set of challenges due to its hybrid nature and varying ecosystems. Navigating these challenges requires a combination of terminal base testing, user journey mapping and analysis, audiovisual media playback and synchronization verification, security assessments etc. By understanding and addressing these challenges, developers can ensure that their HbbTV applications deliver a consistent user experience across a wide range of devices and scenarios. Embracing all the HbbTV specifics and tailoring respective testing strategies enables HbbTV developers to create high-quality HbbTV applications.