We are pleased to introduce content adaptive streaming to the Jaunt XR Platform. Content adaptive streaming allows for video delivery to be optimized for each individual piece of content and leads to a higher quality user experience across all platforms. With content adaptive encoding becoming a new standard for traditional video streaming on entertainment platforms like Netflix, we’re bringing this same premium quality to the delivery of immersive experiences.
What is Content Adaptive Streaming?
With traditional adaptive bitrate streaming, each title is transcoded to create multiple representations at different resolutions, frame rates, layouts (mono or stereo), and bitrates. These representations are then placed into a quality hierarchy or ranking, also known as a bitrate ladder. During playback, the client downloads the best representation based on the available bandwidth and device capabilities of each individual user.
The introduction of content adaptive encoding means that each title on our platform is first analyzed for its complexity, and then encoded into multiple representations. The result is the delivery of the highest quality experience possible based on the content itself and the viewer’s unique playback conditions.
Fixed Bitrate Ladder Streaming
Titles published through the Jaunt’s XR platform have been delivered to users using adaptive bitrate streaming technology since 2016. Content adaptive streaming represents a shift from this “fixed bitrate ladder” streaming model that facilitates delivery optimization per piece of content vs. a “one size fits all” approach
In the fixed bitrate model, we generate a predetermined set of representations for all pieces of content. Each output rung of this ladder is encoded with a unique combination of resolution, frame-rate and bitrate for each encoder and device profile. As an example, we created a fixed ladder specifically tuned for virtual reality content, which, based on the bandwidth and capability of the device, delivered between 720p 30fps to 4K 60fps, mono or stereo VR video at bitrates ranging from ~1 Mbps to upwards of 60 Mbps. This fixed bitrate ladder offered high resolution, high frame-rate stereo delivery at much higher bitrates than traditional 2D videos. This ensured that users streaming over broadband had a premium quality VR experience, for a variety of content ranging from slow-moving or animation content to fast-moving action scenes.
In reality, due to limited broadband availability, majority of our mobile users stream content at bandwidth under 8 Mbps. It can be very challenging to deliver a highly immersive experience to such users using a fixed ladder. To that end, we’ve deployed a new content adaptive bitrate encoding (CABE) technology to improve upon the fixed ladder for adaptive bitrate streaming.
Evolution to Content Adaptive Bitrate Encoding (CABE)
One of the primary issues with using a “one size fits all” bitrate ladder is that it has different implications for each piece of content. For visually simplistic content such as the animated piece “Asteroids,” the encoder can compress 4K content under 8 Mbps without significant loss in visual quality. However for complex content with rich detail and camera motion such as “Under the Canopy,” 4K content would need to be encoded at over 50 Mbps to provide visually acceptable quality. If we use a fixed bitrate ladder, all pieces of content are treated equally. This can at times result in sub-optimal encoding for simpler content, and compromise the viewing experience at low bandwidth.
With the new CABE algorithm, we tune the bitrate ladder to each title. This allows us to deliver “simple” content at lower bitrates without sacrificing image quality. We are also able to continue to delivering more “complex” content at optimal resolutions and bitrates based on playback conditions.
Intelligent Content Analysis
In order to create a bitrate ladder catered specifically for each title, we run an analysis pass on the title to learn about the nature of the content. We try to learn if the content is difficult, i.e. fast moving with lot of details or noise, or easy, i.e,. slow moving without too many fine textures in the scene. A typical output of the first pass stage is a set of rate-distortion (R-D) curves as shown below.
The figure above shows R-D curves for a set of predetermined configurations (resolution, frame-rate and layouts), that would allow for a “Jaunt-approved” user experience for streaming VR content. Based on the first pass analysis we derive a bitrate ladder for each title which has the optimal number of outputs based on some Just Noticeable Difference (JND) criterion. Currently, we are using a combination of well known objective quality metrics such as PSNR and SSIM to determine the final adaptive ladder, but also exploring weighted metrics more suitable for immersive content.
Implications for Jaunt XR Platform
For streaming 360 video content, only a small portion of the video frame corresponding to the “field of view” of the HMD gets scaled, and displayed on the screen. If the video is delivered at low resolution, the scaled image appears blurry, and the user experience is severely compromised. With content adaptive streaming, ladder levels can be optimized for each piece of content to ensure that the highest possible resolution is streamed at a given bandwidth. The visual impact of higher resolution is clearly visible from the side-by-side comparisons of scenes from the Invasion and Invisible, content below.
Besides streaming, we also generate high-quality transcodes for offline viewing using the Jaunt XR platform. A scene from the Collisions title shows how the bitrate can be nearly halved without any loss of visual quality using content adaptive encoding. This enables smooth playback of high quality content even over resource constrained mobile platforms.
CABE technology powering the Jaunt XR Platform means that users are able to optimize high quality content delivery per device and also reduce CDN costs. Ideally, when a piece of content moves up one rung of its unique bitrate ladder, the visual quality should improve as well. A fixed ladder designed to achieve this behavior can at times lead to inefficiencies for simpler content. For simpler content moving from one ladder rung of the fixed ladder to the next, may not always lead to improvement in visual quality. With the new CABE technology, we can design each rung of the ladder such that it provides a perceptually significant improvement for each individual title. Depending on the content, this can eliminate anywhere between two to four redundant ladder rungs from the bitrate ladder, thereby significantly reducing CDN costs.
With CABE technology now live on the Jaunt XR Platform, we’re excited to be introducing yet another tool for advancing the delivery of high quality immersive content. For more information on our platform, visit our Jaunt XR Platform page and check out more immersive content news on the Jaunt Blog.