VVC Teams Work on End-to-End Film-Grain Management for Today's Diverse Digital Video Environment — MC-IF

Technology teams associated with the Versatile Video Coding (VVC) community are working on new ways to optimize the efficacy of film grain.
Film grain has been a concern since the transition from celluloid to digital video due to its lack of consistency and compression challenges using standard algorithmic methodologies.
Dolby and InterDigital are active participants in the Media Coding Industry Forum (MC-IF), exploring ways VVC can improve the ability to enhance and preserve artistic intent while delivering the most compelling visual experiences to broadcast and streaming audiences.

Technology teams associated with the Versatile Video Coding (VVC) community are working on new ways to optimize the efficacy of film grain to accomplish creative objectives and overcome technical limitations in today's diverse digital video environment, according to Sean McCarthy, director of video strategy & standards at Dolby Laboratories and Philippe de Lagrange, senior engineer at InterDigital in a recent interview for journalists.

Dolby and InterDigital are active participants in the Media Coding Industry Forum (MC-IF), exploring ways VVC can improve the entertainment technology sector's ability to enhance and preserve artistic intent while delivering the most compelling visual experiences to broadcast and streaming audiences.

Among the first creative concerns expressed in the early transition from celluloid to digital video was the potential loss of film grain, the random physical textures made by small metallic silver particles common in processed analog photography. While the earliest film pioneers considered film grain a technical flaw that detracted from the clarity and sharpness of the image, it did not take long for some of the greatest storytellers in history to turn this visual "noise" into a creative asset.

Filmmakers like Stanley Kubrick and Martin Scorsese often embraced the grainy look of film to create mood and atmosphere in their productions. Translating this capability into the digital realm has been challenging.

"Film grain is difficult to compress using standard algorithmic methodologies," explains Lagrange. "Video compression relies on temporal and spatial consistency to predict and compress pixels. However, film grain is a random and high-entropy signal that lacks spatial and temporal consistency, making it difficult to compress."

To this end, Lagrange has been working on a film grain modeling technique that addresses this challenge by compressing video without the grain and then using parameters of a statistical model to synthesize replacement grain during decoding.

"The goal is to preserve the appearance of grain in the compressed video," he says.

Film Grain Management Addresses Creative and Technical Issues

According to McCarthy, there is more to film grain than achieving excellent creative outcomes.

"Film grain is used in the digital imaging world for two main reasons. Firstly, it provides a perceived sharpness to the image, enhancing the underlying visual experience," he says.

This subjective sharpness cannot be measured but is beneficial for creating a more natural and less synthetic image.

"Secondly, film grain can hide underlying artifacts -- such as compression or image processing flaws -- especially in low-bitrate situations. Adding film grain to the imagery can mask these artifacts and improve the overall subjective quality of the image by about 20 to 25%," says McCarthy.

McCarthy explains that film grain analysis, distribution, and synthesis are all part of managing film grain in digital imaging environments. The analysis involves parameterizing the film grain, while the distribution includes sending metadata -- specifically supplemental enhancement information (SEI) messages -- containing film grain information.

"The synthesis is the step where the film grain is recreated and added to the imagery during the rendering process. This end-to-end digital management of film grain has improved a lot recently thanks to the work of many experts in content creation, distribution, and consumption. It enables film grain characterizations -- or models -- to be included as metadata, maintaining creative intent and hiding blemishes caused by bitrate inconsistencies," says McCarthy.

According to Lagrange, VVC offers a path to better modeling and more cost-efficient deployment of film grain management. Moreover, the SEI message that conveys film grain characteristics for VVC provides the same quality and efficiency boosts when used with other MPEG video compression technologies such as AVC and HEVC.

During the DVB World Conference in March, Lagrange and his team showcased two film grain-related technologies.

The first demo exhibited a parameter tuning tool that allows users to adjust the strengths and sharpness of the grain in real time on a test picture.
The second demo featured real-time video decoding with grain synthesis.

"We are working on accelerating a hardware implementation of these tools to provide the community with a cost-efficient and technically effective end-to-end way to manage film grain for high-value content that is streamed or broadcast to devices -- including content that is protected with digital rights management (DRM) technology," concludes Lagrange.

To learn more about MC-IF, please visit: https://www.mc-if.org/