As digital animators continue to push the boundaries of technology and creativity, the technical teams that support them are turning to artificial intelligence and machine learning to deliver the tools they need. That’s the case at Pixar, where the company has made new machine learning breakthroughs it hopes will both improve quality and reduce costs.
Vaibhav Vavilala, technical director at Pixar Animation Studios, has been leading some of those efforts after carefully studying recent scientific literature and monitoring the work at the R&D labs of parent company Disney. Vavilala said those advances have the potential to reduce Pixar’s datacenter footprint by half for some stages of production.
Vavilala made his remarks during a presentation at VentureBeat’s Transform 2020 conference.
In recent years, Vavilala has worked on the lighting and rendering optimization team on such Pixar movies as Coco, Incredibles 2, and Toy Story 4. But a couple of years ago, he began focusing on a concept called Deep Learned Super Resolution. Vavilala and a collaborator at Pixar, Mark Meyer, published a technical paper detailing their work.
During his presentation, Vavilala explained that a major challenge the digital animation industry faces is the time and cost in rendering animation at high resolutions such as 2K or 4K.
“It typically takes at least 50 CPU hours to render one frame at 2K resolution,” he said. “We render 24 frames per second for a 90-minute film and render each shot many times as the artists iterate. Further, we have multiple films in flight at a given time. All this adds up to a lot of demand for the render farm, which is a finite resource. If we render at a higher resolution like 4K, that’s four times the pixels and more than double the cost.”
In effect, with the evolution of current systems, solving this issue is complicated by the fact that there’s a bit of a cat and mouse game going on between animators and the technical teams.
“As rendering becomes cheaper with better technologies, artists often eat up those gains by trying to push creative boundaries such as more lights, more geometry, more fog, and more complex lighting,” he said. “All of this often results in longer render times. And of course, there’s perpetually demand for more content. So, there’s still room for innovation to make rendering cheaper.”
That led Vavilala to Deep Learned Super Resolution. In effect, this technique would let animators create their work in lower resolutions, and then a machine learning system would “upscale” them into the higher resolutions.
To develop this idea, Vavilala set up a PyTorch development environment and then prepared a Linux instance with Nvidia GPUs. His team then began training the system using high and low-resolution images from recent Pixar movies.
Teaching the system to correctly fill in details such as light and color proved to be tricky. So the team turned to generative adversarial networks (GANs) to help improve the outcome.
GANs are a kind of machine learning where two neural networks play off each other to generate the desired outcome. Typically this includes a generator for the samples and then a discriminator that identifies the differences with the goal of eventually reconciling the two images.
In the case of Pixar, shifting high-resolution rendering off of its render farms and onto its separate upscaling system produced interesting efficiency gains.
Vavilala said the latest models demonstrated that when images were rendered at 1K and then shifted to the second system to be upscaled to 2K, the studio could save between 50% and 75% of the render farm processing footprint. He’s hopeful this technology will be ready to use in upcoming productions.
“It took a lot of experimentation and iteration to get anything close to what would be considered production-ready,” he said.
Finally, Vavilala said Pixar’s experiments in this field offered some lessons for companies in other industries.
“Promote a culture of risk-taking in your organization,” he said. “One of my observations at Pixar is that folks are open to new technologies and experimentation early during the development of the film. But once we’re deep into production, we instead focus on stability and try to avoid the technology breaking. If you can find a balance between risk-taking and tried-and-true processes, you can successfully innovate and mostly avoid chaos.”