By Enne Kim, NoCamels -
An emerging field of generative artificial intelligence (AI) technology has elevated voice dubbing to a new level—one where English-speaking acting icons can indeed appear to speak their famous lines in fluent Spanish or Greek or even Russian.
Voice dubbing is when a new voice replaces the original voice for a role, usually to translate the content into another language and an Israeli startup is leading the way, thanks to its artificial intelligence platform.
Traditional voice dubbing studios can take up to months to produce dubbed versions of content, with a complex duplication process that includes finding the right voice actor and recording the script reading. Studios spend up to $150K on dubbing a single film.
However, Deepdub’s AI software can dramatically reduce the time and effort needed to dub. By using deep learning algorithms, their AI dubbing software can be trained to quickly learn the patterns of a particular voice sample and convey the voice actor’s specific tone and emotion.
And if the delivery isn’t quite right, the AI can understand prompts like “I want this sentence to be a little more giggly.” The software then generates a real-time dub of the content, saving thousands of dollars and working hours.
Based in Tel Aviv, the company was co-founded in 2019 by brothers Ofir and Nir Krakowski. With backgrounds in machine learning and artificial intelligence, the two wanted to address the emerging needs of streaming television, an abundance of content that is not just about entertainment, and the lack of infrastructure for voice dubbing.
“The realization was that there is a market here where you can create a very big impact on people around the world,” says Oz Krakowksi, chief revenue officer at Deepdub and sibling of the co-founders.
“Large populations don’t have access to audible content right now, like television and film. And what about e-learning courses that are only in English? If they don’t speak English, they’re limited in the type of access to education that we have because we speak English. And that’s when we talk about democratizing content.”
By enabling voice dubbing in different languages, people who struggle with reading and writing or have vision problems can have access to new types of audio content. Deepdub’s software is capable of translating content into 65 languages—including regional dialects.
Dialects are critical for properly localizing content, or adapting the speech to fit cultural nuances and subtleties. Spanish, the second most spoken native language in the world, has multiple dialects in the different regions of Latin America. There is a universal version of Spanish called “Neutral Spanish” that only exists for voice dubbing, which eliminates certain pronunciations or specific words.
However, Deepdub wants to push the boundaries of “neutral” dubbing languages and actually enable the dialects that make each region unique.
“We help those countries to create and maintain their own self identity in terms of, this is our specific version of Spanish that we’d like to maintain and we’d like to consume content in our dialect of Spanish,” Oz Krakowksi tells NoCamels. Moreover, countries like India (which has more than 1,900 languages and dialects) could have content tailored to their specific region.
Krakowksi says that is what sets Deepdub apart–the ability to expand into a range of languages that wouldn’t be possible without their deep learning infrastructure.
Prominent AI voice dubbing companies like Lovo.ai and Synthesys also offer a range of emotions and languages for people who want their content dubbed. However, the Deepdub algorithm is able to reproduce and localize the same speaker in a variety of languages, which Krakowksi says is what allowed them to break into the Hollywood industry before other companies.
Ethics And Actors
Amid a furor over AI in the entertainment industry, Deepdub addresses the ethical challenges of its use in voice dubbing. Voice actors can find themselves uncredited and unpaid, like Canadian voice actor Bev Standing, who is currently suing the TikTok social media platform for allegedly using a synthetic version of her voice “without compensation or credit.”
However, Deepdub has Trusted Partner Network (TPN) accreditation, which means the company has passed strict security and legal measures in order to work with the Motion Picture Association, the organization that represents major film studios like Disney, Netflix, and Universal.
“We work in an ethical way that is controlled,” Krakowksi says. Deepdub provides the platform instead of the technology so that the voice samples remain secured.
“That’s part of the reason why studios work with us. We don’t give people a way to start using just anyone’s voice and create those malicious defects that you can see on the web.”
However, the growth of AI voice dubbing also means the voice acting industry will have to adapt. In 2018, there were over 500,000 voice actors employed in the United States.
What does that mean for voice actors when generative AI can create a dubbed film in minutes?
Oz Krakowksi believes that Deepdub’s new AI technology will create different opportunities for voice actors and content. “If we were too afraid to adopt a new technology because it might take away jobs, we would still be driving horse drawn carriages,” he says.
Furthermore, Deepdub still works with the voice actors to ensure that the generated dubs are “Hollywood quality.”
In fact, about half of Deepdub’s staff is made up of people who come from the voice dubbing industry. The startup’s director of creative and quality control is Maya Mitelpunkt-Aldor, who actually voiced Mirabel in the Hebrew version of the 2021 Disney animation Encanto. The staff, Krakowksi says, help represent the ethical concerns of voice actors in the “creative tech” environment that Deepdub has cultivated.
In December 2021, Deepdub announced a deal with the U.S.-based streaming service Topic, which features content in more than 20 languages from 40 countries. In January 2022, the startup closed their Series A funding round with over $20 million in funding, led by New York-based VC firm Insight Partners.
Deepdub is continuing to expand their platform. The startup recently launched Deepdub GO, which allows anyone to register online and access their voice dubbing services. From YouTubers to independent filmmakers, Oz says the new platform enables content creators to take their videos and affordably translate them into a variety of dubbed voices.
“Traditionally, dubbing a film would take 6-10 weeks. Depending on the type of film, we could be 30-50% faster, and in some cases we could also complete the entire work within 2-4 days and into multiple languages simultaneously,” he says. Additionally, it can take as little as a few weeks to integrate a new language into the platform.
And while Krakowksi is reticent about revealing pricing, he does say that Deepdub GO is more affordable for those creators who do not have the deep pockets of movie studios.
The company also allows an English-speaking audience to access non-English language content.
One of Deepdub’s biggest projects has been creating an English dubbed version of Vanda, a TV heist drama series that was originally in Portuguese.
Earlier this year, the company partnered with the U.S. streaming giant Hulu to release the first season in English.
“You can tell that there is a lot of attention to the voices, the lip syncing, like how the background sounds with the voices. You can hear the flames, you can hear the chatter on the TV and the kids and everything as if it was originally in English.”
