This AI can perfectly dub videos in Indic languages — and correct lip syncing

People in India watch a lot of videos on the internet. According to a report from The Wall Street Journal, Indians spend more than 8.5GB of mobile data on average, and most of it on video. Last year, YouTube said more than 95% of content consumption is in regional languages. So naturally, there’s a lot of appetite for vernacular videos, but not all creators know all Indic languages.

One solution is dubbing. Last week, just after Parasite won the Oscar award, Mother Jones claimed dubbing is superior than translated subtitles. But let me tell you it sucks. I’ve seen plenty of English language movies dubbed in Hindi, and I either can’t stand them or I die laughing. Lip-syncing is often off, and dubbing seems quite unnatural.

Now, researchers from the International Institute of Information Technology from the southern city of Hyderabad, India have developed a new AI model that translates and lip-syncs a video from one language to another with great accuracy.

The team said we already have the technology to get translated textual or speech output from a video. However, visual translation — such as the movement of lips — is lost during dubbing. To solve that, Indian researchers have developed a new Generative Adversarial Network (GAN) called LipGAN. While it can match lip movements of translated text in an original video, it can also correct lip movements in a dubbed movie.

To translate videos, the model transcribes speech in the video using speech recognition. Then it uses a specialized model trained for Indic languages to translate the text, for example from English to Hindi. Then the speech recognition model converts it into voice. The speech-to-speech translation coupled with LipGAN forms the whole model. Researchers noted that their translation model was more accurate than Google Translate.

Prof. C.V. Jawahar, Dean, Research and Development, IIIT-H, said this technology will help in creating more content in regional languages:

Manually creating vernacular content from scratch, or even manually translating and dubbing existing videos will not scale at the rate of digital content is being created. That’s why we want it to be completely automatic.

There are superbly created videos on various topics by MIT and other prestigious institutions that are inaccessible to a larger Indian audience simply because they cannot comprehend the accent(s). Forget the rural folks, even I won’t understand!

The team added the model still struggles with moving or multiple faces in videos. Apart from addressing these problems, the team wants to work on improving expressions on face after translation.

We’ve seen plenty of efforts of GANs recreating face or body movement using just a single photo. But most of that research is in a single language; most likely to be English. It has taken a couple of iterations for these AI models to become quite convincing to the human eye. So, while the above model has its drawbacks, it’s one of the few GANs working on a multilingual model. It can definitely improve over time with more data.

Last year, deepfakes were in news for its negative use. But, video modification AI for educational or entertainment purposes is quite an important use case of that technology. This model is a perfect example of that.

You can read the full paper here.

You’re here because you want to learn more about artificial intelligence. So do we. So this summer, we’re bringing Neural to TNW Conference 2020, where we will host a vibrant program dedicated exclusively to AI. With keynotes by experts from companies like Spotify, RSA, and Medium, our Neural track will take a deep dive into new innovations, ethical problems, and how AI can transform businesses. Get your early bird ticket and check out the full Neural track.

Original post:

10 comentários em “This AI can perfectly dub videos in Indic languages — and correct lip syncing

  1. Having read this I thought it was extremely enlightening.
    I appreciate you finding the time and energy to put this short
    article together. I once again find myself spending a significant amount
    of time both reading and commenting. But so what, it was still

  2. Just want to say your article is as astounding. The clearness for your post is just excellent
    and that i can assume you’re a professional in this subject.

    Fine along with your permission let me to snatch your feed to stay updated with drawing close post.
    Thanks a million and please keep up the rewarding work.

  3. I’m the business owner of JustCBD brand ( and I am currently planning to grow my wholesale side of business. I really hope that someone at targetdomain is able to provide some guidance . I thought that the best way to accomplish this would be to talk to vape stores and cbd stores. I was really hoping if someone could recommend a trusted website where I can buy Vape Shop B2B Marketing List I am already examining, and Unsure which one would be the very best option and would appreciate any assistance on this. Or would it be much simpler for me to scrape my own leads? Ideas?

  4. I’m the owner of JustCBD company ( and am looking to develop my wholesale side of business. I really hope that someone at targetdomain can help me 🙂 I considered that the most ideal way to accomplish this would be to connect to vape companies and cbd stores. I was really hoping if someone could recommend a trustworthy web site where I can get Vape Shop Business Marketing Data I am presently checking out, and Not sure which one would be the most suitable solution and would appreciate any support on this. Or would it be much simpler for me to scrape my own leads? Suggestions?

  5. Do you mind if I quote a couple of your articles as long as
    I provide credit and sources back to your webpage? My website is in the very same niche as yours and my visitors would
    definitely benefit from some of the information you provide here.
    Please let me know if this alright with you. Thank you!

  6. Pretty nice post. I just stumbled upon your blog and wanted to say that I’ve truly enjoyed
    browsing your blog posts. In any case I’ll be subscribing to your feed and I hope you write again very soon!

  7. I am not sure where you are geting your info, but great topic.
    I needs to spend some time learning much more or understanding more.
    Thanks for great information I was looking for
    this information for my mission.

Leave a Reply

Your email address will not be published. Required fields are marked *