Facebook today unveiled a highly efficient, AI text-to-speech (TTS) system that can be hosted in real time using regular processors. It’s currently powering Portal, the company’s brand of smart displays, and it’s available as a service for other apps, like VR, internally at Facebook.
In tandem with a new data collection approach, which leverages a language model for curation, Facebook says the system — which produces a second of audio in 500 milliseconds — enabled it to create a British-accented voice in six months as opposed to over a year for previous voices.
Most modern AI TTS systems require graphics cards, field-programmable gate arrays (FPGAs), or custom-designed AI chips like Google’s tensor processing units (TPUs) to run, train, or both. For instance, a recently detailed Google AI system was trained across 32 TPUs in parallel. Synthesizing a single second of humanlike audio can require outputting as many as 24,000 samples — sometimes even more. And this can be expensive; Google’s latest-generation TPUs cost between $2.40 and $8 per hour in Google Cloud Platform.
TTS systems like Facebook’s promise to deliver high-quality voices without the need for specialized hardware. In fact, Facebook says its system attained a 160 times speedup compared with a baseline, making it fit for computationally constrained devices. Here’s how it sounds:
Components
Facebook’s system has four parts, each of which focuses on a different aspect of speech: a linguistic front-end, a prosody model, an acoustic model, and a neural vocoder.
The front-end converts text into a sequence of linguistic features, such as sentence type and phonemes (units of sound that distinguish one word from another in a language, like p, b, d, and t in the English words pad, pat, bad, and bat). As for the prosody model, it draws on the linguistic features, style, speaker, and language embeddings — i.e., numerical representations that the model can interpret — to predict sentences’ speech-level rhythms and their frame-level fundamental frequencies. (“Frame” refers to a window of time, while “frequency” refers to melody.)
Style embeddings let the system create new voices including “assistant,” “soft,” “fast,” “projected,” and “formal” using only a small amount of additional data on top of an existing training set. Only 30 to 60 minutes of data is required for each style, claims Facebook — an order of magnitude less than the “hours” of recordings a similar Amazon TTS system takes to produce new styles.
Facebook’s acoustic model leverages a conditional architecture to make predictions based on spectral inputs, or specific frequency-based features. This enables it to focus on information packed into neighboring frames and train a lighter and smaller vocoder, which consists of two components. The first is a submodel that upsamples (i.e., expands) the input feature encodings from frame rate (187 predictions per second) to sample rate (24,000 predictions per second). A second submodel similar to DeepMind’s WaveRNN speech synthesis algorithm generates audio a sample at a time at a rate of 24,000 samples per second.
Performance boost
The vocoder’s autoregressive nature — that is, its requirement that samples be synthesized in sequential order — makes real-time voice synthesis a major challenge. Case in point: An early version of the TTS system took 80 seconds to generate just one second of audio.
The nature of the neural networks at the heart of the system allowed for optimization, fortunately. All models consist of neurons, which are layered, connected functions. Signals from input data travel from layer to layer and slowly “tune” the output by adjusting the strength (weights) of each connection. Neural networks don’t ingest raw pictures, videos, text, or audio, but rather embeddings in the form of multidimensional arrays like scalars (single numbers), vectors (ordered arrays of scalars), and matrices (scalars arranged into one or more columns and one or more rows). A fourth entity type that encapsulates scalars, vectors, and matrices — tensors — adds in descriptions of valid linear transformations (or relations).
With the help of a tool called PyTorch JIT, Facebook engineers migrated from a training-oriented setup in PyTorch, Facebook’s machine learning framework, to a heavily inference-optimized environment. Compiled operators and tensor-level optimizations, including operator fusion and custom operators with approximations for the activation function (mathematical equations that determine the output of a model), led to additional performance gains.
Another technique called unstructured model sparsification reduced the TTS system’s training inference complexity, achieving 96% unstructured sparsity without degrading audio quality (where 4% of the model’s variables, or parameters, are nonzero). Pairing this with optimized sparse matrix operators on the inference model led to a 5 times speed increase.
Blockwise sparsification, where nonzero parameters are restricted to blocks of 16-by-1 and stored in contiguous memory blocks, significantly reduced bandwidth utilization and cache usage. Various custom operators helped attain efficient matrix storage and compute, so that compute was proportional to the number of nonzero blocks in the matrix. And knowledge distillation, a compression technique where a small network (called the student) is taught by a larger trained neural network (called the teacher), was used to train the sparse model, with a denser model as the teacher.
Finally, Facebook engineers distributed heavy operators over multiple processor cores on the same socket, chiefly by enforcing nonzero blocks to be evenly distributed over the parameter matrix during training and segmenting and distributing matrix multiplication among several cores during inference.
Data collection
Modern commercial speech synthesis systems like Facebook’s use data sets that often contain 40,000 sentences or more. To collect sufficient training data, the company’s engineers adopted an approach that relies on a corpus of open domain speech recordings — utterances — and selects lines from large, unstructured data sets. The data sets are filtered by a language model based on their readability criteria, maximizing the phonetic and prosodic diversity present in the corpus while ensuring the language remains natural and readable.
Facebook says this led to fewer annotations and edits for audio recorded by a professional voice actor, as well as improved overall TTS quality; by automatically identifying script lines from a more diverse corpus, the method let engineers scale to new languages rapidly without relying on hand-generated data sets.
Future work
Facebook next plans to use the TTS system and data collection method to add more accents, dialogues, and languages beyond French, German, Italian, and Spanish to its portfolio. It’s also focusing on making the system even more light and efficient than it is currently so that it can run on smaller devices, and it’s exploring features to make Portal’s voice respond with different speaking styles based on context.
Last year, Facebook machine learning engineer Parthath Shah told The Telegraph the company was developing technology capable of detecting people’s emotions through voice, preliminarily by having employees and paid volunteers re-enact conversations. Facebook later disputed this report, but the seed of the idea appears to have germinated internally. In early 2019, company researchers published a paper on the topic of producing different contextual voice styles, as well as a paper that explores the idea of building expressive text-to-speech via a technique called join style analysis.
“For example, when you’re rushing out the door in the morning and need to know the time, your assistant would match your hurried pace,” Facebook proposed. “When you’re in a quiet place and you’re speaking softly, your AI assistant would reply to you in a quiet voice. And later, when it gets noisy in the kitchen, your assistant would switch to a projected voice so you can hear the call from your mom.”
It’s a step in the direction toward what Amazon accomplished with Whisper Mode, an Alexa feature that responds to whispered speech by whispering back. Amazon’s assistant also recently gained the ability to detect frustration in a customer’s voice as a result of a mistake it made, and apologetically offer an alternative action (i.e., offer to play a different song) — the fruit of emotion recognition and voice synthesis research begun as far back as 2017.
Beyond Amazon, which offers a range of speaking styles (including a “newscaster” style) in Alexa and its Amazon Polly cloud TTS service, Microsoft recently rolled out new voices in several languages within Azure Cognitive Services. Among them are emotion styles like cheerfulness, empathy, and lyrical, which can be adjusted to express different emotions to fit a given context.
“All these advancements are part of our broader efforts in making systems capable of nuanced, natural speech that fits the content and the situation,” said Facebook. “When combined with our cutting-edge research in empathy and conversational AI, this work will play an important role in building truly intelligent, human-level AI assistants for everyone.”
Original post: https://venturebeat.com/2020/05/15/facebooks-voice-synthesis-ai-generates-speech-in-500-milliseconds/
Excellent web site you’ve got here.. It’s difficult to find excellent writing like yours these
days. I honestly appreciate people like you!
Take care!!
Thankfulness to my father who shared with me on the topic of this web site, this web site is genuinely remarkable.
We are a group of volunteers and opening a new scheme in our community.
Your web site provided us with valuable information to
work on. You have done an impressive activity and our entire community
can be thankful to you.
Great post.
It’s actually very difficult in this busy life to listen news on TV, so I only use the web for that
reason, and take the newest news.
Great goods from you, man. I’ve be mindful your stuff prior to and you’re just extremely magnificent.
I actually like what you have got right here, certainly like what you are saying and the way
wherein you say it. You’re making it entertaining and you still
care for to keep it smart. I cant wait to read far more from
you. This is really a tremendous web site.
Wow! This blog looks just like my old one! It’s on a
totally different topic but it has pretty much the same layout and design.
Wonderful choice of colors!
First off I want to say excellent blog! I had a quick question which
I’d like to ask if you don’t mind. I was curious to find out how you center yourself and clear your mind before
writing. I’ve had trouble clearing my mind in getting my ideas out.
I do enjoy writing but it just seems like the first 10 to 15 minutes
are lost simply just trying to figure out how to begin. Any recommendations or tips?
Many thanks!
It’s fantastic that you are getting ideas from this paragraph as well as from our
argument made at this place.
I am the proprietor of JustCBD Store company (justcbdstore.com) and I’m presently planning to expand my wholesale side of business. I am hoping anybody at targetdomain share some guidance 🙂 I thought that the most suitable way to do this would be to connect to vape companies and cbd stores. I was really hoping if anybody could suggest a trustworthy site where I can get Vape Shop B2B Data I am currently taking a look at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Unsure which one would be the best selection and would appreciate any assistance on this. Or would it be simpler for me to scrape my own leads? Suggestions?
It is perfect time to make some plans for the long run and it is
time to be happy. I have learn this put up and if
I may I want to counsel you some interesting issues or
tips. Maybe you can write next articles regarding this article.
I desire to learn more issues approximately it!
I am the business owner of JustCBD company (justcbdstore.com) and I’m presently aiming to expand my wholesale side of company. I really hope that someone at targetdomain give me some advice . I thought that the most ideal way to do this would be to talk to vape shops and cbd stores. I was really hoping if someone could suggest a reputable site where I can get Vape Shop B2B Database I am already looking at creativebeartech.com, theeliquidboutique.co.uk and wowitloveithaveit.com. Not exactly sure which one would be the best choice and would appreciate any support on this. Or would it be simpler for me to scrape my own leads? Suggestions?
This site truly has all of the info I wanted about this subject and didn’t
know who to ask.
I used to be recommended this website by way of my cousin. I’m no longer sure whether this post is written by means of him as nobody
else realize such distinctive about my difficulty. You’re incredible!
Thanks!
Very good information. Lucky me I came across your blog by accident (stumbleupon). I have saved as a favorite for later!
I blog frequently and I genuinely thank you for your information. This great article has really peaked my interest. I will book mark your site and keep checking for new information about once per week. I opted in for your RSS feed as well.
Way cool! Some extremely valid points! I appreciate you penning this write-up and also the rest of the website is also very good.
Unquestionably believe that which you said.
Your favorite reason seemed to be on the web the easiest thing to
be aware of. I say to you, I certainly get annoyed
while people consider worries that they plainly don’t know about.
You managed to hit the nail upon the top as well as defined
out the whole thing without having side effect , people could take a signal.
Will likely be back to get more. Thanks
Good info. Lucky me I came across your website by chance (stumbleupon). I have saved as a favorite for later!
Having read this I thought it was extremely informative. I appreciate you finding the time and effort to put this information together. I once again find myself spending a significant amount of time both reading and posting comments. But so what, it was still worth it!
Spot on with this write-up, I seriously believe that this web site needs a lot more attention. I’ll probably be back again to see more, thanks for the advice!
If you wish for to improve your knowledge simply keep
visiting this web page and be updated with the most up-to-date information posted here.
I blog quite often and I seriously appreciate your information. This article has really peaked my interest. I will book mark your blog and keep checking for new information about once per week. I opted in for your Feed as well.
This website was… how do I say it? Relevant!! Finally I’ve found something which helped me. Many thanks!
Right here is the right website for everyone who hopes to understand this topic.
You know so much its almost tough to argue with
you (not that I actually would want to…HaHa). You certainly put a new spin on a topic that’s
been discussed for ages. Great stuff, just wonderful!
I’m gone to say to my little brother, that he should
also go to see this blog on regular basis to take updated from latest news update.
Saved as a favorite, I like your web site!
Good article. I absolutely love this site. Keep it up!
Spot on with this write-up, I actually think this site needs far more attention. I’ll probably be back again to read through more, thanks for the advice!
I was able to find good advice from your content.
Very good info. Lucky me I ran across your website by accident (stumbleupon). I’ve bookmarked it for later!
Great web site you have got here.. It’s difficult to find quality writing like yours nowadays. I truly appreciate people like you! Take care!!
After I originally commented I appear to have clicked on the -Notify me when new comments are added- checkbox and from now on whenever a comment is added I recieve four emails with the exact same comment. Is there a means you are able to remove me from that service? Thanks!
I absolutely love your blog.. Excellent colors & theme. Did you make this site yourself? Please reply back as I’m wanting to create my own personal site and want to learn where you got this from or what the theme is called. Thank you!
Greetings! Very helpful advice within this post! It is the little changes that make the biggest changes. Many thanks for sharing!
Aw, this was a really nice post. Taking a few minutes and actual effort to make a superb article… but what can I say… I hesitate a lot and never manage to get nearly anything done.
I quite like looking through an article that can make people think. Also, thanks for permitting me to comment!
I couldn’t refrain from commenting. Exceptionally well written!
Excellent blog you have here.. It’s hard to find excellent writing like yours these days. I really appreciate individuals like you! Take care!!
I’m more than happy to discover this web site. I want to to thank you for your time for this wonderful read!! I definitely liked every bit of it and i also have you saved to fav to look at new stuff in your site.
Saved as a favorite, I love your blog!
Howdy, I do believe your site could be having browser compatibility issues. When I take a look at your blog in Safari, it looks fine however, when opening in Internet Explorer, it has some overlapping issues. I just wanted to provide you with a quick heads up! Other than that, fantastic site!
It’s difficult to find well-informed people in this particular topic, but you sound like you know what you’re talking about! Thanks
Spot on with this write-up, I absolutely believe that this website needs much more attention. I’ll probably be returning to see more, thanks for the advice!
Hiya! Quick question that’s completely off topic. Do
you know how to make your site mobile friendly?
My web site looks weird when viewing from my iphone
4. I’m trying to find a template or plugin that might be able to fix this issue.
If you have any suggestions, please share. Thanks!
This is a topic which is close to my heart… Cheers! Exactly where are your contact details though?
I truly love your website.. Excellent colors & theme. Did you create this web site yourself? Please reply back as I’m wanting to create my own site and would like to find out where you got this from or exactly what the theme is named. Thank you!
Good post! We are linking to this great article on our website.
Keep up the good writing.
I couldn’t resist commenting. Perfectly written!
I blog frequently and I genuinely thank you for your content. The article has truly peaked my interest. I’m going to book mark your website and keep checking for new information about once per week. I subscribed to your Feed as well.
Good post. I absolutely love this site. Keep it up!
Aw, this was a really good post. Finding the time and actual effort to create a very good article… but what can I say… I put things off a whole lot and never manage to get anything done.
Howdy! I simply would like to give you a big thumbs up for your excellent info you’ve got here on this post. I will be returning to your blog for more soon.
I could not refrain from commenting. Exceptionally well written!
Having read this I believed it was very informative. I appreciate you spending some time and effort to put this informative article together. I once again find myself personally spending a lot of time both reading and posting comments. But so what, it was still worthwhile!
Everything is very open with a precise explanation of the challenges. It was definitely informative. Your website is useful. Thank you for sharing!
Ridiculous story there. What happened after? Thanks!
You are so awesome! I do not suppose I’ve read through anything like this before. So great to discover somebody with genuine thoughts on this topic. Really.. many thanks for starting this up. This web site is one thing that is needed on the web, someone with a little originality!
Good post. I learn something new and challenging on blogs I stumbleupon everyday. It will always be helpful to read articles from other writers and use something from their websites.
It’s hard to find knowledgeable people in this particular subject, however, you sound like you know what you’re talking about! Thanks
Spot on with this write-up, I actually believe that this website needs a lot more attention. I’ll probably be returning to read through more, thanks for the information!
Oh my goodness! Impressive article dude! Many thanks, However I am having troubles with your RSS. I don’t understand why I can’t subscribe to it. Is there anybody else having similar RSS problems? Anyone who knows the answer will you kindly respond? Thanks!!
Hi there! I could have sworn I’ve visited your blog before but after browsing through some of the articles I realized it’s new to me. Anyhow, I’m definitely delighted I stumbled upon it and I’ll be bookmarking it and checking back regularly!
Good day! I could have sworn I’ve been to this website before but after going through many of the posts I realized it’s new to me. Anyhow, I’m certainly happy I stumbled upon it and I’ll be book-marking it and checking back often!
Excellent blog you have here.. It’s hard to find excellent writing like yours these days. I seriously appreciate people like you! Take care!!
Greetings! Very useful advice in this particular post! It’s the little changes that produce the most significant changes. Thanks a lot for sharing!
I quite like looking through an article that can make men and women think. Also, thanks for permitting me to comment!
You’ve made some decent points there. I checked on the web for additional information about the issue and found most individuals will go along with your views on this web site.
This is a very good tip especially to those fresh to the blogosphere. Simple but very accurate info… Thanks for sharing this one. A must read article!
Having read this I believed it was rather informative. I appreciate you taking the time and energy to put this information together. I once again find myself spending way too much time both reading and commenting. But so what, it was still worthwhile!
You ought to be a part of a contest for one of the greatest blogs on the net. I will highly recommend this site!
Spot on with this write-up, I honestly believe that this web site needs a lot more attention. I’ll probably be back again to see more, thanks for the info!
Way cool! Some extremely valid points! I appreciate you writing this article and the rest of the website is really good.
I was extremely pleased to uncover this page. I wanted to thank you for your time due to this wonderful read!! I definitely loved every little bit of it and I have you saved as a favorite to see new things on your web site.
Greetings! Very helpful advice in this particular post! It is the little changes that will make the biggest changes. Many thanks for sharing!
You’ve made some really good points there. I checked on the web to learn more about the issue and found most individuals will go along with your views on this website.
Very good blog post. I certainly love this site. Stick with it!
Wow, this post is good, my sister is analyzing these kinds of things, so I am going to tell her.|
Saved as a favorite, I love your site!
This web site certainly has all the info I needed concerning this subject and didn’t know who to ask.
That is a great tip particularly to those new to the blogosphere. Brief but very accurate information… Thank you for sharing this one. A must read article!
I enjoy reading an article that will make people think. Also, many thanks for allowing me to comment!
Spot on with this write-up, I truly believe that this site needs much more attention. I’ll probably be back again to read more, thanks for the information!
Way cool! Some very valid points! I appreciate you writing this write-up and also the rest of the site is also very good.
I’m really loving the theme/design of your site.
Do you ever run into any internet browser compatibility issues?
A handful of my blog visitors have complained about my blog not operating correctly in Explorer
but looks great in Safari. Do you have any suggestions to
help fix this issue?
Achieving your fitness goal does not need a certified personal trainer or an expensive gym membership, especially when you have the budget and the space to consider practically every workout machine on the market.
Nice post. I was checking continuously this blog and I’m impressed!
Very helpful info specifically the last part 🙂 I care for such info much.
I was looking for this particular information for a long time.
Thank you and best of luck. cheap flights 2CSYEon
I’m amazed, I have to admit. Seldom do I come across a blog that’s both equally educative and entertaining, and without a doubt, you have hit the nail on the head. The problem is something which too few folks are speaking intelligently about. I am very happy I found this during my search for something regarding this.
Way cool! Some extremely valid points! I appreciate you writing this post and the rest of the website is extremely good.
This is the perfect web site for everyone who hopes to understand this topic. You know a whole lot its almost hard to argue with you (not that I personally will need to…HaHa). You certainly put a fresh spin on a subject that has been discussed for many years. Excellent stuff, just excellent!
Excellent article. I am experiencing a few of these issues as well..
I just couldn’t go away your website prior to suggesting that I extremely loved
the standard info an individual supply in your visitors? Is going to be again steadily in order to
inspect new posts
That is a very good tip especially to those fresh to the blogosphere. Brief but very precise info… Thank you for sharing this one. A must read article!
Excellent blog you have got here.. It’s hard to find high quality writing like yours nowadays. I really appreciate individuals like you! Take care!!
Very nice write-up. I definitely love this site. Continue the good work!
I’m amazed, I must say. Rarely do I come across a blog that’s both educative and interesting, and without a doubt, you have hit the nail on the head. The problem is something that too few people are speaking intelligently about. Now i’m very happy that I found this during my hunt for something concerning this.
Howdy! I simply wish to offer you a huge thumbs up for the excellent info you have right here on this post. I will be returning to your blog for more soon.
I blog quite often and I truly appreciate your content. Your article has truly peaked my interest. I’m going to book mark your site and keep checking for new details about once per week. I opted in for your Feed too.
Pretty! This has been an extremely wonderful post. Thank you for providing this info.
Pretty! This was an incredibly wonderful article. Many thanks for providing this information.
This is a topic that’s near to my heart… Many thanks! Where are your contact details though?
Spot on with this write-up, I truly feel this site needs a lot more attention. I’ll probably be returning to see more, thanks for the advice!
Hello! I just would like to offer you a huge thumbs up for the excellent information you have got right here on this post. I’ll be returning to your site for more soon.
Excellent post. I definitely love this website. Keep writing!
Spot on with this write-up, I absolutely feel this web site needs a lot more attention. I’ll probably be back again to read more, thanks for the info!
There is definately a lot to know about this subject. I love all of the points you have made.
Greetings! Very useful advice within this article! It’s the little changes that make the greatest changes. Thanks for sharing!
An outstanding share! I have just forwarded this onto a friend who had been doing a little homework on this. And he in fact bought me breakfast due to the fact that I found it for him… lol. So allow me to reword this…. Thank YOU for the meal!! But yeah, thanks for spending time to discuss this matter here on your internet site.
Hi, I do believe this is an excellent website. I stumbledupon it 😉 I am going to return once again since I saved as a favorite it. Money and freedom is the greatest way to change, may you be rich and continue to help other people.
Hi there! This article could not be written any better! Looking through this article reminds me of my previous roommate! He constantly kept preaching about this. I’ll forward this information to him. Pretty sure he’s going to have a good read. I appreciate you for sharing!
bookmarked!!, I love your website!
That is a really good tip particularly to those fresh to the blogosphere. Short but very accurate information… Many thanks for sharing this one. A must read post!
Spot on with this write-up, I truly think this website needs a lot more attention. I’ll probably be returning to see more, thanks for the advice!
Hey there! I simply would like to offer you a huge thumbs up for your excellent information you have got here on this post. I will be coming back to your web site for more soon.