<p>Hello, this is Patrick. I am an engineering manager at Leapmind.</p>
<p>Recently, in our hackdays in January 2022, I thought up some fun ideas to experiment. This was done over 2-3 days. (About our past hackdays activity, please see <a href="https://leapmind.io/blog/2019/04/15/hackdays-2019-2q/">this article</a>)</p>
<h1 id="the-idea">The Idea</h1>
<p>Most people are afraid of speaking foreign languages (eg. English). From what I understand, they lack the confidence and are afraid of making grammatical mistakes, or thinking that they have a funny accent.</p>
<p>But what if we can "simulate" what we would have sounded like, if we have perfect English capability? One solution is using voice cloning.</p>
<h1 id="the-method">The Method</h1>
<p>Recently, an impressive advance in voice cloning domain has been made -- it is called <a href="https://coqui.ai/blog/tts/yourtts-zero-shot-text-synthesis-low-resource-languages">YourTTS</a>. This model enables us to easily embed a new speaker's voice without training and with minimal data required. An example colab code is <a href="https://colab.research.google.com/drive/1ftI0x16iqKgiQFgTjTDgRpOM1wC1U-yS?usp=sharing">here</a>.</p>
<p><strong>Figure: TTS model architecture (source from coqui.ai)</strong>
<img src="https://coqui.ai/static/aeec1cf73c251540bf76af923d1737af/39a40/blog-tts-yourtts-zero-shot-text-synthesis-low-resource-languages-inference.png" alt="TTS model architecture"></p>
<h1 id="the-data">The Data</h1>
<p>One of the challenges with most machine learning projects is how to find relevant, high quality data. In this case, I asked a few leapmind employees' (including our CEO!) permission to use their voice sample as input data. Fortunately, we have a few video recordings of our internal presentations. These are mostly 05-30 min uninterrupted speeches of each presenter, which I can extract with the following steps below.</p>
<p>Additionally, for the text-to-speech part to work, a short speech (say -- a few sentences) should be prepared. Note that for custom names (company products, human names), the normal wording might fail so we have to "craft" a natural-sounding text with trial and error.</p>
<ol>
<li>find source reference data from old company presentation videos</li>
<li>download as mp3/wav, then put into <a href="https://www.audacityteam.org/">Audacity</a> (audio editor)</li>
<li>clean out other noises (other people’s voice, coughing, etc.)</li>
<li>export as wav file</li>
<li>normalize and sample with <a href="https://ffmpeg.org/">ffmpeg</a></li>
<li>prepare “English text” file for AI to read</li>
<li>run the prepared input data through YourTTS model to generate output .wav file.</li>
</ol>
<h1 id="the-result">The Result</h1>
<p>I found that at the minimum, a 5-min sample speech can enable the TTS model to somewhat mimic the speaker's voice. However, to product high-quality output, a more varied and high-quality sample speech data is needed (more varied topics, vocabularies, and quiet background noises). Additionally, certain speakers can have a more pronouced speaking personality and tone that is more easily emulated.</p>
<p>Here is one example of the output... the speaker is our CEO!</p>
<ul>
<li><a href="/images/posts/techblog/20220301-voice-cloning-hackdays-2022q1/matsuda-san_input_excerpt.mp3">input speaker data (excerpt) mp3</a></li>
<li><a href="/images/posts/techblog/20220301-voice-cloning-hackdays-2022q1/matsuda-san_output.mp3">output TTS result mp3</a></li>
</ul>
<p>English text used:</p>
<pre><code>Since its inception in 2012, LeapMind has engaged in 
research and development of software frameworks, algorithms, and hardware, 
with a focus on embedded technologies, to solve business problems 
with leading-edge machine learning technologies.
</code></pre>
<h1 id="conclusion">Conclusion</h1>
<p>Since this is "a little fun mini-project" that took little time, I didn't expect the result to be good. However, the quality does surprise me since I only prepared minimum amount of data (less than 25 min speech) where usually a much larger set is required for speech projects.</p>
<p>Finally, I want to thank my colleagues who allow me to use their voice data for this experiment and the open-source community for allowing people like me to try out and tinker with the latest technologies.</p>

Tech Blog