资源算法Real-Time-Voice-Cloning

Real-Time-Voice-Cloning

2019-11-26 | |  69 |   0 |   0

Real-Time Voice Cloning

This repository is an implementation of Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (SV2TTS) with a vocoder that works in real-time. Feel free to check my thesis if you're curious or if you're looking for info I haven't documented yet (don't hesitate to make an issue for that too). Mostly I would recommend giving a quick look to the figures beyond the introduction.

SV2TTS is a three-stage deep learning framework that allows to create a numerical representation of a voice from a few seconds of audio, and to use it to condition a text-to-speech model trained to generalize to new voices.

Video demonstration (click the picture):

68747470733a2f2f692e696d6775722e636f6d2f386c46556c677a2e706e67.png

Papers implemented

URLDesignationTitleImplementation source
1806.04558SV2TTSTransfer Learning from Speaker Verification to Multispeaker Text-To-Speech SynthesisThis repo
1802.08435WaveRNN (vocoder)Efficient Neural Audio Synthesisfatchord/WaveRNN
1712.05884Tacotron 2 (synthesizer)Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram PredictionsRayhane-mamah/Tacotron-2
1710.10467GE2E (encoder)Generalized End-To-End Loss for Speaker VerificationThis repo

News

13/11/19: I'm sorry that I can't maintain this repo as much as I wish I could. I'm working full time on improving voice cloning techniques and I don't have the time to share my improvements here. Plus this repo relies on a lot of old tensorflow code and it's hard to work with. If you're a researcher, then this repo might be of use to you. If you just want to clone your voice, do check our demo on Resemble.AI - it can run for free but it will be a bit slower, and it will give much better results than this repo.

20/08/19: I'm working on resemblyzer, an independent package for the voice encoder. You can use your trained encoder models from this repo with it.

06/07/19: Need to run within a docker container on a remote server? See here.

25/06/19: Experimental support for low-memory GPUs (~2gb) added for the synthesizer. Pass --low_mem to demo_cli.py or demo_toolbox.py to enable it. It adds a big overhead, so it's not recommended if you have enough VRAM.

Quick start

Requirements

You will need the following whether you plan to use the toolbox only or to retrain the models.

Python 3.7. Python 3.6 might work too, but I wouldn't go lower because I make extensive use of pathlib.

Run pip install -r requirements.txt to install the necessary packages. Additionally you will need PyTorch (>=1.0.1).

A GPU is mandatory, but you don't necessarily need a high tier GPU if you only want to use the toolbox.

Pretrained models

Download the latest here.

Preliminary

Before you download any dataset, you can begin by testing your configuration with:

python demo_cli.py

If all tests pass, you're good to go.

Datasets

For playing with the toolbox alone, I only recommend downloading LibriSpeech/train-clean-100. Extract the contents as <datasets_root>/LibriSpeech/train-clean-100 where <datasets_root> is a directory of your choosing. Other datasets are supported in the toolbox, see here. You're free not to download any dataset, but then you will need your own data as audio files or you will have to record it with the toolbox.

Toolbox

You can then try the toolbox:

python demo_toolbox.py -d <datasets_root>
or
python demo_toolbox.py

depending on whether you downloaded any datasets. If you are running an X-server or if you have the error Aborted (core dumped), see this issue.

Wiki

Contributions & Issues

I'm working full-time as of June 2019. I don't have time to maintain this repo nor reply to issues. Sorry.


上一篇:CHN

下一篇:yellowbrick

用户评价
全部评价

热门资源

  • Keras-ResNeXt

    Keras ResNeXt Implementation of ResNeXt models...

  • seetafaceJNI

    项目介绍 基于中科院seetaface2进行封装的JAVA...

  • spark-corenlp

    This package wraps Stanford CoreNLP annotators ...

  • capsnet-with-caps...

    CapsNet with capsule-wise convolution Project ...

  • inferno-boilerplate

    This is a very basic boilerplate example for pe...