update training tips and faiss tips (#208)
This commit is contained in:
nadare
GitHub
parent
6d0ec4b00c
commit
69ea94609b
|
|
@ -12,10 +12,7 @@ In '/logs/your-experiment/3_feature256' where the model is located, features ext
|
|||
From here we read the npy files in order sorted by filename and concatenate the vectors to create big_npy. (This vector has shape [N, 256].)
|
||||
After saving big_npy as /logs/your-experiment/total_fea.npy, train it with faiss.
|
||||
|
||||
As of 2023/04/18, IVF based on L2 distance is used using the index factory function of faiss.
|
||||
The number of IVF divisions (n_ivf) is N//39, and n_probe uses int(np.power(n_ivf, 0.3)). (Look around train_index in infer-web.py.)
|
||||
|
||||
In this article, I will first explain the meaning of these parameters, and then write advice for developers to create a better index.
|
||||
In this article, I will explain the meaning of these parameters.
|
||||
|
||||
# Explanation of the method
|
||||
## index factory
|
||||
|
|
@ -103,44 +100,3 @@ https://github.com/facebookresearch/faiss/wiki/Fast-accumulation-of-PQ-and-AQ-co
|
|||
## RFlat
|
||||
RFlat is an instruction to recalculate the rough distance calculated by FastScan with the exact distance specified by the third argument of index factory.
|
||||
When getting k neighbors, k*k_factor points are recalculated.
|
||||
|
||||
# Techniques for embedding
|
||||
## alpha query expansion
|
||||
Query expansion is a technique used in searches, for example in full-text searches, where a few words are added to the entered search sentence to improve search accuracy. Several methods have also been proposed for vector search, among which α-query expansion is known as a highly effective method that does not require additional learning. In the paper, it is introduced in [Attention-Based Query Expansion Learning](https://arxiv.org/abs/2007.08019), etc., and [2nd place solution of kaggle shopee competition](https://www.kaggle.com/code/lyakaap/2nd-place-solution/notebook).
|
||||
|
||||
α-query expansion can be done by summing a vector with neighboring vectors with weights raised to the power of similarity. How to paste the code example. Replace big_npy with α query expansion.
|
||||
|
||||
```python
|
||||
alpha = 3.
|
||||
index = faiss.index_factory(256, "IVF512,PQ128x4fs,RFlat")
|
||||
original_norm = np.maximum(np.linalg.norm(big_npy, ord=2, axis=1, keepdims=True), 1e-9)
|
||||
big_npy /= original_norm
|
||||
index.train(big_npy)
|
||||
index.add(big_npy)
|
||||
dist, neighbor = index.search(big_npy, num_expand)
|
||||
|
||||
expand_arrays = []
|
||||
ixs = np.arange(big_npy.shape[0])
|
||||
for i in range(-(-big_npy.shape[0]//batch_size)):
|
||||
ix = ixs[i*batch_size:(i+1)*batch_size]
|
||||
weight = np.power(np.einsum("nd,nmd->nm", big_npy[ix], big_npy[neighbor[ix]]), alpha)
|
||||
expand_arrays.append(np.sum(big_npy[neighbor[ix]] * np.expand_dims(weight, axis=2),axis=1))
|
||||
big_npy = np.concatenate(expand_arrays, axis=0)
|
||||
|
||||
# normalize index version
|
||||
big_npy = big_npy / np.maximum(np.linalg.norm(big_npy, ord=2, axis=1, keepdims=True), 1e-9)
|
||||
```
|
||||
|
||||
This is a technique that can be applied both to the query that does the search and to the DB being searched.
|
||||
|
||||
## Compress embedding with MiniBatch KMeans
|
||||
If total_fea.npy is too large, it is possible to shrink the vector using KMeans.
|
||||
Compression of embedding is possible with the following code. Specify the size you want to compress for n_clusters, and specify 256 * number of CPU cores for batch_size to fully benefit from CPU parallelization.
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
from sklearn.cluster import MiniBatchKMeans
|
||||
kmeans = MiniBatchKMeans(n_clusters=10000, batch_size=256 * multiprocessing.cpu_count(), init="random")
|
||||
kmeans.fit(big_npy)
|
||||
sample_npy = kmeans.cluster_centers_
|
||||
```
|
||||
|
|
@ -9,13 +9,9 @@ RVCではHuBERTで変換した特徴量のEmbeddingに対し、学習データ
|
|||
|
||||
# 実装のoverview
|
||||
モデルが配置されている '/logs/your-experiment/3_feature256'には各音声データからHuBERTで抽出された特徴量が配置されています。
|
||||
ここからnpyファイルをファイル名でソートした順番で読み込み、ベクトルを連結してbig_npyを作成します。(このベクトルのshapeは[N, 256]です。)
|
||||
big_npyを/logs/your-experiment/total_fea.npyとして保存した後、faissを学習させます。
|
||||
ここからnpyファイルをファイル名でソートした順番で読み込み、ベクトルを連結してbig_npyを作成しfaissを学習させます。(このベクトルのshapeは[N, 256]です。)
|
||||
|
||||
2023/04/18時点ではfaissのindex factoryの機能を用いて、L2距離に基づくIVFを用いています。
|
||||
IVFの分割数(n_ivf)はN//39で、n_probeはint(np.power(n_ivf, 0.3))が採用されています。(infer-web.pyのtrain_index周りを探してください。)
|
||||
|
||||
本Tipsではまずこれらのパラメータの意味を解説し、その後よりよいindexを作成するための開発者向けアドバイスを書きます。
|
||||
本Tipsではまずこれらのパラメータの意味を解説します。
|
||||
|
||||
# 手法の解説
|
||||
## index factory
|
||||
|
|
@ -103,44 +99,3 @@ https://github.com/facebookresearch/faiss/wiki/Fast-accumulation-of-PQ-and-AQ-co
|
|||
## RFlat
|
||||
RFlatはFastScanで計算した大まかな距離を、index factoryの第三引数で指定した正確な距離で再計算する指示です。
|
||||
k個の近傍を取得する際は、k*k_factor個の点について再計算が行われます。
|
||||
|
||||
# Embeddingに関するテクニック
|
||||
## alpha query expansion
|
||||
クエリ拡張は検索で使われるテクニックで、例えば全文検索では入力された検索文に単語を幾つか追加することで検索精度を上げることがあります。ベクトル検索にもいくつか提唱されていて、その内追加の学習がいらず効果が高い手法としてα-query expansionが知られています。論文では[Attention-Based Query Expansion Learning](https://arxiv.org/abs/2007.08019)などで紹介されていて、[kaggleのshopeeコンペの2位の解法](https://www.kaggle.com/code/lyakaap/2nd-place-solution/notebook)にも用いられていました。
|
||||
|
||||
α-query expansionはあるベクトルに対し、近傍のベクトルを類似度のα乗した重みで足し合わせることでできます。いかにコードの例を張ります。big_npyをα query expansionしたものに置き換えます。
|
||||
|
||||
```python
|
||||
alpha = 3.
|
||||
index = faiss.index_factory(256, "IVF512,PQ128x4fs,RFlat")
|
||||
original_norm = np.maximum(np.linalg.norm(big_npy, ord=2, axis=1, keepdims=True), 1e-9)
|
||||
big_npy /= original_norm
|
||||
index.train(big_npy)
|
||||
index.add(big_npy)
|
||||
dist, neighbor = index.search(big_npy, num_expand)
|
||||
|
||||
expand_arrays = []
|
||||
ixs = np.arange(big_npy.shape[0])
|
||||
for i in range(-(-big_npy.shape[0]//batch_size)):
|
||||
ix = ixs[i*batch_size:(i+1)*batch_size]
|
||||
weight = np.power(np.einsum("nd,nmd->nm", big_npy[ix], big_npy[neighbor[ix]]), alpha)
|
||||
expand_arrays.append(np.sum(big_npy[neighbor[ix]] * np.expand_dims(weight, axis=2),axis=1))
|
||||
big_npy = np.concatenate(expand_arrays, axis=0)
|
||||
|
||||
# normalize index version
|
||||
big_npy = big_npy / np.maximum(np.linalg.norm(big_npy, ord=2, axis=1, keepdims=True), 1e-9)
|
||||
```
|
||||
|
||||
これは、検索を行うクエリにも、検索対象のDBにも適応可能なテクニックです。
|
||||
|
||||
## MiniBatch KMeansによるembeddingの圧縮
|
||||
total_fea.npyが大きすぎる場合、KMeansを用いてベクトルを小さくすることが可能です。
|
||||
以下のコードで、embeddingの圧縮が可能です。n_clustersは圧縮したい大きさを指定し、batch_sizeは256 * CPUのコア数を指定することでCPUの並列化の恩恵を十分に得ることができます。
|
||||
|
||||
```python
|
||||
import multiprocessing
|
||||
from sklearn.cluster import MiniBatchKMeans
|
||||
kmeans = MiniBatchKMeans(n_clusters=10000, batch_size=256 * multiprocessing.cpu_count(), init="random")
|
||||
kmeans.fit(big_npy)
|
||||
sample_npy = kmeans.cluster_centers_
|
||||
```
|
||||
|
|
@ -6,8 +6,12 @@ This TIPS explains how data training is done.
|
|||
I will explain along the steps in the training tab of the GUI.
|
||||
|
||||
## step1
|
||||
Set the experiment name here. You can also set here whether the model should take pitch into account.
|
||||
Data for each experiment is placed in `/logs/experiment name/`.
|
||||
Set the experiment name here.
|
||||
|
||||
You can also set here whether the model should take pitch into account.
|
||||
If the model doesn't consider pitch, the model will be lighter, but not suitable for singing.
|
||||
|
||||
Data for each experiment is placed in `/logs/your-experiment-name/`.
|
||||
|
||||
## step2a
|
||||
Loads and preprocesses audio.
|
||||
|
|
@ -23,14 +27,14 @@ After converting to int16 with ffmpeg, convert to float32 and normalize between
|
|||
The audio is smoothed by scipy's filtfilt.
|
||||
|
||||
### Audio Split
|
||||
First, the input audio is divided by detecting parts of silence that last longer than a certain period (max_sil_kept=5 seconds?). After splitting the audio on silence, split the audio every 4 seconds with an overlap of 0.3 seconds. For audio separated within 4 seconds, after normalizing the volume, convert the wav file to `/logs/experiment name/0_gt_wavs` and then convert it to 16k sampling rate to `/logs/experiment name/1_16k_wavs ` as a wav file.
|
||||
First, the input audio is divided by detecting parts of silence that last longer than a certain period (max_sil_kept=5 seconds?). After splitting the audio on silence, split the audio every 4 seconds with an overlap of 0.3 seconds. For audio separated within 4 seconds, after normalizing the volume, convert the wav file to `/logs/your-experiment-name/0_gt_wavs` and then convert it to 16k sampling rate to `/logs/your-experiment-name/1_16k_wavs ` as a wav file.
|
||||
|
||||
## step2b
|
||||
### Extract pitch
|
||||
Extract pitch information from wav files. Extract the pitch information (=f0) using the method built into parselmouth or pyworld and save it in `/logs/experiment name/2a_f0`. Then logarithmically convert the pitch information to an integer between 1 and 255 and save it in `/logs/experiment name/2b-f0nsf`.
|
||||
Extract pitch information from wav files. Extract the pitch information (=f0) using the method built into parselmouth or pyworld and save it in `/logs/your-experiment-name/2a_f0`. Then logarithmically convert the pitch information to an integer between 1 and 255 and save it in `/logs/your-experiment-name/2b-f0nsf`.
|
||||
|
||||
### Extract feature_print
|
||||
Convert the wav file to embedding in advance using HuBERT. Read the wav file saved in `/logs/experiment name/1_16k_wavs`, convert the wav file to 256-dimensional features with HuBERT, and save in npy format in `/logs/experiment name/3_feature256`.
|
||||
Convert the wav file to embedding in advance using HuBERT. Read the wav file saved in `/logs/your-experiment-name/1_16k_wavs`, convert the wav file to 256-dimensional features with HuBERT, and save in npy format in `/logs/your-experiment-name/3_feature256`.
|
||||
|
||||
## step3
|
||||
train the model.
|
||||
|
|
@ -40,11 +44,20 @@ In deep learning, the data set is divided and the learning proceeds little by li
|
|||
Therefore, the learning time is the learning time per step x (the number of data in the dataset / batch size) x the number of epochs. In general, the larger the batch size, the more stable the learning becomes (learning time per step ÷ batch size) becomes smaller, but it uses more GPU memory. GPU RAM can be checked with the nvidia-smi command. Learning can be done in a short time by increasing the batch size as much as possible according to the machine of the execution environment.
|
||||
|
||||
### Specify pretrained model
|
||||
RVC starts training the model from pretrained weights instead of from 0, so it can be trained with a small dataset. By default it loads `rvc-location/pretrained/f0G40k.pth` and `rvc-location/pretrained/f0D40k.pth`. When learning, model parameters are saved in `logs/experiment name/G_{}.pth` and `logs/experiment name/D_{}.pth` for each save_every_epoch, but by specifying this path, you can start learning. You can restart or start training from model weights learned in a different experiment.
|
||||
RVC starts training the model from pretrained weights instead of from 0, so it can be trained with a small dataset.
|
||||
|
||||
By default
|
||||
|
||||
- If you consider pitch, it loads `rvc-location/pretrained/f0G40k.pth` and `rvc-location/pretrained/f0D40k.pth`.
|
||||
- If you don't consider pitch, it loads `rvc-location/pretrained/f0G40k.pth` and `rvc-location/pretrained/f0D40k.pth`.
|
||||
|
||||
When learning, model parameters are saved in `logs/your-experiment-name/G_{}.pth` and `logs/your-experiment-name/D_{}.pth` for each save_every_epoch, but by specifying this path, you can start learning. You can restart or start training from model weights learned in a different experiment.
|
||||
|
||||
### learning index
|
||||
RVC saves the HuBERT feature values used during training, and during inference, searches for feature values that are similar to the feature values used during learning to perform inference. In order to perform this search at high speed, the index is learned in advance.
|
||||
For index learning, we use the approximate neighborhood search library faiss. Read the feature value of `/logs/experiment name/3_feature256`, save the combined feature value as `/logs/experiment name/total_fea.npy`, and use it to learn the index `/logs/experiment name Save it as /add_XXX.index`.
|
||||
For index learning, we use the approximate neighborhood search library faiss. Read the feature value of `logs/your-experiment-name/3_feature256` and use it to learn the index, and save it as `logs/your-experiment-name/add_XXX.index`.
|
||||
|
||||
(From the 20230428update version, it is read from the index, and saving / specifying is no longer necessary.)
|
||||
|
||||
### Button description
|
||||
- Train model: After executing step2b, press this button to train the model.
|
||||
|
|
|
|||
|
|
@ -6,7 +6,10 @@ RVCの訓練における説明、およびTIPS
|
|||
GUIの訓練タブのstepに沿って説明します。
|
||||
|
||||
## step1
|
||||
実験名の設定を行います。また、モデルにピッチを考慮させるかもここで設定できます。
|
||||
実験名の設定を行います。
|
||||
|
||||
また、モデルに音高ガイド(ピッチ)を考慮させるかもここで設定できます。考慮させない場合はモデルは軽量になりますが、歌唱には向かなくなります。
|
||||
|
||||
各実験のデータは`/logs/実験名/`に配置されます。
|
||||
|
||||
## step2a
|
||||
|
|
@ -40,11 +43,19 @@ HuBERTを用いてwavファイルを事前にembeddingに変換します。`/log
|
|||
そのため、学習時間は 1step当たりの学習時間 x (データセット内のデータ数 ÷ バッチサイズ) x epoch数 かかります。一般にバッチサイズを大きくするほど学習は安定し、(1step当たりの学習時間÷バッチサイズ)は小さくなりますが、その分GPUのメモリを多く使用します。GPUのRAMはnvidia-smiコマンド等で確認できます。実行環境のマシンに合わせてバッチサイズをできるだけ大きくするとより短時間で学習が可能です。
|
||||
|
||||
### pretrained modelの指定
|
||||
RVCではモデルの訓練を0からではなく、事前学習済みの重みから開始するため、少ないデータセットで学習を行えます。デフォルトでは`RVCのある場所/pretrained/f0G40k.pth`と`RVCのある場所/pretrained/f0D40k.pth`を読み込みます。学習時はsave_every_epochごとにモデルのパラメータが`logs/実験名/G_{}.pth`と`logs/実験名/D_{}.pth`に保存されますが、このパスを指定することで学習を再開したり、もしくは違う実験で学習したモデルの重みから学習を開始できます。
|
||||
RVCではモデルの訓練を0からではなく、事前学習済みの重みから開始するため、少ないデータセットで学習を行えます。
|
||||
|
||||
デフォルトでは
|
||||
|
||||
- 音高ガイドを考慮する場合、`RVCのある場所/pretrained/f0G40k.pth`と`RVCのある場所/pretrained/f0D40k.pth`を読み込みます。
|
||||
- 音高ガイドを考慮しない場合、`RVCのある場所/pretrained/G40k.pth`と`RVCのある場所/pretrained/D40k.pth`を読み込みます。
|
||||
|
||||
学習時はsave_every_epochごとにモデルのパラメータが`logs/実験名/G_{}.pth`と`logs/実験名/D_{}.pth`に保存されますが、このパスを指定することで学習を再開したり、もしくは違う実験で学習したモデルの重みから学習を開始できます。
|
||||
|
||||
### indexの学習
|
||||
RVCでは学習時に使われたHuBERTの特徴量を保存し、推論時は学習時の特徴量から近い特徴量を探してきて推論を行います。この検索を高速に行うために事前にindexの学習を行います。
|
||||
indexの学習には近似近傍探索ライブラリのfaissを用います。`/logs/実験名/3_feature256`の特徴量を読み込み、全て結合させた特徴量を`/logs/実験名/total_fea.npy`として保存、それを用いて学習したindexを`/logs/実験名/add_XXX.index`として保存します。
|
||||
indexの学習には近似近傍探索ライブラリのfaissを用います。`/logs/実験名/3_feature256`の特徴量を読み込み、それを用いて学習したindexを`/logs/実験名/add_XXX.index`として保存します。
|
||||
(20230428updateよりtotal_fea.npyはindexから読み込むので不要になりました。)
|
||||
|
||||
### ボタンの説明
|
||||
- モデルのトレーニング: step2bまでを実行した後、このボタンを押すとモデルの学習を行います。
|
||||
|
|
|
|||
Loading…
Reference in New Issue