badge icon

This article was automatically translated from the original Turkish version.

Blog
Blog
Avatar
AuthorBüşra Medine KurnazNovember 14, 2026 at 4:00 AM

How One Sound Is Transformed Into Another: The Technical Anatomy of RVC

T3 AI+1 More
Quote
T3 AI Developer
T3 AI Developer

Transformation of the Source Waveform on the RVC Line Based on Content, Pitch, and Target Voice Components (Generated by Artificial Intelligence.)

When the voice identity of a speaker is to be altered in a speech recording while preserving rhythm and emphasis, the voice conversion problem arises. This problem cannot be solved merely by raising the pitch (perceived pitch of the voice) or passing the voice through an effects chain. The speech content, pitch movements, vocal tract (the sound pathway formed by the throat, mouth and nasal cavities) characteristics, and other speaker-specific cues must be sufficiently disentangled from one another.


Retrieval-based Voice Conversion (RVC) is one of the open-source approaches that can implement this disentanglement. Content features extracted from the source voice, fundamental frequency information, and learned representations of the target voice are used together. Additionally, similar examples can be retrieved via an index of features built from training data. This aims to reduce the lingering timbral traces from the source speaker.【1】 A similar retrieval concept is also employed in the kNN-VC approach, where source features are replaced with the nearest neighbors among the target speaker’s features.【2】 In RVC, the retrieved features are mixed with the source features.


This article will explain the components of the RVC system, the problems each component addresses, and the decisions that must be made when designing an application. The provided settings will not be presented as universal optima; rather, they will serve as starting points enabling controlled comparisons on the same test inputs.

What Is Preserved and What Is Changed in Voice Conversion?

A single recording carries multiple types of information simultaneously. The spoken words, their durations, sentence melody, speech energy, accent, emotion, and traces of the recording environment are all superimposed within a single waveform. During voice conversion, not all of these are intended to be modified.

Flow Diagram of the Transformation of the Source Speech According to Its Content, Performance, and Speaker Identity Components (Drawing: Büşra Medine Kurnaz)

Typically, the following properties are expected to be preserved:

  • Linguistic content and word order
  • Timing of phonemes
  • Sentence rhythm and pauses
  • Emphasis and expression in the source performance.

The property targeted for change is usually the speaker identity. However, since speaker identity is not defined by a single parameter, this goal cannot be easily specified. Multiple cues are evaluated together, including the spectral envelope (the overall shape of energy across frequencies), formants (resonant frequencies amplified by the vocal tract), phonation type (the manner of vocal fold vibration), average pitch range (the typical high-low register of the voice), rhythmic habits, and accent.


In a simple pitch shifting (pitch shifting) operation, the entire spectrum can be shifted by a fixed factor. This may make the voice sound higher or lower, but it does not create a new speaker identity. Similarly, altering only the formants does not automatically generate the target speaker’s articulation, phonation, and prosody (melody, emphasis, rhythm, and duration patterns). Therefore, voice conversion systems require richer representations.

TTS,Voice Cloning and Voice Conversion: Key Differences

Although these concepts appear similar, they use different inputs and serve different purposes.


Table 1: Comparison of Text-to-Speech, Voice Conversion, Voice Cloning and DSP Approaches

Comparison of Voice Conversion and Text-to-Speech (TTS) Flows (Diagram: Büşra Medine Kurnaz)


In a TTS system, linguistic content is derived from text and speech is synthesized from scratch. In a voice conversion system, timing and expression are already present in the source audio, so much of the original performance can be directly transferred.


DSP is an abbreviation of the English term "digital signal processing" and is used in Turkish as "sayısal sinyal işleme". In this approach, a new speaker identity does not need to be learned by a model; instead, direct modifications such as pitch shifting, formant shifting, EQ, compression or various effects are applied to the existing waveform. Therefore, DSP can alter the character of the voice but should not be expected to fully replicate all characteristics of a target speaker.


Voice cloning can be used as a broader term. Extracting a speaker embedding from a short reference recording, adapting a TTS model to a target voice, or training a target speaker model for RVC can all fall under this category.【3】 Therefore, not every voice-cloning system is RVC; nor does every RVC application require extracting a new identity from scratch.

Internal Structure of the RVC Transformation Pipeline

The RVC pipeline can be divided into three main stages: analysis, matching, and resynthesis.

Analysis, Mapping, and Resynthesis Stages of the RVC Transformation Line (Drawing: Büşra Medine Kurnaz)

  1. Content features and fundamental frequency (F0) are extracted from the source voice.
  2. Content features are combined with similar features retrieved from the target training data.
  3. The combined representation is converted into a “waveform” conditioned on F0 and speaker identity.


In the official RVC implementation, HuBERT/ContentVec-based features are used, training features are searched within a FAISS index, and audio is generated using a VITS-derived synthesizer. The current model architecture includes a prior encoder (a network that extracts the probable latent distribution from content), a posterior encoder (a network that extracts latent representations from the actual target voice during training), normalizing flow (layers that establish a reversible transformation between the two latent distributions), speaker embedding, and an NSF-based decoder (a HiFi-GAN-style waveform generator conditioned on a harmonic source signal derived from F0).【4】

Training and Inference Stages

During training, posterior latent variables can be extracted from the actual target waveform or spectrogram and aligned with the prior distribution. During inference, since the actual target voice is unavailable, the prior encoder generates parameters of the latent distribution from the content features and F0 extracted from the source voice; the representation sampled from this distribution is passed through the inverse flow to the decoder. The variational structure and adversarial training in the VITS approach form the foundation of this distinction.【5】


The audio generation network leverages the HiFi-GAN approach, which employs parallel blocks operating at different widths to jointly process short-term and long-term audio patterns.【6】 During training, discriminative networks (discriminator) are used to compare the generated audio with the real audio. The waveform is analyzed across different period intervals to facilitate learning of the periodic and harmonic structure characteristic of natural speech. The discriminative networks are used only during training and do not participate in audio generation during inference.【7】

Layer Structure of HuBERT/ContentVec Feature Extraction and RVC v1/v2 Output Points (Drawing: Büşra Medine Kurnaz)

Content Representation with HuBERT and ContentVec

HuBERT is a self-supervised speech representation model designed to learn phonetic and acoustic patterns from speech signals. Latent features aligned with the temporal flow of the audio are extracted from the waveform. During pretraining, clustered hidden units are predicted for masked regions, enabling the learning of features that carry information about phonetic and acoustic structure.【8】


Different types of information can be carried at different layers. Lower layers may emphasize local acoustic details, while higher layers strengthen more abstract patterns related to content.【9】 However, speaker information is not entirely eliminated. In ContentVec, mechanisms are added on top of HuBERT to reduce speaker information while preserving content.【10】


In the current RVC code, two feature configurations are clearly distinguished:

  • In RVC v1, the output of the 9th layer is passed through a projection layer to produce a 256-dimensional feature.
  • In RVC v2, the 768-dimensional output of the 12th (final) encoder layer is used directly.

The conversion of the HuBERT input to 16 kHz is performed to ensure compatibility with the training configuration of the feature encoder.【11】 The final model’s ability to generate audio at 40 or 48 kHz does not require the content encoder to operate at the same sampling rate. Different copies of the same audio can be maintained at different sampling rates for different purposes.

Why Is F0 Extracted Separately in RVC?

HuBERT features represent the content of speech but do not explicitly encode the pitch movements of the utterance. To preserve the melody of the source speech, F0 is extracted using a separate method and provided to the RVC model as additional information.


Voicing is the condition in which the vocal folds vibrate during sound production. F0 indicates the number of vibrations per second. It is measurable in voiced regions such as vowels but is meaningless in voiceless sounds like s or f or during silence.

Diagram showing that the F0 curve is found only in phonated regions, along with energy and phonation status (generated by artificial intelligence).


Three common problems arise during F0 estimation:

  • The absence of meaningful F0 in voiceless regions
  • Octave errors, where the estimated frequency is half or double the true value
  • Incorrect tracking due to noise, reverb (room echo), or music

Names such as RMVPE, CREPE, Harvest, and DIO seen in RVC interfaces do not refer to different voice-conversion models but rather to different methods for extracting the F0 contour.【12】 Knowledge of their internal workings is not necessary for most use cases:

  • RMVPE is a strong starting option, especially for vocal and noisy recordings.【13】
  • CREPE can provide detailed pitch tracking but may require more computational resources.【14】
  • Harvest and DIO are signal processing based alternatives.【15】

The appropriate method can be selected by converting the same short recording with several options and listening for F0 breaks, sudden pitch jumps, or metallic distortions. Not all of these methods may be available in the RVC version being used.


Pitch transpose can be applied in semitone increments. For an n-semitone shift, the F0 value is transformed using a formula. In RVC, this transformation is applied to the F0 contour.【16】 However, large transpositions can disrupt the natural relationship between formant and pitch. Therefore, the transpose value should be chosen by listening to both the target speaker’s natural range and the source performance.

Difference Between .pth and .index Files

In RVC models, .pth and .index files serve distinct roles:


Table 2: Roles of .pth and .index Files in RVC Models

Speech Generation with Feature Blending of .pth and .index Components (Illustration: Büşra Medine Kurnaz)


.pth contains the model’s learned weights and forms the foundation of speech generation. .index organizes HuBERT features extracted from target voices into a searchable catalog. Features extracted from the source voice can be augmented by matching them with similar target features in this catalog.【17】 This blending can be conceptually represented as:



Here, α is the index_rate value. A value of 0 disables .index usage; increasing the value enhances the influence of target features. However, if the target data is of poor quality, this may also amplify noise and articulation artifacts.

How Is Target Speech Data Prepared?

Data Preparation Flow from Raw Records to Training Data (Diagram: Büşra Medine Kurnaz)

The consistency of the dataset is as important as the model architecture. If the room, microphone, or noise conditions change constantly alongside the target speaker, it becomes unclear which features are being learned.


The key characteristics to observe during data selection can be summarized as follows:

  • Select a single target speaker with authorized usage rights.
  • Clipping, recordings with heavy reverb, or fixed background music must be filtered out.
  • Phonetic coverage must be diversified; do not rely solely on short sentences with identical patterns.
  • Balance normal speech, questions, emphasis, and varying sentence lengths.
  • Separate training and evaluation sentences from the outset.


The official RVC README recommends at least ten minutes of low-noise speech as a practical starting point.【18】 This value does not guarantee quality. Phonetic diversity, recording cleanliness, and the intended use case may be more decisive than total duration.

Preprocessing

Waveform, Spectrogram, F0, and Energy Representations of the Same Utterance (Generated by Artificial Intelligence.)

Silence trimming, splitting long recordings into segments, resampling rate conversion, and loudness control can be applied. However, the output of automated processes must not be accepted without listening. An aggressive trim may cut off plosive consonants at word beginnings or natural breaths. Too-short segments may lose context; too-long segments may increase memory consumption and cause batch imbalance.


Before feature extraction, file-feature alignment must be preserved. The waveform segment, F0 sequence, and HuBERT matrix must represent the same time interval. Incorrect filenames or mismatched hop sizes (number of samples between consecutive analysis frames) can cause silent but destructive training errors.


Practical Check: Before starting training, plot the waveform, spectrogram, F0, and feature length for a few randomly selected segments together.

How Is RVC Data Diversified with Limited Recordings?

When only a few short recordings of the target person are available, voice identity can be partially learned; however, phonetic diversity, varying sentence lengths, and pitch movements may not be adequately represented. When the recommended minimum of ten minutes of low-noise recordings cannot be achieved, synthetic data can be used to supplement real recordings. A critical distinction must be made here: generated text alone is not training data. These texts must be vocalized in the target person’s voice using a TTS system with voice-cloning capabilities that can reference the real short recordings. Adding a standard TTS voice from another speaker to the target dataset will cause the RVC model to attempt learning two distinct voice identities simultaneously.

Real and Synthetic Data Distinction and Evaluation Pipeline for Low-Resource Target Speaker (Diagram: Büşra Medine Kurnaz)


The applicable workflow is as follows:

  1. Noisy, reverberant, or corrupted portions of real recordings are filtered out, and clean speech is split into short segments.
  2. Where possible, a few real segments are reserved solely for evaluation and not used in TTS or RVC training.
  3. The remaining real recordings are provided as reference voice input to a voice-cloning TTS system.
  4. Texts covering diverse sounds, words, question structures, and sentence lengths are synthesized in the target voice.
  5. Synthetic recordings with mispronunciations, metallic tone, unstable pitch, or timbre deviating from the target person are reviewed and filtered out.
  6. Accepted synthetic recordings are combined with real training recordings, and the RVC model is trained on this mixed dataset.

The proportion of synthetic recordings within the total dataset should be limited, and the weight of real recordings must be preserved. If necessary, real recordings can be sampled more frequently in the training list. Two RVC models can be trained for direct comparison: one using only real recordings and another using both real and synthetic recordings. When both models are evaluated against real recordings excluded from training, it becomes clear whether synthetic data provides genuine benefit.


Synthetic data does not create new or reliable speaker details absent from real recordings; it merely extends the inferred voice identity from the limited reference to more varied sentences. Artificial rhythm, missing breaths, or spectral distortions in the TTS system may also be learned by RVC. Therefore, similarity to the target voice and recording quality must take precedence over data quantity. Reusing the output of an initial RVC model as new training data should be avoided, as it risks amplifying the same errors.

TTS and RVC Integration

Inference, when direct speech generation from text targeting a specific character is desired, TTS and RVC can be chained together. TTS generates linguistic content, pronunciation, durations, and basic prosody. RVC then transforms this performance to match the speaker identity of the target model. RVC does not automatically correct mispronunciations or incorrect pauses from TTS; these errors are carried forward as part of the source performance.


The following points should be checked between the two stages:

  • Sampling rate and number of channels
  • Clipping and DC offset (the persistent shift of the waveform away from the zero axis)
  • Silence durations
  • Loudness consistency
  • Artifacts produced by TTS
  • Alignment of the F0 range with the target voice

If the intermediate WAV file is saved in a lossless format, it becomes easier to determine whether the issue originates from TTS or RVC. Even if the production pipeline runs entirely in memory, saving intermediate outputs during debugging mode can be beneficial.

Real-Time Transformation and Streaming

In file-based inference, transformation can occur only after the entire audio has been generated. In a real-time system, short segments (chunks) must be processed and fed into the playback queue.

Streaming Diagram Showing Real-Time Conversion of TTS Segments with RVC, Overlap and Crossfade Regions, and Latency Components (Generated by Artificial Intelligence.)


Chunk size that is too short may lack sufficient phonetic and acoustic context. If too long, the initial latency increases. Additionally, at the boundaries between two chunks, the waveform phase, F0, loudness, or model context may differ, leading to clicks, discontinuities, and sudden timbral shifts.


To mitigate these issues:

  • Adjacent chunks can be prepared with overlap.
  • A crossfade (gradually reducing the volume of one segment while increasing the other) can be applied in overlapping regions.
  • Left and right context can be added to the model input, with only the central portion used for output.
  • The F0 contour can be smoothed jointly across chunk boundaries.
  • A jitter buffer can be maintained between generation and playback.
  • The real-time factor (RTF; the ratio of processing time to audio duration) can be monitored by comparing processing time to actual audio duration.

Real-time performance is not determined solely by GPU inference time. Audio capture buffers, resampling, feature extraction, retrieval, decoder, crossfade, and audio driver latencies must all be measured together.

How Is the RVC Output Evaluated?

Separation of Voice Conversion Output According to Evaluation Criteria (Diagram: Büşra Medine Kurnaz)

Voice conversion quality cannot be summarized by a single metric. An output that resembles the target speaker but is unintelligible, or one that sounds very natural but resembles the wrong speaker, can both be considered failures.


  • Speaker Similarity: Embeddings can be extracted using speaker-verification models such as ECAPA-TDNN, which assess whether two audio recordings belong to the same speaker, and then cosine similarity can be computed against target references. Using the average of multiple clean recordings rather than a single reference can provide a more stable target.【19】 However, the training domain of the speaker encoder may affect scores due to language and recording conditions.
  • Intelligibility: Word error rate (WER) or character error rate (CER) can be measured using a Turkish automatic speech recognition (ASR) system. The source and converted audio can be compared against the same transcription to assess content preservation. The score of the clean source recording should also be reported to avoid misattributing ASR errors directly to voice conversion errors.
  • F0 and Prosody: Mean and standard deviation of F0 can serve as initial metrics; however, contour correlation, voiced/unvoiced error rate, and duration alignment provide more detailed information. Simply matching the mean F0 to the target is not sufficient evidence of speaker identity.
  • Naturalness and Artifacts: In human listening tests, naturalness, speaker similarity, and intelligibility should be measured with separate questions. Files should be presented with matched loudness and anonymized identities. Only developer-led open listening is susceptible to expectation bias.
  • Performance: Initial speech latency, processing time per chunk, real-time factor, GPU memory usage, and number of interruptions can be measured. When quality and latency are presented in the same table, the system’s suitability for its intended use case becomes clearer.

Conclusion

In RVC, content representations extracted from the source speech via HuBERT or ContentVec are combined with separately obtained F0 information, the target speaker condition, and target features retrieved through retrieval to generate a new waveform. The resulting quality depends not only on the model architecture but also on clean and diverse training data, carefully curated synthetic recordings, appropriate inference settings, and consistent evaluation criteria. Therefore, RVC should not be viewed as a single-step audio effect but as a multi-stage artificial intelligence system whose components must be analyzed individually.

Bibliographies

Baas, Matthew, Benjamin van Niekerk, and Herman Kamper. “Voice Conversion With Just Nearest Neighbors.” arXiv 2023. Accessed October 31, 2026.https://arxiv.org/abs/2305.18975

Desplanques, Brecht, Jenthe Thienpondt, and Kris Demuynck. “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification.” arXiv 2020. Accessed November 3, 2026.https://arxiv.org/abs/2005.07143

Facebook Research. “Faiss.” GitHub. Accessed November 4, 2026.https://github.com/facebookresearch/faiss

Hsu, Wei-Ning, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units.” arXiv 2021. Accessed October 29, 2026.https://arxiv.org/abs/2106.07447

Jia, Ye, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis.” arXiv 2019. Accessed November 1, 2026.https://arxiv.org/abs/1806.04558

Johnson, Jeff, Matthijs Douze, and Hervé Jégou. “Billion-Scale Similarity Search With GPUs.” arXiv 2017. Accessed November 2, 2026.https://arxiv.org/abs/1702.08734

Kim, Jaehyeon, Jungil Kong, and Juhee Son. “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.” Proceedings of the 38th International Conference on Machine Learning, PMLR 139 (2021): 5530-5540. Accessed November 4, 2026.https://proceedings.mlr.press/v139/kim21f.html

Kim, Jong Wook, Justin Salamon, Peter Li, and Juan Pablo Bello. “CREPE: A Convolutional Representation for Pitch Estimation.” arXiv 2018. Accessed October 25, 2026.https://arxiv.org/abs/1802.06182

Kong, Jungil, Jaehyeon Kim, and Jaekyoung Bae. “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis.” Advances in Neural Information Processing Systems 33 (NeurIPS 2020): 1-12. Accessed November 5, 2026.https://proceedings.neurips.cc/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html

Morise, Masanori, Fumiya Yokomori, and Kenji Ozawa. “WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications.” IEICE Transactions on Information and Systems E99-D (7) (2016): 1877-1884. Accessed November 1, 2026.https://doi.org/10.1587/transinf.2015EDP7457

Qian, Kaizhi, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang. “ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers.” Proceedings of the 39th International Conference on Machine Learning, PMLR 162 (2022): 18003-18017. Accessed November 2, 2026.https://proceedings.mlr.press/v162/qian22b.html

RVC-Project. “Retrieval-based Voice Conversion WebUI.” GitHub. Accessed October 26, 2026.https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI

Wang, Xin, Shinji Takaki, and Junichi Yamagishi. “Neural Source-Filter-Based Waveform Model for Statistical Parametric Speech Synthesis.” arXiv 2019. Accessed October 31, 2026.https://arxiv.org/abs/1810.11946

Wei, Haojie, Xueke Cao, Tangpeng Dan, and Yueguo Chen. “RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music.” Interspeech. (2023): 5421-5425. Accessed November 3, 2026.https://www.isca-archive.org/interspeech_2023/wei23b_interspeech.pdf

Citations

  • [1]

    RVC Project, "Retrieval-based Voice Conversion WebUI," GitHub, Accessed October 26, 2026.https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI

  • [2]

    Matthew Baas, Benjamin van Niekerk, and Herman Kamper, “Voice Conversion With Just Nearest Neighbors,” arXiv 2023, 1-2, Accessed October 31, 2026.https://arxiv.org/abs/2305.18975

  • [3]

    Ye Jia, Yu Zhang, Ron J. Weiss, et al., “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis,” arXiv 2019, 1-2, Accessed November 1, 2026.https://arxiv.org/abs/1806.04558; RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [4]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [5]

    Jaehyeon Kim, Jungil Kong, and Juhee Son, “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech,” Proceedings of the 38th International Conference on Machine Learning, PMLR 139 (2021), 5531–5532, Accessed November 4, 2026.https://proceedings.mlr.press/v139/kim21f.html

  • [6]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2-3, Accessed November 5, 2026.https://proceedings.neurips.cc/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html

  • [7]

    Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” 3-4.

  • [8]

    Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” arXiv 2021, 1-3, Accessed October 29, 2026.https://arxiv.org/abs/2106.07447

  • [9]

    Wei-Ning Hsu et al., “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” 7-8.

  • [10]

    Kaizhi Qian, Yang Zhang, Heting Gao, et al. “ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers,” Proceedings of the 39th International Conference on Machine Learning, PMLR 162 (2022): 18003–18007. Accessed November 2, 2026.https://proceedings.mlr.press/v162/qian22b.html

  • [11]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [12]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [13]

    Haojie Wei, Xueke Cao, Tangpeng Dan, and Yueguo Chen, “RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music,” Interspeech, (2023), 5421, 5423–5424, Accessed November 3, 2026.https://www.isca-archive.org/interspeech_2023/wei23b_interspeech.pdf

  • [14]

    Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello, “CREPE: A Convolutional Representation for Pitch Estimation,” arXiv 2018, 1-3, Accessed October 25, 2026.https://arxiv.org/abs/1802.06182

  • [15]

    Masanori Morise, Fumiya Yokomori, and Kenji Ozawa, “WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications,” IEICE Transactions on Information and Systems E99-D (7), (2016), 1877-1884, Accessed November 1, 2026.https://doi.org/10.1587/transinf.2015EDP7457

  • [16]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [17]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [18]

    RVC-Project, “Retrieval-based Voice Conversion WebUI.”

  • [19]

    Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” arXiv 2020, 1, 3-4, Accessed November 3, 2026.https://arxiv.org/abs/2005.07143

T3 AI
Türkiye's Large Language Model,is growing together with you.

Contents

  • What Is Preserved and What Is Changed in Voice Conversion?

  • TTS,Voice Cloning and Voice Conversion: Key Differences

  • Internal Structure of the RVC Transformation Pipeline

    • Training and Inference Stages

  • Content Representation with HuBERT and ContentVec

  • Why Is F0 Extracted Separately in RVC?

  • Difference Between .pth and .index Files

  • How Is Target Speech Data Prepared?

    • Preprocessing

  • How Is RVC Data Diversified with Limited Recordings?

  • TTS and RVC Integration

  • Real-Time Transformation and Streaming

  • How Is the RVC Output Evaluated?

  • Conclusion