This article was automatically translated from the original Turkish version.


Transformation of the Source Waveform on the RVC Line Based on Content, Pitch, and Target Voice Components (Generated by Artificial Intelligence.)
When the voice identity of a speaker is to be altered in a speech recording while preserving rhythm and emphasis, the voice conversion problem arises. This problem cannot be solved merely by raising the pitch (perceived pitch of the voice) or passing the voice through an effects chain. The speech content, pitch movements, vocal tract (the sound pathway formed by the throat, mouth and nasal cavities) characteristics, and other speaker-specific cues must be sufficiently disentangled from one another.
Retrieval-based Voice Conversion (RVC) is one of the open-source approaches that can implement this disentanglement. Content features extracted from the source voice, fundamental frequency information, and learned representations of the target voice are used together. Additionally, similar examples can be retrieved via an index of features built from training data. This aims to reduce the lingering timbral traces from the source speaker.【1】 A similar retrieval concept is also employed in the kNN-VC approach, where source features are replaced with the nearest neighbors among the target speaker’s features.【2】 In RVC, the retrieved features are mixed with the source features.
This article will explain the components of the RVC system, the problems each component addresses, and the decisions that must be made when designing an application. The provided settings will not be presented as universal optima; rather, they will serve as starting points enabling controlled comparisons on the same test inputs.
A single recording carries multiple types of information simultaneously. The spoken words, their durations, sentence melody, speech energy, accent, emotion, and traces of the recording environment are all superimposed within a single waveform. During voice conversion, not all of these are intended to be modified.

Flow Diagram of the Transformation of the Source Speech According to Its Content, Performance, and Speaker Identity Components (Drawing: Büşra Medine Kurnaz)
Typically, the following properties are expected to be preserved:
The property targeted for change is usually the speaker identity. However, since speaker identity is not defined by a single parameter, this goal cannot be easily specified. Multiple cues are evaluated together, including the spectral envelope (the overall shape of energy across frequencies), formants (resonant frequencies amplified by the vocal tract), phonation type (the manner of vocal fold vibration), average pitch range (the typical high-low register of the voice), rhythmic habits, and accent.
In a simple pitch shifting (pitch shifting) operation, the entire spectrum can be shifted by a fixed factor. This may make the voice sound higher or lower, but it does not create a new speaker identity. Similarly, altering only the formants does not automatically generate the target speaker’s articulation, phonation, and prosody (melody, emphasis, rhythm, and duration patterns). Therefore, voice conversion systems require richer representations.
Although these concepts appear similar, they use different inputs and serve different purposes.
Table 1: Comparison of Text-to-Speech, Voice Conversion, Voice Cloning and DSP Approaches

Comparison of Voice Conversion and Text-to-Speech (TTS) Flows (Diagram: Büşra Medine Kurnaz)
In a TTS system, linguistic content is derived from text and speech is synthesized from scratch. In a voice conversion system, timing and expression are already present in the source audio, so much of the original performance can be directly transferred.
DSP is an abbreviation of the English term "digital signal processing" and is used in Turkish as "sayısal sinyal işleme". In this approach, a new speaker identity does not need to be learned by a model; instead, direct modifications such as pitch shifting, formant shifting, EQ, compression or various effects are applied to the existing waveform. Therefore, DSP can alter the character of the voice but should not be expected to fully replicate all characteristics of a target speaker.
Voice cloning can be used as a broader term. Extracting a speaker embedding from a short reference recording, adapting a TTS model to a target voice, or training a target speaker model for RVC can all fall under this category.【3】 Therefore, not every voice-cloning system is RVC; nor does every RVC application require extracting a new identity from scratch.
The RVC pipeline can be divided into three main stages: analysis, matching, and resynthesis.

Analysis, Mapping, and Resynthesis Stages of the RVC Transformation Line (Drawing: Büşra Medine Kurnaz)
In the official RVC implementation, HuBERT/ContentVec-based features are used, training features are searched within a FAISS index, and audio is generated using a VITS-derived synthesizer. The current model architecture includes a prior encoder (a network that extracts the probable latent distribution from content), a posterior encoder (a network that extracts latent representations from the actual target voice during training), normalizing flow (layers that establish a reversible transformation between the two latent distributions), speaker embedding, and an NSF-based decoder (a HiFi-GAN-style waveform generator conditioned on a harmonic source signal derived from F0).【4】
During training, posterior latent variables can be extracted from the actual target waveform or spectrogram and aligned with the prior distribution. During inference, since the actual target voice is unavailable, the prior encoder generates parameters of the latent distribution from the content features and F0 extracted from the source voice; the representation sampled from this distribution is passed through the inverse flow to the decoder. The variational structure and adversarial training in the VITS approach form the foundation of this distinction.【5】
The audio generation network leverages the HiFi-GAN approach, which employs parallel blocks operating at different widths to jointly process short-term and long-term audio patterns.【6】 During training, discriminative networks (discriminator) are used to compare the generated audio with the real audio. The waveform is analyzed across different period intervals to facilitate learning of the periodic and harmonic structure characteristic of natural speech. The discriminative networks are used only during training and do not participate in audio generation during inference.【7】

Layer Structure of HuBERT/ContentVec Feature Extraction and RVC v1/v2 Output Points (Drawing: Büşra Medine Kurnaz)
HuBERT is a self-supervised speech representation model designed to learn phonetic and acoustic patterns from speech signals. Latent features aligned with the temporal flow of the audio are extracted from the waveform. During pretraining, clustered hidden units are predicted for masked regions, enabling the learning of features that carry information about phonetic and acoustic structure.【8】
Different types of information can be carried at different layers. Lower layers may emphasize local acoustic details, while higher layers strengthen more abstract patterns related to content.【9】 However, speaker information is not entirely eliminated. In ContentVec, mechanisms are added on top of HuBERT to reduce speaker information while preserving content.【10】
In the current RVC code, two feature configurations are clearly distinguished:
The conversion of the HuBERT input to 16 kHz is performed to ensure compatibility with the training configuration of the feature encoder.【11】 The final model’s ability to generate audio at 40 or 48 kHz does not require the content encoder to operate at the same sampling rate. Different copies of the same audio can be maintained at different sampling rates for different purposes.
HuBERT features represent the content of speech but do not explicitly encode the pitch movements of the utterance. To preserve the melody of the source speech, F0 is extracted using a separate method and provided to the RVC model as additional information.
Voicing is the condition in which the vocal folds vibrate during sound production. F0 indicates the number of vibrations per second. It is measurable in voiced regions such as vowels but is meaningless in voiceless sounds like s or f or during silence.

Diagram showing that the F0 curve is found only in phonated regions, along with energy and phonation status (generated by artificial intelligence).
Three common problems arise during F0 estimation:
Names such as RMVPE, CREPE, Harvest, and DIO seen in RVC interfaces do not refer to different voice-conversion models but rather to different methods for extracting the F0 contour.【12】 Knowledge of their internal workings is not necessary for most use cases:
The appropriate method can be selected by converting the same short recording with several options and listening for F0 breaks, sudden pitch jumps, or metallic distortions. Not all of these methods may be available in the RVC version being used.
Pitch transpose can be applied in semitone increments. For an n-semitone shift, the F0 value is transformed using a formula. In RVC, this transformation is applied to the F0 contour.【16】 However, large transpositions can disrupt the natural relationship between formant and pitch. Therefore, the transpose value should be chosen by listening to both the target speaker’s natural range and the source performance.
In RVC models, .pth and .index files serve distinct roles:
Table 2: Roles of .pth and .index Files in RVC Models

Speech Generation with Feature Blending of .pth and .index Components (Illustration: Büşra Medine Kurnaz)
.pth contains the model’s learned weights and forms the foundation of speech generation. .index organizes HuBERT features extracted from target voices into a searchable catalog. Features extracted from the source voice can be augmented by matching them with similar target features in this catalog.【17】 This blending can be conceptually represented as:
Here, α is the index_rate value. A value of 0 disables .index usage; increasing the value enhances the influence of target features. However, if the target data is of poor quality, this may also amplify noise and articulation artifacts.

Data Preparation Flow from Raw Records to Training Data (Diagram: Büşra Medine Kurnaz)
The consistency of the dataset is as important as the model architecture. If the room, microphone, or noise conditions change constantly alongside the target speaker, it becomes unclear which features are being learned.
The key characteristics to observe during data selection can be summarized as follows:
The official RVC README recommends at least ten minutes of low-noise speech as a practical starting point.【18】 This value does not guarantee quality. Phonetic diversity, recording cleanliness, and the intended use case may be more decisive than total duration.

Waveform, Spectrogram, F0, and Energy Representations of the Same Utterance (Generated by Artificial Intelligence.)
Silence trimming, splitting long recordings into segments, resampling rate conversion, and loudness control can be applied. However, the output of automated processes must not be accepted without listening. An aggressive trim may cut off plosive consonants at word beginnings or natural breaths. Too-short segments may lose context; too-long segments may increase memory consumption and cause batch imbalance.
Before feature extraction, file-feature alignment must be preserved. The waveform segment, F0 sequence, and HuBERT matrix must represent the same time interval. Incorrect filenames or mismatched hop sizes (number of samples between consecutive analysis frames) can cause silent but destructive training errors.
Practical Check: Before starting training, plot the waveform, spectrogram, F0, and feature length for a few randomly selected segments together.
When only a few short recordings of the target person are available, voice identity can be partially learned; however, phonetic diversity, varying sentence lengths, and pitch movements may not be adequately represented. When the recommended minimum of ten minutes of low-noise recordings cannot be achieved, synthetic data can be used to supplement real recordings. A critical distinction must be made here: generated text alone is not training data. These texts must be vocalized in the target person’s voice using a TTS system with voice-cloning capabilities that can reference the real short recordings. Adding a standard TTS voice from another speaker to the target dataset will cause the RVC model to attempt learning two distinct voice identities simultaneously.

Real and Synthetic Data Distinction and Evaluation Pipeline for Low-Resource Target Speaker (Diagram: Büşra Medine Kurnaz)
The applicable workflow is as follows:
The proportion of synthetic recordings within the total dataset should be limited, and the weight of real recordings must be preserved. If necessary, real recordings can be sampled more frequently in the training list. Two RVC models can be trained for direct comparison: one using only real recordings and another using both real and synthetic recordings. When both models are evaluated against real recordings excluded from training, it becomes clear whether synthetic data provides genuine benefit.
Synthetic data does not create new or reliable speaker details absent from real recordings; it merely extends the inferred voice identity from the limited reference to more varied sentences. Artificial rhythm, missing breaths, or spectral distortions in the TTS system may also be learned by RVC. Therefore, similarity to the target voice and recording quality must take precedence over data quantity. Reusing the output of an initial RVC model as new training data should be avoided, as it risks amplifying the same errors.
Inference, when direct speech generation from text targeting a specific character is desired, TTS and RVC can be chained together. TTS generates linguistic content, pronunciation, durations, and basic prosody. RVC then transforms this performance to match the speaker identity of the target model. RVC does not automatically correct mispronunciations or incorrect pauses from TTS; these errors are carried forward as part of the source performance.
The following points should be checked between the two stages:
If the intermediate WAV file is saved in a lossless format, it becomes easier to determine whether the issue originates from TTS or RVC. Even if the production pipeline runs entirely in memory, saving intermediate outputs during debugging mode can be beneficial.
In file-based inference, transformation can occur only after the entire audio has been generated. In a real-time system, short segments (chunks) must be processed and fed into the playback queue.

Streaming Diagram Showing Real-Time Conversion of TTS Segments with RVC, Overlap and Crossfade Regions, and Latency Components (Generated by Artificial Intelligence.)
Chunk size that is too short may lack sufficient phonetic and acoustic context. If too long, the initial latency increases. Additionally, at the boundaries between two chunks, the waveform phase, F0, loudness, or model context may differ, leading to clicks, discontinuities, and sudden timbral shifts.
To mitigate these issues:
Real-time performance is not determined solely by GPU inference time. Audio capture buffers, resampling, feature extraction, retrieval, decoder, crossfade, and audio driver latencies must all be measured together.

Separation of Voice Conversion Output According to Evaluation Criteria (Diagram: Büşra Medine Kurnaz)
Voice conversion quality cannot be summarized by a single metric. An output that resembles the target speaker but is unintelligible, or one that sounds very natural but resembles the wrong speaker, can both be considered failures.
In RVC, content representations extracted from the source speech via HuBERT or ContentVec are combined with separately obtained F0 information, the target speaker condition, and target features retrieved through retrieval to generate a new waveform. The resulting quality depends not only on the model architecture but also on clean and diverse training data, carefully curated synthetic recordings, appropriate inference settings, and consistent evaluation criteria. Therefore, RVC should not be viewed as a single-step audio effect but as a multi-stage artificial intelligence system whose components must be analyzed individually.
Baas, Matthew, Benjamin van Niekerk, and Herman Kamper. “Voice Conversion With Just Nearest Neighbors.” arXiv 2023. Accessed October 31, 2026.https://arxiv.org/abs/2305.18975
Desplanques, Brecht, Jenthe Thienpondt, and Kris Demuynck. “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification.” arXiv 2020. Accessed November 3, 2026.https://arxiv.org/abs/2005.07143
Facebook Research. “Faiss.” GitHub. Accessed November 4, 2026.https://github.com/facebookresearch/faiss
Hsu, Wei-Ning, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units.” arXiv 2021. Accessed October 29, 2026.https://arxiv.org/abs/2106.07447
Jia, Ye, Yu Zhang, Ron J. Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang, Ignacio Lopez Moreno, and Yonghui Wu. “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis.” arXiv 2019. Accessed November 1, 2026.https://arxiv.org/abs/1806.04558
Johnson, Jeff, Matthijs Douze, and Hervé Jégou. “Billion-Scale Similarity Search With GPUs.” arXiv 2017. Accessed November 2, 2026.https://arxiv.org/abs/1702.08734
Kim, Jaehyeon, Jungil Kong, and Juhee Son. “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech.” Proceedings of the 38th International Conference on Machine Learning, PMLR 139 (2021): 5530-5540. Accessed November 4, 2026.https://proceedings.mlr.press/v139/kim21f.html
Kim, Jong Wook, Justin Salamon, Peter Li, and Juan Pablo Bello. “CREPE: A Convolutional Representation for Pitch Estimation.” arXiv 2018. Accessed October 25, 2026.https://arxiv.org/abs/1802.06182
Kong, Jungil, Jaehyeon Kim, and Jaekyoung Bae. “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis.” Advances in Neural Information Processing Systems 33 (NeurIPS 2020): 1-12. Accessed November 5, 2026.https://proceedings.neurips.cc/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html
Morise, Masanori, Fumiya Yokomori, and Kenji Ozawa. “WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications.” IEICE Transactions on Information and Systems E99-D (7) (2016): 1877-1884. Accessed November 1, 2026.https://doi.org/10.1587/transinf.2015EDP7457
Qian, Kaizhi, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang. “ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers.” Proceedings of the 39th International Conference on Machine Learning, PMLR 162 (2022): 18003-18017. Accessed November 2, 2026.https://proceedings.mlr.press/v162/qian22b.html
RVC-Project. “Retrieval-based Voice Conversion WebUI.” GitHub. Accessed October 26, 2026.https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
Wang, Xin, Shinji Takaki, and Junichi Yamagishi. “Neural Source-Filter-Based Waveform Model for Statistical Parametric Speech Synthesis.” arXiv 2019. Accessed October 31, 2026.https://arxiv.org/abs/1810.11946
Wei, Haojie, Xueke Cao, Tangpeng Dan, and Yueguo Chen. “RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music.” Interspeech. (2023): 5421-5425. Accessed November 3, 2026.https://www.isca-archive.org/interspeech_2023/wei23b_interspeech.pdf
[1]
RVC Project, "Retrieval-based Voice Conversion WebUI," GitHub, Accessed October 26, 2026.https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI
[2]
Matthew Baas, Benjamin van Niekerk, and Herman Kamper, “Voice Conversion With Just Nearest Neighbors,” arXiv 2023, 1-2, Accessed October 31, 2026.https://arxiv.org/abs/2305.18975
[3]
Ye Jia, Yu Zhang, Ron J. Weiss, et al., “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis,” arXiv 2019, 1-2, Accessed November 1, 2026.https://arxiv.org/abs/1806.04558; RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[4]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[5]
Jaehyeon Kim, Jungil Kong, and Juhee Son, “Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech,” Proceedings of the 38th International Conference on Machine Learning, PMLR 139 (2021), 5531–5532, Accessed November 4, 2026.https://proceedings.mlr.press/v139/kim21f.html
[6]
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2-3, Accessed November 5, 2026.https://proceedings.neurips.cc/paper/2020/hash/c5d736809766d46260d816d8dbc9eb44-Abstract.html
[7]
Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,” 3-4.
[8]
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed, “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” arXiv 2021, 1-3, Accessed October 29, 2026.https://arxiv.org/abs/2106.07447
[9]
Wei-Ning Hsu et al., “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units,” 7-8.
[10]
Kaizhi Qian, Yang Zhang, Heting Gao, et al. “ContentVec: An Improved Self-Supervised Speech Representation by Disentangling Speakers,” Proceedings of the 39th International Conference on Machine Learning, PMLR 162 (2022): 18003–18007. Accessed November 2, 2026.https://proceedings.mlr.press/v162/qian22b.html
[11]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[12]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[13]
Haojie Wei, Xueke Cao, Tangpeng Dan, and Yueguo Chen, “RMVPE: A Robust Model for Vocal Pitch Estimation in Polyphonic Music,” Interspeech, (2023), 5421, 5423–5424, Accessed November 3, 2026.https://www.isca-archive.org/interspeech_2023/wei23b_interspeech.pdf
[14]
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello, “CREPE: A Convolutional Representation for Pitch Estimation,” arXiv 2018, 1-3, Accessed October 25, 2026.https://arxiv.org/abs/1802.06182
[15]
Masanori Morise, Fumiya Yokomori, and Kenji Ozawa, “WORLD: A Vocoder-Based High-Quality Speech Synthesis System for Real-Time Applications,” IEICE Transactions on Information and Systems E99-D (7), (2016), 1877-1884, Accessed November 1, 2026.https://doi.org/10.1587/transinf.2015EDP7457
[16]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[17]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[18]
RVC-Project, “Retrieval-based Voice Conversion WebUI.”
[19]
Brecht Desplanques, Jenthe Thienpondt, and Kris Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” arXiv 2020, 1, 3-4, Accessed November 3, 2026.https://arxiv.org/abs/2005.07143
What Is Preserved and What Is Changed in Voice Conversion?
TTS,Voice Cloning and Voice Conversion: Key Differences
Internal Structure of the RVC Transformation Pipeline
Training and Inference Stages
Content Representation with HuBERT and ContentVec
Why Is F0 Extracted Separately in RVC?
Difference Between .pth and .index Files
How Is Target Speech Data Prepared?
Preprocessing
How Is RVC Data Diversified with Limited Recordings?
TTS and RVC Integration
Real-Time Transformation and Streaming
How Is the RVC Output Evaluated?
Conclusion