Microsoft's VALL-E 2 AI Speech Generator Faces Misuse Risks

Microsoft's new artificial intelligence (AI) speech generator, VALL-E 2, has demonstrated an impressive capability to recreate human voices using just a few seconds of audio. However, due to the potential risks associated with its release, the technology will not be made available to the public.

Capabilities of VALL-E 2

VALL-E 2 is a text-to-speech (TTS) generator that can convincingly reproduce a human speaker’s voice from minimal audio input. According to a paper published on June 17 on the pre-print server arXiv, Microsoft researchers claim that VALL-E 2 can generate speech that is "accurate, natural, and comparable to human performance."

Key Features of VALL-E 2

The AI engine's advanced capabilities are attributed to two key features: "Repetition Aware Sampling" and "Grouped Code Modeling."

• Repetition Aware Sampling improves the AI’s ability to convert text into speech by managing repetitions of language tokens, preventing infinite loops of sounds or phrases, and making the speech sound more natural.

• Grouped Code Modeling enhances efficiency by reducing the sequence length of tokens processed, speeding up speech generation and handling long strings of sounds more effectively.

Evaluation and Performance

Microsoft researchers evaluated VALL-E 2 using audio samples from LibriSpeech and VCTK speech libraries. They also employed ELLA-V, an evaluation framework designed to measure speech accuracy and quality. The results showed that VALL-E 2 surpassed previous zero-shot TTS systems in robustness, naturalness, and speaker similarity, achieving human parity in these benchmarks.

Concerns and Restrictions

Despite its advanced capabilities, Microsoft has decided not to release VALL-E 2 due to potential misuse risks, such as voice spoofing or impersonation. This decision aligns with increasing concerns around voice cloning and deepfake technology, with other AI companies like OpenAI also placing restrictions on their voice technologies.

Future Potential

Although VALL-E 2 will remain a research project without plans for public release, Microsoft researchers acknowledge its potential practical applications. These include educational learning, entertainment, journalistic content, accessibility features, interactive voice response systems, translation, and chatbots. The researchers emphasize the need for protocols to ensure speaker approval and detection of synthesized speech if the model is generalized for real-world use.