VOXLET: Pioneering Private Voice-Changing Technology (Updated)

The world is awash with electronic devices equipped with microphones and capable of recording speech. Add in biometric recognition, emotional screening, and similar audio analysis technology, and you have a context in which it has become very difficult to speak anonymously. This not only threatens individual privacy – particularly where censorship or retaliation are risks – it also directly impacts sensitive intelligence operations where the ability to speak anonymously can mean the difference between life and death.

While many voice obfuscation solutions already exist, none work in real-time, nor are they secure in settings where guarantees of user privacy and prevention of identification attacks are essential for user safety.

“Today, you can use an AI voice transformer online to record yourself saying something and play it back sounding like a famous person,” said Galois Principal Scientist Taisa Kushner. “But that’s currently at the input/output level. It doesn’t change the sound of your voice as you’re speaking in real-time. More importantly, someone could analyze that output and not only discover that it’s fake, but also be able to backtrack out my original voice from it. … We’re trying to fill that gap – developing a solution for transforming speech in real-time, with formal privacy guarantees.” 

To meet this challenge, Galois, GE Aerospace, the University of Vermont, and North Point Defense collaborated on the Intelligence Advanced Research Projects Activity (IARPA)’s Anonymous Real-Time Speech (ARTS) program, which seeks to develop new technologies for anonymizing conversational speech to help safeguard individual speakers’ identities. 

The team’s solution, Vocal Obfuscation eXpert Language Encoding Technology (VOXLET), combines learning neural network models with state-of-the-art formal privacy guarantees provided via differential privacy techniques. VOXLET also utilizes novel compression techniques to enable execution on resource-limited hardware. 

History of Anonymization

Voice anonymization technology stems from the field of voice conversion. In the 19th century, linguists recognized that different speakers formed the same articulatory configurations with their vocal tracts to form the same units of sound for speech. Early speech processing pioneers sought to replicate this phenomena. By 1960, Gunnar Fant modeled the vocal tract as a simple model of connected tubes with different diameters – the diameters changing over time corresponded to articulation of the vocal tract. The resonant frequencies that came from this tube model could be computed, and when excited by a source signal generated by the voice box, acoustics for synthetic human speech could be generated. This source-filter model enabled the manipulation of the spectral envelope via Linear Predictive Coding (LPC) and pitch via fundamental frequency (F0), separating the articulation of the vocal tract from the excitement of the source and providing a way to both analyze and synthesize speech signals. This ability to encode articulatory states separate from the acoustic signal provided mechanisms for separating speaker identity from speech content, and allowing for many different speech processing tasks.

Statistical Voice Conversion (VC) emerged in the 1980s when researchers began mapping source spectral features directly to a target speaker’s acoustic space, using signal processing techniques to time-align the speech content. By the 2000s, Gaussian Mixture Models were leveraging high-quality vocoders to resynthesize spectral envelopes and pitch models into high-quality, voice-converted acoustics. However, these systems required hundreds, if not thousands, of utterances (hours of recordings) from both source and target speakers to create such synthetic audio.

The deep learning revolution of the 2010s replaced mixture models and time-alignment with neural encoder-decoder architectures operating in high-dimensional latent spaces. These models explicitly factorized speech into disentangled components:  linguistic content (e.g., phonemes), prosody (e.g., pitch and intonation), and speaker identity. Combined with transfer learning across large multi-speaker datasets, voice conversion became high-quality even with minimal sample audio.

Modern voice anonymization extends these same factorization techniques. By retaining the underlying linguistic content while modifying prosody and identity, VOXLET ensures not only that the original speaker remains imperceptible, but also that it’s impossible to recover their voice.

Real-Time Results

One of the more significant advances being developed with VOXLET is phoneme-level voice transformation.

Like many voice-transformation technologies, VOXLET uses an encoder-decoder neural network architecture. In this framework, input speech is first encoded into a latent representation of relevant features: linguistic content, prosody, and speaker identity. Next, the encoded prosody and speaker identity representations are modified while preserving the input linguistic content – the actual words being said. Finally, the decoder takes the modified latent features and generates new speech, retaining the linguistic content of the input speech, but with a different speaker identity. The core problem with current real-time voice transformation technologies using this framework is that they can only reliably transform complete words, rather than the individual sounds that make up words.

“That’s a massive blocker for real-time systems,” said Kushner. “It’s just not how people actually talk. If I’m talking too slow or two fast, or if I’m in a conversation and I start saying one word, but stop myself halfway through when I decide I want to say a different word, it won’t work.”

Instead, VOXLET has the ability to work with phonemes – separating out, analyzing, and modifying the smallest possible phonetic unit rather than full words. This is a much finer grain level of detail for voice transformation.

For example, the spoken phrase: “Hi, um, trying to” would be processed as three transformable words by current deepfake technology. By contrast, VOXLET’s Phoneme Pipeline extracts 16 phonemes, including disfluencies (like “um”) and silences.

The result is a technology that optimizes that latent space in the middle of speech transformation, working at a much finer grain of detail, and yielding much more believable results in real-time and opening the door for high quality speech transformation capabilities across multiple languages. [1]

Novel Privacy Guarantees

The next challenge is how to make VOXLET secure – such that the output not only sounds natural, but can’t be identified as a fake nor reverse engineered to reproduce the speaker’s actual voice. For this, Galois is using a cryptography technique called Differential Privacy. Differential privacy, most often used to secure sensitive statistical data, introduces noise to protect individuals’ private information, while ensuring that aggregate statistical results remain significant and useful. With VOXLET, Galois is pioneering a new application for the technique, adding randomization noise to speech data to make individual speaker identification impossible.

The challenge here, and the focus of the team’s current research, is figuring out exactly how much noise to add and where to add it.

“We want to be able to divide the latent space into static and dynamic features,” Kushner explained. “Static features may be markers of speech, like average frequency, that are constant for an individual speaker; whereas dynamic features might be lexical content, the actual words and phrases spoken, or even background noise.”

By identifying and dividing key features in the latent space, VOXLET can introduce targeted noise to static features–such as markers of identity extracted from the speech decoder–to disguise the original speaker’s identity. This is a fine balancing act between safety and believability: Add too much noise, or add noise to the wrong feature in the latent space, and the speaker’s identity will be disguised, but it will be “too noisy” and obvious that their voice has been changed. Add too little noise and the output audio may still be able to be reverse engineered to uncover the speaker’s true identity.

“We’re figuring out a big optimization problem,” said Kushner. “What’s the best input? What’s the best latent space? How do we add noise to achieve our goals? That raw signal can be sorted into a bunch of moving parts, and we’re trying to work with those parts to get a final output that is synthesized speech that both sounds good and is not invertible.”

The goal is surgically precise noise insertion – just the right amount, in just the right place.

System Progress and Modularity

The VOXLET team recently completed and demonstrated our Phase 1 prototype system. A core strength of the platform is its modular, plug-and-play architecture. Users can plug in any off-the-shelf or custom voice encoder-decoder and VOXLET will add the differential privacy guarantees. This modularity allows the technology to evolve over time and breaks users free from vendor lock.

Different encoders are useful (or optimized) for different use cases. Want an encoder that operates at the phoneme level instead of the word level? No problem. Want to switch between an off-the-shelf encoder-decoder and one that was trained by you? Yup, that’s doable. Want an encoder optimized to work in a particular language? VOXLET now has the ability to work in English, Russian, Hindi, Bengali, and Greek, and is crafted to be able to plug in additional languages as needed. 

It’s worth noting that, while the differential privacy component adds zero latency, the encoder/decoder introduces a slight delay. In testing, we found that VOXLET worked optimally at a latency of one second, (i.e. the gap between the original user speaking and their output coming out in a natural-sounding different voice is one second). This is almost real-time, and moreover, for identity-masking applications where the original speaker is unseen (i.e., video off), the listener will not realize the latency.

Finally, the VOXLET system has been engineered to be easily usable. It comes with a UI where different options can be customized and the user can speak into a microphone and have their changed voice come out through the speakers. 

VOXLET represents a fundamental shift in how we approach secure communication. By seamlessly blending high-fidelity voice transformation with robust, mathematically-verified privacy, it offers a necessary safeguard in an era of pervasive surveillance. As the project matures, its potential to protect identities—from journalists and activists to intelligence personnel—positions VOXLET as a vital tool in the ongoing effort to defend individual privacy in a hyper-connected world.

VOXLET in Action

To see our voice-changing technology for yourself, check out the live demonstrations below:

• DEMO 1, from Galois Research Engineer Hazel Shackleton: https://youtu.be/17mnnzwZBuE

• DEMO 2, from Galois Research Engineer Sourya Dey, including a look at the current UI: https://youtu.be/Ga1iPcWWmyE

‍

_{[1] This also opens up enormous possibilities for multi-lingual voice transformation, use of proper names, etc.}