← all projects

SP-004 ~/the-official-record Applied ML Built

Audio Emotion Style Transfer

Change the emotion of a voice recording (angry to sad, and back) without changing the words: speech becomes a mel-spectrogram image, a CycleGAN translates it between emotion domains, and Griffin-Lim turns it back into audio.

Role
3-person team, Machine Intelligence course (PES University, V semester)
Dates
2022

The brief

Speech carries emotion in its prosody, not its words. This project reframes “change the emotion of a voice” as an image problem: convert each clip to a mel-spectrogram, treat angry and sad recordings as two unpaired image domains, and train a CycleGAN to translate between them. The converted spectrogram is then inverted back into a playable waveform.

The reframing matters because there is no paired data. Nobody records the same sentence angry and sad on demand at dataset scale. CycleGAN’s cycle-consistency loss learns the mapping from unpaired domains, which is the entire reason this approach works.

Pipeline

  1. Audio to image. librosa loads each clip and computes a 128-band mel-spectrogram, saved as single-channel float TIFFs. Roughly 1,345 clips per emotion domain.
  2. CycleGAN. Pix2Pix U-Net generators and discriminators, surgically modified to single-channel input for spectrograms, with instance normalization. Two generators (angry to sad, sad to angry), adversarial plus cycle-consistency plus identity losses, trained about 100 epochs on Colab with checkpoints to Drive.
  3. Image back to audio. Griffin-Lim inversion (librosa.feature.inverse.mel_to_audio) reconstructs a waveform from the generated spectrogram.
Two mel-spectrograms from the training data: the angry clip shows dense curved harmonic stacks, the sad clip is flatter with one bright band
Real training data. Left: an angry-domain clip, dense curved harmonics. Right: a sad-domain clip, flatter energy. The model learns to translate between these textures.

The honest bit

The synopsis proposed TraVeLGAN plus a speech-emotion-recognition network as a quantitative validator; what shipped is the CycleGAN pipeline, and the evaluation is qualitative. Three people built this for a fifth-semester course. It is a working end-to-end prototype and an honest generative-ML story, not a benchmark result.

Credit
Built with Shreyash Chatterjee and Shrujan for the V-semester Machine Intelligence course.