Descriptor-aware latent granular mosaicing for real-time audio re-synthesis

Author

Pichard, Alfred and Lattner, Stefan

Abstract

We introduce descriptor-aware latent granular mosaicing, a method for real-time audio reconstruction based on grain selection in the latent space of a neural audio autoencoder (Pasini et al. (2024)). Prior work shows that granular synthesis in neural codec latent spaces enables corpus-based transformation by matching grains between source and target signals (Tokui and Baker (2025)), but relies solely on latent similarity and offers limited additional perceptual control. We address this by augmenting latent grain retrieval with learned perceptual descriptors, including loudness, transientness, brightness, and spectral flatness. Grain selection is formulated as a multi-factor distance combining latent proximity and descriptor similarity, enabling continuous control while preserving source structure. The method is implemented as Latent Mosaicer, a Max for Live device for interactive DAW use. Evaluation on Slakh-based codebooks and sources (Manilow et al. (2019)) measures descriptor tracking andindicates that explicit descriptor goals can steer retrieval in the rendered output.