semantiC segmentatiON oF compLexe soUnd scEnes oN edge deviCEs – CONFLUENCE

The CONFLUENCE project aims to develop artificial intelligence (AI) technologies for sound semantic segmentation of acoustic signals that can recognize sound events and separate/isolate the signals of the sound sources forming semantic entities. This can be seen as an audio extension of the image semantic segmentation task that has been receiving much attention in the AI field.
The objective is to design an embedded AI system that can implement these technologies for two telecom application services: immersive communication and home monitoring/assistance devices. Specifically, by analyzing, separating, and augmenting the separated signals with metadata such as location information, it will be possible to transmit only the necessary sounds from a sound scene and spatially recreate that scene in a remote location, or understand what is happening remotely. This could allow advanced communication services, such as privacy-conscious web conferencing systems and home monitoring/assistance devices/systems, that filter out the sounds of daily life and allow only transmit adult speech and important sounds, or new inclusive communication systems that would allow local and remote participants in international conferences to feel in the same location. In this project, we will mainly address the following issues: (1) the development of sound source separation and target sound extraction technologies as a basis for semantic separation, (2) open source implementation of the developed technology and hardware implementation in embedded devices, (3) dissemination of the technology through a proposal to the International Standardization of Mobile Communication Systems (3GPP) and deployment to privacy-preserving home monitoring devices, (4) Contribution to the acceleration of the academic field by disclosing the data acquired in this project and holding international technology evaluation challenges. Through the combination of scientific and industrial expertise from Japan and France, the CONFLUENCE project will develop innovative sound segmentation solutions for embedded devices.