PRojection of the Oral language to PICTOgraphic units – PROPICTO

PROPICTO aims to develop a research axis focusing on alternative and augmentative communication, with a special focus on the automatic transcription of French speech into pictographic representation. PROPICTO addresses many societal needs in the field of disability (communicating with people with cognitive problems) and medicine (communicating with patients who do not speak the same language as the medical). It also satisfies the legal requirements adopted in Switzerland (Federal Act on the Elimination of Inequalities Affecting Persons with Disabilities of 2002, as well as the UN Convention on the Rights of Persons with Disabilities, ratified by Switzerland in 2014) and in France (Act of 2 January 2002, reinforced by the Act of 11 February 2005). PROPICTO faces many research challenges related to the automatic processing of natural language. The aim of the project is to propose methods and corpora that will make it possible to transcribe speech directly into a series of pictograms, either free (ARASAAC) or specially created for specific needs (medical, family, etc.). We will start from a specialized field (medical emergencies) to extend it to spontaneous fields used in institutes or in the family. The project will have to deal with two major problems: 1) the small amount of data which is a constraint to the implementation of state of the art techniques based on machine learning and 2) the need to evaluate our methods with very diversified target populations. It will adopt a modular approach that will deal independently with the four stages of the project, namely :

-automatic recognition of spontaneous speech,
-Syntax analysis of the spoken word,
-simplification of oral language to a standard: Easy to Read and Understand (FALC) and
-translation of the FALC into pictograms.

Each step will be supported by hybrid approaches in which language rules will be used as a starting point for robust systems to overcome the initial lack of data. This linguistic expertise will include, for example, expert grammars, synthetic corpus generation and expert syntactic modeling. This modular approach will also facilitate the evaluation of the different steps according to the target population, so that needs may differ at different levels: specialized/simple/spontaneous language, complex/simplified syntax and specialized/generic/advanced/simplified pictograms. PROPICTO will supply to the scientific community all the resources created: audio corpus associated to its translation into pictograms (with different ecological situations), database linking pictograms and their semantic meaning, Automatic Speech Recognition (ASR) systems of the project, simplification system for the FALC, Automatic Speech/Pictogram Translation (MT) system and metrics for human or automatic evaluation of the speech/pictogram translation. At the end of the project, three prototypes for different target audiences will be put into production:

1) for emergencies at the "Hôpitaux Universitaires de Genève" (HUG, Switzerland),

2) in an institution within the Coste-Rousse Institution for Children and Adults with Multiple Disabilities (EEAP, France) and 3 French medico-educational institutes (IME) in Valence, Orange, Meylan, and

3) in a family/daily setting with volunteers from the French Association of Rett Syndrome (AFSR). They will be tested in real conditions and evaluated with human and automatic methods.