Multimodal language practices in French family dinners – DINLANG

This project is based on naturally occurring interactions produced during dinner conversations. The centrality of multimodality in language grounded in situated activities distinguishes this study’s approach. The core of the project consists of multimodal qualitative and quantitative analyses of language processes captured in detail through their formal components as they are deployed moment to moment in situated events. We use video recordings linked to transcripts of interactions surrounding dinnertime collected in ecological environments.
Our recording set-up is designed to collect as much information as possible without being a hindrance to the dinner participants. We have three recording points where a camera and a sound recorder are placed. A 360° camera is placed in the middle of the table. A 360° sound recorder is also positioned near it. Additional recordings are done with two classic wide-angle cameras located at two different sides of the table. Each camera provides a frontal/side view of half of the participants and a rear-view of the other half. The cameras also record sound, and we have added better quality microphones. All the video and audio files are synchronized with a classical video clap.
The data is annotated using ELAN, a multimodal annotation software. Data can be transferred from ELAN to other programs such as CLAN for queries, PHON for phonological analyses, Praat for prosodic analyses and Excel for quantitative analyses.
The consortium has collectively devised conventions for annotation and coding of gesture, signs and actions, reflecting our combined theoretical backgrounds. We assume that the combination of components in gestures and sign have physiological and interactional motivations. We follow Boutet’s (2008, 2010) intrinsic physiological approach to gesture, which accounts for gesture units based on their formal characteristics and constraints.
There are four to five participants in our recordings. Two parents, typically but not necessarily the mother and the father, and two or three children. Each participant is associated to a unique tag. Each participant has the same set of dimensions coded.
In ELAN, The Multiple Layer Search can find any temporal relation, even when a long time has elapsed between the occurrences in each tier. It can also find sequences of annotations for the same participant, or any combination of sequences and overlaps. Our queries are very useful to structure the data available in our corpus. We can thus obtain results on all the possible research questions we might have concerning the orchestration of actions, gaze, signs, speech and gestures in the varying participation frameworks that occur in spontaneous ecological conversations.

Our results are in progress. Studying two different communicative modalities sheds a strong light on socialization through ‘talk’ (which we call “languaging” to express the dynamic process involved in speech, gesture and sign) of French children, as well as their developing French identity and French heritage. Studying deaf families where communication may be different than in hearing families supports the argument that the French family meal is a common site of socialization to Frenchness for both deaf and hearing children. By taking the same analytical approach for both spoken French and LSF, we demonstrate the dominance of this mode of socialization to French identity through specialized practices. Those specific practices contribute to French children’s learning attitudes and behaviors as they become members of the community of French diners, be they speakers or signers. At the same time we are studying how children develop and embody the two official languages of France with all their multimodal components in a multiparty, multiactivity framework.
The multimodal nature of language has led us to develop a method to investigate how various semiotic systems such as speech, sign, gesture, posture, facial expressions and gaze but also actions and object manipulations, are simultaneously deployed, transmitted and used in the situated activities combined in family dinners. During multiparty, multimodal situated interactions in coordination with other body activities, every move, every part of the body, every object is potentially meaningful. They are deployed in a multitude of skillful variations in the collective coordination of bodies, activities and artifacts.
We are getting results that outline both the similarities and the differences between language and dinner practices in French speaking and LSF signing children and adults.

The world-wide period of confinement we have been experiencing has highlighted the relevance of studying the transmission of language practices as our most valuable cultural heritage, at home, in family dinners, a time when we share food, emotions, dreams, stories, and enjoy each other’s indispensable physical presence.
The ambition of our complementary analyses is to implement a new theoretical model in which we articulate the embodied forms of expression and the symbolic functions of language to demonstrate how languaging is relative to its forms of expression and the affordances of its context.
This approach is supported by the choice of a shared cultural event, French family dinners, for analysis of common features and variations between speech, sign, co-verbal and co-signed gesture, (including gaze and facial expressions) in situated activities and in interaction. This study is grounded in the exploration of the actional roots of language, which are shaped and expressed differently according to the ongoing activities, channels of communication and age of children. This research allows the development of conceptual tools to capture children’s progressive mastery of the most remarkable feature of human languaging: how to use our bodies (eyes, face, mouth, hands, arms…) to express not only what is perceivable in the here and now (proximal) but what is absent or highly subjective (distal): remembrances of things past, projects, dreams, affective and evaluative stances, abstractions and figments of our imagination.
We hope to deliver a pioneering multimodal archive of signed and spoken languaging. We will also share methodological tools in the form of a multimodal platform in which the interoperability between the different types of software to analyze multimodal productions will have been fully tested. The platform will integrate the pairing of manual and semi-automatic annotations conducted on speech, sign and gesture. The results will integrate detailed kinesiological analyses of gesture, thick analyses of interactive sequences and quantitative analyses on a large number of annotated forms paired in context with their function.
The results of our scientific analyses will be useful for several scientific communities and for professionals in different fields as well as for parents: 1) language acquisition specialists, gesture and sign language specialists who seldom analyze multi-party interaction; 2) computer scientists working on spontaneous conversational skills to inspire machine learning and modeling of multimodal interaction; 3) education professionals who could transfer our results on parental scaffolding in speech and sign to institutional settings; 4) parents and parent associations advocating for the recognition of sign language as an effective communication media; 5) medical teams and medical anthropologists who could benefit from our data to study eating behaviors and food practices.

1. Morgenstern, A. Symposium at l’ISGS (International Society for Gesture Studies). Action, gesture, speech and sign in family dinners. 13 July 2022, Chicago.
2. Caet. S.; Le Menet, M. ; Parisse, C. ; Danet, C. ; Kosmala, L. ; Debras, C. ; Morgenstern, A. Inter-languaging and dining in signing and speaking families. ISGS 2022. 13 July 2022, Chicago.
3. Martel, K.; Blondel. M.; Dodane, C. ; Catteau, C. ; Vincent, C. Multimodal focalization processes during French family dinners : A comparison between speaking and signing families. ISGS 2022. 13 July 2022. Chicago.
4. Beaupoil-Hourdel, P.; de Pontonx, S. ; Nguyen, A. ; Jaquemet, S. ; Morgenstern, A. Materializing school in middle-class family dinners: Objects, speech, sign and gesture. ISGS 2022. 13 July 2022. Chicago.
5. Brunet, A.; Morgenstern, A. ; Atangana, F.; Ali-Akinci, M. Language dominance and gesture in bilingual family dinners. ISGS 2022. 13 July 2022. Chicago.
6. Morgenstern, A.; Beaupoil-Hourdel, P.; From Plurisemiotic productions to the primacy of speech in child language development: the role of adult scaffolding. ISGS 2022. 14 July 2022. Chicago.
7. Morgenstern, A. Beaupoil-Hourdel, P.; Dodane, C. The coordination of gestural, prosodic and verbal patterns: A developmental perspective on multimodal negation in French speaking children. ISGS 2022. 14 July
2022. Chicago.
8. Morgenstern, A. Presentation of the Dinlang project during a public event at Sorbonne Nouvelle, 18 May 2022.
9. Workshop of thetDinlang project, July 1st 2022, Maison de la Recherche, Sorbonne Nouvelle (with FR-LSF interpreters). Guest discussants : Debbie Chen Pichler, Tami Kremer-Sadlik, Christelle Pepin.
10. Morgenstern, A. Dynamique du dialogue adulte-enfant: Variations en solo, en duo ou à plusieurs. Colloque DIAPASO. 3 June 2022 Sorbonne Nouvelle.
11. Brunet, A & Morgenstern A. Lessons from multimodal and multilingual languaging in family meals. 18 March 2022. Presentation at the SeSyLiA seminar. Sorbonne Nouvelle.
12. Parisse, C.; Blondel, M.; Caët, S. ; Danet, C. ; Vincent, C. ; Morgenstern, A. Multidimensional Coding of Multimodal Languaging in Multi-Party Settings. LREC 2022, June 2022, Marseille, France.
13. Internet website Dinlang. dinlang.ortolang.fr
14. Promotion clip to recruit LSF signing families ct3.ortolang.fr/pages/info-lsf-dinlang/dinlang-lsf.mp4
15. Blondel, M & Danet, C. : Short presentaiton of the project for professional BA students in LSF teaching (LSF) in instutional settings.

The DINLANG project aims to capture how semiotic resources (speech/sign, gestures, actions) are coordinated to construct meaning through the language practices transmitted to and used by children during French family dinners. Ethnographic methods will be used by four teams to collect dinners in two age groups with hearing and deaf participants using two official languages of France: French and French sign language (LSF). Our quantitative and qualitative analyses will lead to innovative results on the function and forms of situated interactive practices as they shape and propel the dynamics of family life. It will target social, discursive and semiotic variations in a valued French cultural habitat with its sensory experiences and its situated activities. The data collection, the multimodal platform, the semi-automatic annotation system developed in the project and our scientific results, will contribute to enrich the methods and theoretical models employed to understand the transmission of multimodal resources to speaking and signing children.
The proposed project is closely fitted to the mission of CES 27 on culture, creations and heritage through its focus on family dinners as French cultural heritage. Through family dinner conversations spiced with food and the company of loved ones, we aim to capture the transmission of one of our most precious cultural treasures – language.
Family dinners grounded in commensality and conversation are a major collective ritual that plays a key role in French people’s identity and constitutes an inherent part of French cultural heritage. Those shared moments of everyday life present a perfect opportunity to study how language and interactive practices are transmitted to and used by children in order for them to construct meaning. Because the subtle interweaving of these practices while eating fully engages the body, this project will highlight the semiotic differences between children using a spoken language, French, and a sign language, LSF, at different ages.
Our theoretical framework will incorporate 1) the models of linguistic anthropology focused on language socialization; 2) multimodal approaches to language; 3) a formal, kinesiological approach to gesture and sign; and 4) a linguistic model combining usage-based and constructional linguistics and an interactive approach to pragmatics. Our expected results involve the identification, characterization and comparison of language practices in signing and speaking families of France, which will clearly highlight French identity and its interactive transmission during French meals, as cultural heritage.
The ambition of our complementary analyses is to implement a new theoretical model in which we articulate the embodied forms of expression and the symbolic functions of language to demonstrate how languaging is relative to its forms of expression and the affordances of its context. This approach will be supported by the choice of a shared cultural event, French family dinners, for analysis of common features and variations between speech, sign, co-verbal and co-signed gesture, (including gaze and facial expressions) in situated activities and in interaction. This study will be grounded in the exploration of the actional roots of language which are shaped and expressed differently according to the ongoing activities, channels of communication and age of children. Its linguistic focus will be the use of personal and temporal refence in proximal and distal discourse. This research will allow the development of conceptual tools to capture children’s progressive mastery of the most remarkable feature of human languaging: how to use our bodies (mouths, eyes, face, mouth, hands, arms…) to express not only what is perceivable in the here and now (proximal) but what is absent or highly subjective (distal): remembrances of things past, projects, dreams, affective and evaluative stances, abstractions and figments of our imagination.