Towards a computational multimodal analysis of film discursive aesthetics – TRACTIVE

TRACTIVE aims to establish a novel framework for the analysis of gender representation in visual
media. We integrate AI, linguistics, and media studies in an iterative approach that both pinpoints the multimodal discourse patterns of gender in film, and quantitatively reveals their prevalence. We deploy a multidisciplinary approach to data creation. We design AI models to analyze the multimodal representation of characters. We devise a new interpretative framework for media and gender studies incorporating modern AI capabilities. Our models, published through an online tool, will engage the general public through participative science to raise awareness towards gender-in-media issues from a multi-disciplinary perspective.

These first 18 months enabled us to set up data creation processes (corpus delimitation, investigation of existing datasets, annotation methods, processing workflow), to initiate data creation and data analysis, in particular with deep learning models.

We are preparing conference submissions to share our innovative data (expert analyses of gender representation in videos) and our first baseline deep learning models, and enable the AI and CIS communities to address these new questions.

1. H.-Y. Wu, L. Nguyen, Y. Tabei, and L. Sassatelli., 2022.04, “Evaluation of deep pose detectors for automatic analysis of film style”. 10th Eurographics Workshop on Intelligent Cinematography and Editing. Eurographics Digital Library. Reims, France.
2. Vanni L., Mahmoudi H., Longrée D., Mayaffre D. (à paraitre). Multi-channel Convolutional Transformer and intertextuality : a Latin case study. In Text Analytics. Advances and Challenges, Springer.
3. Guaresi M., Haris S., et Vanni L. 2023. Text Analysis Using Convolutional Neural Networks with Multi-Head Attention. 12th International Quantitative Linguistics Conference QUALICO 2023. Lausanne, Switzerland.
4. Vanni L., Guaresi M., Magri V. 2022. Convolution et marqueurs multidimensionnels. Description des représentations genrées dans un corpus de films français. 16th International Conference on Statistical Analysis of Textual Data ( JADTS 2022 ), Jul 2022, Naples, Italie.
5. Mayaffre D. et Vanni L. (2023 - sous presse), “Sémantique de corpus numérique et deep learning, Espaces linguistiques, n°6, 2023
6. L. Andolfi, 2023.06.15, “Communautés de fans en ligne et folklore numérique : Pour une anthropologie des publics”. XXIIIe congrès de la Sfsic : La numérisation des sociétés.
7. L. Andolfi, 2023.04.17, “A la recherche du genre perdu : la construction nostalgique de représentations du genre par le dispositif des films d'époque”. Ve atelier doctoral Philomel : La représentation du genre à l’écran.

Surveys on the quantitative representation of women in visual media are insufficient to grasp the issue of gender inequality, the visual modality being key to do so. In film studies, two relevant visual discursive regimes have been identified and recently revisited: the male gaze and the female gaze. Yet, how can we pinpoint such complex, subtle, but wide-spread visual discourse patterns, that may convey biased gender representation, and how to quantify the extent of their respective usage?

With the advances in Artificial Intelligence (AI) and computational linguistics, we are now in place to conduct quantitative analysis to identify and extract recurrent visual and textual patterns from media content. Yet, such an analysis requires an iterative approach in concert with qualitative media studies, to recognize what is characteristic of a discourse style in visual media, and how the computational findings fit into the wider narrative and socio-historical context for which the content was produced. To enable such a bi-directional dialog, the AI models will have to provide sufficient explainability and allow expert-in-the-loop analysis and refinement.

TRACTIVE’s objective is to characterize and quantify gender representation and women objectification in films and visual media, by designing an AI-driven multimodal (visual and textual) discourse analysis.

TRACTIVE aims to establish a novel framework for the analysis of gender representation in visual media. We integrate AI, linguistics, and media studies in an iterative approach that both pinpoints the multimodal discourse patterns of gender in film, and quantitatively reveals their prevalence. We devise a new interpretative framework for media and gender studies incorporating modern AI capabilities. Our models, published through an online tool, will engage the general public through participative science to raise awareness towards gender-in-media issues from a multi-disciplinary perspective.