Multimedia archives enhancement: multimodal machine comprehension of language for new intelligent interfaces of scientific and cultural mediation – ARCHIVAL

The UNESCO ITEN Chair has designed with its partners an interdisciplinary framework on Digital Humanities for this project, structured in 5 Work Packages (WP). WP1 will be in charge of the direction, coordination, scientific follow-up and organization of the seminars, it will include the preparation of the corpus files on the theme of «self-management«, the export of metadata, the definition of use cases, the description of the user paths and the Data Management Plan (DMP). WP2 will be dedicated to the production of a multimodal semantic representation space that will be exploited in WP3 (AMU) to synthesize the results of the interrogation of these documents carried out thanks to the innovative interfaces defined in WP4 by exploiting the complementarity of the HMI experts of FMSH-ITEN and Orange Labs. WP5 is dedicated to evaluation, first of all intrinsic evaluations of the models defined in WP2 and WP3, then the FMSH, the UNESCO ITEN Chair, and Orange Labs will participate, according to agile methods, in the evaluation of the appropriation of the tools by users within the Digital Center for Social Innovation (UP8-ENEDIS) and the FMSH library-laboratory. The diagram below represents the architecture of the ARCHIVAL project.

Translated with www.DeepL.com/Translator (free version)

WP1
- constitution of the corpus, realized from the mixed fund of self-management of the FMSH.
- extraction of a network of documents from Wikipedia Autogestion in the hyperlink network.
- SHS editorial work on semantic interpretation and automatic analysis for ontology.
- ARCHANNOT annotation platform
- collected corpora processed by linguistic tool suites to add automatically extracted metadata

WP2
- Linking baselines have been explored:
- Intra-collection approach by content comparison: thematic segmentation of content and segment matching by tf-idf, LDA or Doc2vec.
- Learning-based link prediction approach: exploitation of attention mechanisms to extract important words and link explanations.
- Indirect extra-collection approach: prediction of links within documents using an encyclopedic corpus, Wikipedia, as a knowledge base.
- Ontology realized to link documents associated to the same thesaurus entries. Experiments on automatic enrichment of expert annotations in progress (Topic Modeling or Hierarchical Topic Modeling approaches)

WP3
- The first works focused on document understanding through the adaptation of Reading Comprehension methods.
- Baseline for document querying is available in the form of an API that can be used in the prototypes.

WP4
- Interfaces developed to visualize digitized documents and better understand the structure and representation formats of the data.
- Prototypes for visualizing links produced in WP2 and for querying documents developed in WP3 and the very first interfaces for querying archived content with a representation of the response integrated with the visualization of documents.

The first period of the project allowed the implementation of a real multidisciplinary dynamic within the consortium, with regular exchanges between ICST and SHS researchers. After an initial phase of acculturation and definition of respective expectations (in terms of content, annotations, uses, evaluations, etc.), the regular meetings of the consortium are an opportunity for a rich dialogue and an agile approach combining development of a proof of concept and validation. These exchanges have also led to the creation of rich corpora for the project.

The first interfaces developed by Orange, as well as the design of several use cases for navigation interfaces developed by the FMSH at the beginning of the project, were the first significant results for all of the players. The Orange interface provides a solid base for future developments in the second part of the project, after validation of the technological choices via the intermediate interfaces.


Work for the next periods
- Integrate documents from other collections, Gallica, INA, Persée, agreements are being signed.
- Exploit data annotated by the FMSH to evaluate and refine methods for predicting and explaining links between documents.
- Extend the work to multimodal representation spaces, particularly audiovisual content (use of films)
- Design interfaces for mediation and promotion of archives

Robin Brochier, Frédéric Béchet. Predicting Links on Wikipedia with Anchor Text Information. ACM SIGIR Conference on Research and Development in Information Retrieval, Jul 2021
hal.archives-ouvertes.fr/hal-03233373

Invited lecture at the Humanistica 2021 conference. Guillaume Gravier. Artificial intelligence to describe, index and explore multimedia collections: how to do without Google? humanistica2021.sciencesconf.org/356765

Five Seminars Workshops Annotations Master 1 « Digital Humanities « University of Paris8 (FMSH / UNESCO Chair ITEN / LIS-AMU)

Digital technology transforms the way people access knowledge, which remain difficult to reach despite numerous attempts to index and structure texts, videos and images according to structured repositories. Often presented as functional lists, search results interfaces offer few navigation and exploration capacities and hardly articulate contents to each other.
The multidisciplinary and multi-actor ARCHIVAL project aims at yielding collaborations between researchers from the fields of Information and Communication Sciences as well as Computer Sciences around archive value enhancing and knowledge sharing for arts, culture and heritage. The project is structured around the following questionings: What part can machine comprehension methods play towards the reinterpretation of thematic archive collections? How can content mediation interfaces exploit results generated by current Artificial Intelligence approaches?
« Automatic Language Understanding » methods can be used to discover a semantic structure in multimodal documents by generating a “semantic space” in which links between documents and document segments can be defined. “End-to-end” neural methods allow such representations to be directly learnt with respect to a task without necessitating explicit specification of intermediate levels of analysis to achieve this task. The resulting semantic vector spaces provide relations between words and documents and allow the corresponding representations to be used for generating outputs that reflect the meaning, the “latent science”, contained in documents, going beyond traditional term extraction (keywords, entities, RDF triples). In parallel, documents can be queried with natural language questions thanks to Machine Reading Comprehension methods, highlighting relevant excerpts with respect to the question. A document is therefore considered through the prism of the relations it holds with other documents, and through the prism of questions related to its content, but also through syntheses (summaries, reformulations, cartographies) that can be dynamically generated from its analysis.
One of the major scientific objectives of the ARCHIVAL project is to perform comprehension in a multimodal approach, by exploiting documents mixing speech, text, fixed images and videos. Neural methods have paved the way to inter-modality thanks to the definition of vector space representations that are common to different modalities. Exploiting this convergence, which has already yielded spectacular results, offers a lot of possibilities for digital mediation.
ARCHIVAL teams will explore heterogeneous document collection structuration in order to explicitly reveal implicit links, to explain the nature of these links and to promote them in an intelligible way towards ergonomic mediation interfaces that will guarantee a successful appropriation of contents. A corpus has been delimited from the FMSH “self-management” collection, recently awarded as Collex, which will be completed from the large Canal-U academic audiovisual portal. The analysis and enhancement of this collection is of particular interest for Humanities and Social Sciences in a context where it becomes a necessity to structurally reconsider new models of socioeconomic development (democratic autonomy, social and solidarity-based economy, alternative development,…).
In terms of impact, ARCHIVAL will be committed to developing a generic technological core that can be eventually replicated into other fields of study, disseminated by all partners, and whose outreach can be promoted in the Condorcet Campus.