Sensemaking and Spatialisation for multi-granular heritage artefacts : 3D annotations, sonification. and formalisation of inferences – SESAMES

The project implements an original approach covering steps encompassing the recording and reuse of acoustic and 3D data, data visualization (extraction and comparison of information patterns) and sonification (representations based on the perception of rhythm and sound sequences) issues, and the formalization of inferences made during research processes, exemplified on the specific case of 3D reconstructions. The main research lines are as follows:

- Multidimensional characterisation of heritage assets: the development of a combined space + sound data acquisition protocol that is intended at bringing together two essential dimensions of the definition and perception of an architectural edifice’s interior: a morphology, an acoustics. The project is also used as an opportunity to reflect on the capacity of ontologies to fully express the particularities of the application field (how can we characterize distances between theoretical objects and objects as we can observe them today?). Finally, the project questions the intersection of often parallel practices: that, at the prescriptive level; of the architect and that, at the descriptive level, of the archaeologist.

- Comparative reading: the project explores (i) sound data visualization solutions (2D / 3D) (ii) architectural data sonification solutions (sound mapping of morphological characteristics) and (iii) a combination of the two approaches in order to better assess their respective contributions, pluses and minuses.

- Formalising and memorising ways of doing: The project’s strategy is to open (and hopefully clarify, then extend to other workflows) the debate on a recurring issue in heritage science: the imperative of formalising and memorising the various steps that lead from “premises” (sources, analogies, when not an expert’s intuition) to an assertive 3D virtual reconstruction – an end line product that remains today most often un-associated with interpretation processes.

Multidimensional characterisation of heritage assets: an innovative protocol for acquiring in a combined way visual / metric / sound data, and a protocol adjusted to constraints found when studying minor heritage, isolated buildings, has been developed and applied to a corpus of 15 edifices. As a result, consistent data sets have been produced that then feed a variety of post-processing chains and of specific developments (in the sense of tools and applications or in the sense of interpretative models). Besides, a first phase of testing made it possible to question the added value of 3D point cloud annotation and resulted in new scenarios for the exploitation of 3D data at the intersection of prescriptive and descriptive dimensions.

Comparative reading: a series of exploratory experiments taking benefit of the data acquired on a corpus of 15 rural chapels has been carried out _ visualization and spatialization (2D / 3D) of acoustic indicators, sonification of architectural features (contours), joint exploitation of sound data and visuals (online panoramas).

Formalising and memorising ways of doing: an online IS (Information System) is being developed that allows the formalization and recording of the protocols mobilized in various steps that lead from premises to 3D virtual reconstructions. The IS was tested on the case study provided for in the project (3D renderings, Marmoutier abbey). Also worth mentioning: knowledge elicitation between teams to refine the activity description model, creation of a justifying matrix and construction of a conceptual model for recording excavation data.

Finally, during the implementation of the project, a complementary objective emerged: to include a historical dimension (diachronic analysis based on the characterization of states and transitions) in the characterization effort, a dimension for which a theoretical model was developed and a web application is being implemented.

The project’s ambition is to act as an eye-opener on characterisation, sensemaking and reproducibility challenges in heritage sciences, in a real-case approach that although naturally non exhaustive encompasses scales and scientific disciplines .

At this stage, the SESAMES project results in the production of substantial “raw material” – may it be raw or post-processed data (3D point clouds, acoustic indicators, description of activity chains, etc. ) or may it be models, or analysis biases (architectural informational patterns, sonification strategies, matrix of inference justifiers, etc.). This material opens up several potential research lines that partners plan to co-explore, including in particular:

o Sonification and visualization: implementation of an «online interactive correlator«, a web application the objective of which is to allow an analyst to make a selection of architectural, visual, metric and sound variables and to study their relationships.
o Perceptual analysis: experiments carried out with panels of participants who report their perception of sounds related to the rural chapel corpus. The experiments are at various stages of progress (experiments already carried out and needing interpretation, or protocols still under development).
o Pattern search in 3D point clouds (seen as “massive data”): investigation and testing of solutions based on PLY files - 3D ascii format.
o Two-voice corpus annotation scenario, the objective of which is to question ontologies and the inter-corpus coherence work - exploratory articulation of the Generic Archeology Model (MGA) and the Ontology of Heritage Architecture (OAP).
o Development of a «Proof of concept« application aimed at spatializing chains of inferences and justifying matrices (access from converted 3D models)

Beyond the methodological exploration aspect mentioned hereabove, the project is expected to result in the production of data or methods that are not only traceable but re-usable.

I.Dudek, J.Y Blaise Diachrograms – a theoretical framework for the modelling and analysis of a heritage artefact’s diachronic evolution
Doc. ouvert ANR SESAMES
halshs.archives-ouvertes.fr/halshs-03070375

J.Y Blaise, I.Dudek, A.Pamart, L.Bergerot, A.Vidal, S.Fargeot, M.Aramaki, S.Ystad, R.Kronland-Martinet, Space & sound characterisation of small-scale architectural heritage: an interdisciplinary, lightweight workflow, International Conference on Metrology for Archaeology and Cultural Heritage 2020 , Trento, Italy
halshs.archives-ouvertes.fr/halshs-02981084

I.Dudek, J.Y Blaise Enabling the comparability of research workflows: a case study, CAA 2019 Computer applications and quantitative methods in archaeology, Krakow, Poland.
halshs.archives-ouvertes.fr/halshs-0292763

J.Y Blaise, I.Dudek, G.Saygi Proportions vs dimensions: shedding a different light on the analysis of 3D datasets, CAA 2019 Computer applications and quantitative methods in archaeology Krakow, Poland.
halshs.archives-ouvertes.fr/halshs-02928189

A. Giacometti, B. Markhoff, A. Soulet : Mining Significant Maximum Cardinalities in Knowledge Bases, International Semantic Web Conference (ISWC 2019), October 2019, Auckland, New Zealand.

A. Giacometti, B. Markhoff, A. Soulet : Comparison Table Generation from Knowledge Bases, Extended Semantic Web Conference (ESWC 2021), June 2021, Hersonissos, Greece.

A. Giacometti, B. Markhoff, A. Soulet : Découverte de cardinalités maximales significatives dans des bases de connaissances, Revue ouverte d'intelligence artificielle (roia.centre-mersenne.org/), à paraître en 2021.

A. Giacometti, B. Markhoff, A. Soulet : VERSUS : Générateur de tableaux comparatifs à partir de bases de connaissances. Extraction et Gestion des Connaissances EGC’21, Jan 2021, Montpellier, France.

Studying heritage artefacts is, in the digital age, impacted by our growing capacity to produce observations, and to cross-examine large datasets. But will this move towards “more” (data) necessarily go along with a move towards “better” practices, and “more” understanding of the items under scrutiny? In heritage sciences, where what is at stake is not defining general rules but analysing and cross-examining individual stories, will this move really turn out to be an opportunity for academics and cultural actors?
As an answer, the project first introduces a series of insight-gaining, reproducible methods, aimed at renewing the way individual heritage items, covering scales that range from archaeological fragments to edifices, can be observed, depicted, and analysed (sound, space and ontology aspects). The core idea is that a multidimensional, multi-criteria characterisation of items can lead to fruitful fine-grain examinations of how patterns and exceptions are met in each line of characterisation, and across lines of characterisation.
In that sense the project also pinpoints at the necessity for heritage scientists to renew the way they can read and make sense of collections as such. The project therefore proposes to investigate the potential benefits of visualisation and sonification strategies in the handling of multidimensional datasets, and to assess their respective applicability setups and ranges of performance with regards to the specificity of heritage information.

But beyond that, the initiative aims at helping actors to better formalise and memorise their ways of doing, including the potentially subjective inferences (often poorly verbalised, not to say modelled) made all along the processing of the data. This definitely is a significant challenge at a moment when technologies, short-term research, and the general move towards more and more digital practices tend to impact in a vital (but often unsaid) way our research methodologies.
The project’s strategy is to open (and hopefully clarify) the debate on a recurring issue in heritage science: the imperative of formalising and memorising the various steps that lead from “premises” (sources, analogies, when not an expert’s intuition) to an assertive 3D virtual reconstruction – an end line product that remains today most often un-associated with interpretation processes.

In that sense the project is a twofold one: real-case exemplification of methods, and illustrative of good practices in terms of reproducibility, of traceability.
At the end of the day, the project’s ambition is to act as an eye-opener on characterisation, sensemaking and reproducibility challenges in heritage sciences, in a real-case approach that although naturally non exhaustive encompasses scales and scientific disciplines .