Photographic cOntents Exploration through iMmersive Environments – POEME

To achieve these objectives, the project is divided into three parts:

- The study of image descriptors dedicated to the collection and the interests of the user, focusing on the study of combinations of image descriptors in order to obtain higher semantic level descriptors.

- The design of a content-based search engine, based on the proposed image descriptors and on a multiple paradigm framework integrating a hybrid keyword and image QbE strategy, an advanced QbE images retrieval and a new interactive research model using online learning and incremental adaptation.

- The design and creation of an immersive environment that integrates the Museum's collections and the content-based search engine. It provides, in the form of innovative and intuitive visual metaphors, new visualization tools highlighting the results provided by the search engine and enriching navigation in the collection. To provide navigation and natural queries, based on dedicated interaction metaphors, an innovative and intuitive human / machine interface has also been designed.

The software components, developed by the CNAM, have been integrated into the final prototype. They contain extraction algorithms based on global image descriptors (color, texture and shape), query modules for similarity visual search, interactive relevance feedback and an reference implementation for the low level object extraction framework.

An operational prototype, consisting of a cylindrical / conical projection structure driven by a tactile interactive interface integrating the search-by-content engine as well as the algorithms developed by the partner’s laboratories, was officially presented during the Paris Photo 2016 exhibition. This innovative interactive device was developed, designed and worked out jointly by the project's industrial partners: On Situ and Nicephore Cité.

The POEME project has made it possible to develop new and versatile software tools that make it easier to offer new digital installations using iconographic collections. The demonstrator presented at Paris Photo 2016 has shown new opportunities for interaction and visualization and thus to convince prospects of the potential of these technologies.

The work on object search and extraction by low level similarity has generated several international scientific publications.

The contributions of IGN in terms of combination of local image features for image retrieval are described in the deliverables D1.2, D1.3 and D2.3, and were published in several international scientific publications: [BGGPB14] [BGBB16] [BWGS17] [BGWB17] [GB17].

The purpose of POEME is to develop in 36 months a professional and innovative prototype for assisted analysis and manipulation of large cultural photographic collections and their exploration. Different innovative tools will be gathered into a single intuitive system, that will present itself as an immersive environment fully dedicated to the manipulation of a large collection of images. These tools will cover various and complementary aspects of the manipulation of visual contents, amongst which search-by-content tools, automatic collection analysis for assisted knowledge discovery and visualization of complex data.

One ambition of the project is to design, prototype and implement this immersive environment inside the Nicéphore Niépce Museum in Chalon-sur-Saône, with the double aim of improving management of the collection by their experts as well as contributing to its enhancement close to the general public. Our target outcomes are technologies for creating personalized and engaging digital cultural or play experiences on digitalized photographs collections, in particular on the collections of the Museum: we address adaptability of systems for personalized interaction with users, investigate technologies that add value and new meaning to digitalized photographs collections and improve user engagement with such cultural resources. To reach these objectives, the project is split into several axes:

- The study of image descriptors dedicated to the collection content and to the user interests. In particular, within the scope of a PhD thesis, we will focus on the study of combinations of image descriptors in order to obtain descriptors of higher semantic level that describe better the collection addressed, that are adaptable and that allow fine ranking of the responses in a query-by-example (QbE) retrieval scenario.

- The design of a content-based search engine, based on the proposed image descriptors and on a multiple paradigm framework studied by a postdoctoral fellow. This framework will include a hybrid keyword and image QbE strategy, advanced QbE retrieval of images and a new interactive search model using online learning and incremental adaptation tailored for the specific needs of the museum experts/visitors.

- The design and creation of an immersive environment that will integrate the collection of the Museum and the content-based search engine. By the way of innovative and intuitive visual metaphors, it will provide new visualization tools with the aims of better highlighting the responses provided by the content-based search engine and of enriching navigation in the collection. To make natural navigation and querying, an innovative and intuitive man/machine interface will also be designed, based on dedicated interaction metaphors.

POEME associates one academic partner (CNAM), two industrial partners (Nicéphore Cité - coordinator, On-situ) plus one privileged partner (Nicéphore Niépce Museum), who constitute a group of multidisciplinary teams having complementary expertise covering all the areas of the project, and having already collaborated within projects in the past.