Multimedia libraries indexing for the preservation and dissemination of the Mexican Culture – MEX-CULTURE

The research activities in this project aim to bring significant new contributions in three areas: content description, content summarization and scalable content-based search. The decomposition into tasks follows the scientific objectives and encourages collaboration between Mexican and French partners. Research on content description is separated in two different tasks, one for each type of content (video and audio), but the resulting descriptions of audio and video are employed together for content summarization and retrieval. The goal of Task 1 “Scalable Description of Visual Encoded Content” is to extract effective local and global spatio-temporal descriptors from JPEG2000 compressed flow, and also consider this for other hierarchical representations and even for formats of different resolution. Task 2, “Description of Speech/Audio Content”, deals with speech/audio signal segmentation, description and classification of sound events, and Mexican speech recognition.

We aim to obtain, at the end of this project:
1. Content description methods that are adapted to high-definition video (required for the preservation of cultural heritage content) and allow to efficiently process encoded video following the Rough Indexing paradigm.
2. Audio indexing with Mexican native language speech recognition and speaker identification. The joint use of speech recognition and speaker identification creates a complete description of contents, supporting the retrieval of all the occurrences of the phrase and/or the particular speaker.
3. Scalable summarization method for audio-visual content.
4. Scalable content-based search methods allowing to provide efficient online multi-user access to large audiovisual collections, following both the query-by-example paradigm and the relevance feedback paradigm. We aim to be able to perform interactive retrieval (from databases comparable in size with TVUNAM, i.e. of about 100,000 hours of video) at least one order of magnitude faster than current state-of-the-art solutions.

The project is expected to have a significant positive impact on the preservation and broad dissemination of national Mexican cultural heritage in a digital form. Also, the project supports researcher training and opens new exciting perspectives for French – Mexican research cooperation.

See mexculture.cnam.fr

Given the importance of cultural heritage content in promoting diversity in a globalized world, making this content quickly available to a broad audience is a critical issue. Large volumes of such content must be indexed and users must be provided with means for a fast and easy access to the multimedia information, making them able to browse (according to multiple criteria) and visualize desirable content stored in the archives. This requires automatic indexing, whose two main principles emerged in the early 1990s. First, the audiovisual content itself is used as basis for indexing. Second, content is processed in the compressed form in order to save the computational cost of decompressing the videos. The MEX-CULTURE project aims to:

1. Devise, with bilateral research efforts, new automated methods for large-scale processing and indexing of multimedia content. These methods concern video indexing directly performed on encoded visual content, audio indexing using descriptors issued from speech recognition and speaker identification, cross-media indexing (image, video, audio and speech), scalable search and retrieval. The resulting techniques will be implemented in a common development platform.

2. Support the preservation and dissemination of Mexican Culture by applying the methods devised in the project to the large databases of the FONOTECA NACIONAL (National Sound Archive of Mexico), part of CONACULTA (National Council for Culture and the Arts of Mexico) and the Video library (TVUNAM, more than 100,000 hours of video) of the UNAM (National Autonomous University of Mexico). MEX-CULTURE is the first multimedia indexing project with these characteristics.

This project will be conducted by five organizations, three from France (CEDRIC-CNAM, LABRI, INA) and two from Mexico (UNAM, IPN) and it is founded on the already existing research co-operation between University of Bordeaux 1 and UNAM in the framework agreement between both universities.

The research activities in this project aim to bring significant new contributions in three areas: content description, content summarization and scalable content-based search. The decomposition into tasks follows the scientific objectives and encourages collaboration between Mexican and French partners. Research on content description is separated in two different tasks, one for each type of content (video and audio), but the resulting descriptions of audio and video are employed together for content summarization and retrieval. The goal of Task 1 “Scalable Description of Visual Encoded Content” is to extract effective local and global spatio-temporal descriptors from JPEG2000 compressed flow, and also consider this for other hierarchical representations and even for formats of different resolution. Task 2, “Description of Speech/Audio Content”, deals with speech/audio signal segmentation, description and classification of sound events, and Mexican speech recognition.

Research concerning content summarization and scalable content-based search will be performed in Task 3 “Audiovisual Summaries and Scalable Retrieval”. The ultimate goal of summarization is content browsing and search; moreover, summarization and search employ audio and video descriptions together. We intend to support several interactive search paradigms, including summary-based browsing and search, query by example (with partial queries) and relevance feedback (with partial queries as well), for large databases. All software integration activities are grouped in Task 4, “Software Development of Multi-Modal Algorithms”, that takes into account standardization requirements and produces an experimental platform employed for the final evaluation. INA has a significant contribution to this evaluation by involving its documentation and indexing professionals, who will also give guidelines on documentation practices. This should also allow INA to evaluate the relevance, for its own use, of the methods devised in this project.