MultimEdia Entity Representation and Question Answering Tasks – MEERQAT

MultimEdia Entity Representation and Question Answering Tasks

We propose to tackle the problem of analyzing ambiguous visual and textual content by learning and combining their representations and by taking into account the existing knowledge about entities. We aim at not only disambiguating one modality by using the other appropriately but also to jointly disambiguate both by representing them in a common space. The full set of contributions proposed in the project will then be used to solve a new task that we define, namely Multimedia Question Answering

disambiguating visual and textual content by learning and combining their representations .

In the project, we consider three types of modality, namely (1) the visual modality extracted<br />from pixels of the images (2) the textual modality extracted from the questions in natural language,<br />the captions and other textual contents that are “near” an image and the textual documents used<br />to populate a knowledge database with regard to entities and, (3) by a slight misnomer that will<br />ease the understanding in the following, the structural modality that reflects the links that are<br />identified between the entities and are recorded in the knowledge database. MEERQAT thus<br />aims at answering a question composed of textual and visual modalities by relying on a knowledge<br />database that contains information relative to the visual, the textual and the structural modalities.<br />One of the main challenges consists in relating the three modalities, that do not lie in the same<br />feature space, in order to combine then compare them. An objective of the MEERQAT project is<br />thus to build a common space to these three modalities.<br />A weak fusion of modalities can quite easily be considered at a late level. This approach is<br />quite classical in information retrieval, either textual and visual; an alternative is to mainly rely on<br />one modality and filter or re-rank the results using the other ones. A more challenging objective<br />is to infer the similarity between information from different modalities at an early stage, possibly<br />at the representation level. An ultimate goal would consist to relate information independently of<br />the modality used to express it. In that sense, pushing the frontier from a late fusion of modalities<br />toward an earlier fusion contributes to such a goal, by proposing a new knowledge representation<br />frame that better matches the actual “sense” of queries and entities.<br />Beyond an efficient combination of the considered modalities to infer the meaning of a question,<br />the performance of a system that would respond to the MQA task strongly depends on the ability<br />to adequately represent the extracted data in each modality. However, since we aim at ultimately<br />mixing the modalities, it makes sense to include some information coming from the other modalities<br />at an early stage.<br />The scientific objective of the project is to learn a representation and an appropriate combi-<br />nation of the three modalities (visual, textual, structure) in a common space that will allow to<br />disambiguate the visual and textual content in order to tackle the multimedia question answering<br />task we propose. To this purpose, several innovative contributions are considered:<br />• tackling visual and textual entity ambiguities, by proposing new embeddings of each modality<br />in relation with the KB (WP2-3) but also by combining them in a common space (WP4);<br />• developing a fundamental work on the definition of an entity, in particular with regard to<br />the content to consider to represent it, depending on its type: person, place, organiza-<br />tion. . . (WP4);<br />• improving the recognition of textual entities using a KB enriched with visual data (WP3-4);<br /> studying the relation between visual and textual modalities for enabling ambiguity resolution<br />and a better understanding of a multimedia input (WP2-4);<br />• defining a new scientific task, Multimedia Question Answering (MQA), that relies on a KB<br />containing millions of multimedia entities (WP5) and releasing a related public benchmark.