Machine Learning for Visual Annotation in Social-media – MLVIS

The objective of the proposal is the conception of new machine learning tools for automating information access tasks in the context of large social media including Flickr and YouTube.
Social media, as major sources of information, represent a substantial progress from the traditional vision of information access, sharing and diffusion. The traditional vision has evolved around the concept of a single user searching alone large homogeneous collections of documents. In social information media, users are diverse and may participate to different communities and share multiple social and semantic interests.
Information is multiple, heterogeneous, organized in large networks with multiple connections between content elements and users. Usage and methods for information access need to be reconsidered within the paradigm of complex information networks.
We will model this type of network as a complex graph, with heterogeneous nodes of different types, and multiple links between the nodes which represent the different types of relations among the network elements.
Machine learning has emerged over the last ten years as a major technology for analyzing and exploiting semantic data. Here again, the main concepts have developed for analyzing simple objects like i.i.d. data or sometimes sequences, and only recently more complex data structures have been considered with the emergence of important needs in fields such as biology or the web.
The development of new concepts, methods and algorithms for modelling and analyzing complex graphs of semantic elements is also a new and extremely challenging field for machine learning. This is exactly the target of the project: revisiting machine learning algorithms in the context of information access tasks for complex information networks. We will focus on two fundamental ML tasks, classification and ranking. They are generic tasks occurring at different stages in most information access problems. They can be used directly as standalone technology or as components in different search applications. This proposal investigates methodological research directions in order to upgrade machine learning algorithms including structured and contextual kernels, collective classification methods and transductive graph based learning methods. A task is also dedicated to a unifying framework for exploring the relations between these methods and their combination. A particular attention will be paid to model selection and the way we may combine different models in order to improve the overall performances. The contributions include the design of network dependent similarities (kernels, laplacians, etc.) which take into account the relationships between data, handle context and use transductive learning and inference for scene label propagation.
Besides this algorithmic and theoretical contribution, we will target a challenging representative application, the annotation of video and image collections in large social sharing media. For this, we will collect rich multimedia and social data from different sites, build an evaluation collection and adapt and evaluate our methods on this corpus.