Plant Recognition for Smartphone Software – REVES

These days, everyone almost has a mobile phone. These phones are used not only for phone calls but incorporate many features useful in everyday life or simply for fun (e.g. GPS, photo or video camera, accelerometer, Internet connection, etc.). Given the characteristics of these new devices (the relatively small computing power of embedded, the limited memory, the poor quality of images, the network connection that can be lost, limited battery) and according to the various possible uses, new scientific challenges exist in computer science but also in humanities.

In order to highlight these challenges, we will study in this project the following scenario: a person during a walk in the forest or in a natural park takes a picture of a plant (flower or/and leaf) with her/his cell phone in order to have information about it. She/he can possibly interact with the application by specifying the part of the plant, which is taken into consideration (flower or leaf) and/or by pointing the object on the touch screen. If the picture has not a good enough quality, the system could propose to the user to take another picture. Then, after a content-based retrieval process, the system suggests one or more responses with their confidence levels. The system may also return one or several photos of the suggested plants, in order to let the user check the validity of the answer by looking at the real plant. The user selects an answer and obtains the description of the plant. He can also ask to see any other available description. It should be noted that in order to give the best answer to the query, the system will be based of course on the analysis of photographs taken by the user, but also on various data available on Smartphones such as the geographic coordinates obtained via a GPS, the date, wheater?, etc. and some other usual features of flowers or of leaves at a given altitude, in a given region or at a given time of the year.

First, we propose to study different methods to characterize the object of interest in the image. Methods of literature should be adapted to take into account on one hand, natural objects a priori rather difficult to treat and secondly, the constraints linked to Smartphone. The various steps are then to estimate the quality of the image, to segment the object of interest and finally to extract geometric and statistics features to describe it. In this task, botanists will help computer scientists to find discriminating features. Secondly, as we propose to retrieve the data recorded in the botanical database, by taking into account the user position (spatial data) but also some other contextual data accessible via the Smartphone such as the season or the weather for example. An important work will be performed on the fusion of heterogeneous data. In this context, geographers will participate to the elaboration of the signature. Third, geographers will study the uses of such a device. We plan to question scientists, park managers, biology educators and walkers about their point of view about a Plant Recognition Smartphone service. We will try to understand the change that can be expected in the practice and the understanding.

A company that is specialized in mobile development is part of the project. It can guide researchers, whose goal is to develop prototype software, to workable solutions on mobile. The company through a consortium agreement could then exploit the fruit of this research.
The scientific results of this work will be published in international journals and conferences and will be usable in other contexts incorporating similar constraints. The work will also be popularized to the general public through a website and many applications of such a project could emerge such as citizen science, nature conservation, environmental serious game to learn to recognize species or contribution to Web 2.0.