Extraction and Transfer of Knowledge in Reinforcement Learning – ExTra-Learn

The methodology followed to develop transfer algorithms differ from the standard design of RL
methods and requires answering three different questions:
• Which “prior” knowledge is effective in improving the learning performance? A first important
step will be to develop different models of knowledge. Optimal policies, subpolicies, features,
sampling strategy, value function, raw samples, are examples of the elements characterizing
RL problems and solutions. The choice of which knowledge to use will be directed by the
evidence that if a human supervisor was able to provide it in the form of prior knowledge,
then the learning algorithm would be able to significantly improve its performance. While in
many cases such evidence is already available (e.g., features adapted to the target value
functions), there are still many scenarios where even if explicit prior knowledge were
available, no RL algorithm would be able to provably take advantage of it.
• How can “prior” knowledge be automatically learned? While in some cases the knowledge of
interest is immediately available after the interaction with a task (e.g., samples), there are
more sophisticated forms of knowledge (e.g., an underlying low-dimensional representation
of value functions) that require defining a specific learning process.
• How can knowledge be integrated into a transfer learning process? In order to obtain a full
transfer algorithm, we need to both collect useful knowledge from past tasks and transfer it
while solving new tasks. In some cases, this may not be trivial and may require solving a
trade-off between learning the desired knowledge as fast and efficiently as possible (e.g., by
performing a thorough exploration of the environment in many tasks) and exploiting the
current learned knowledge to improve the performance of the learning process.

In the empirical evaluation of the proposed algorithms we will study
different forms of improvements. In particular, we expect different qualitative
improvements from a transfer RL algorithm:
• Jumpstart. This improvement only considers the very initial stages of the learning process and
it is mostly related to the initialization of the algorithm. We can expect to achieve this
improvement whenever it is possible to develop a prior over the solution of all tasks, that, on
average, is more accurate then a standard non-informative initialization.
• Learning speed. In exploration-exploitation problems, the learning speed improvement
corresponds to the fact that the RL algorithm can actually reduce the amount of exploration of
the environment needed to find near-optimal policies. In approximated RL algorithms,
improving the learning speed corresponds to a reduction in the sample complexity, which
refers to the amount of samples needed to achieve a desired accuracy. In general, a learning
speed improvement does not change the asymptotic performance of the transfer RL algorithm,
which achieves the same level of performance as the no-transfer version.
• Asymptotic performance. Transfer learning and notably feature learning and hierarchical
decomposition may significantly affect the asymptotic performance of a RL algorithm. In fact, if
the approximation scheme changes (e.g., changing the basis function used in linear value
function approximation), the space of functions and policies that the RL algorithm can learn will
be affected as well. As a result, we expect that, if a transfer algorithm captures the common
structure among different tasks, it could be able to learn features that allow to increase the
asymptotic accuracy of the learning process.

If the outcome of this research is positive, we expect to contribute to the
development of learning algorithms able to interact with complex environments in a much more
intelligent and autonomous way. This has a clear strategic role in facing the future challenges of a
digital society where intelligent and autonomous systems will be more pervasive and ubiquitous.
In the long term, we envision decision-making support systems where transfer learning takes
advantage of the data available from different tasks (e.g., users) to construct high-level knowledge
that allows sophisticated reasoning and learning in complex domains. For instance, online
education systems will build on automatic schedulers that design specific agendas for each student.
While learning methods guarantee to find the most effective track of lessons for each user, transfer
algorithms will be able to dramatically reduce the exploration needed to discover the skills of the
student (e.g., preliminary exercises to assess her level and define the best learning strategy) to the
minimum thanks to effective exploration strategies refined over time and users. Furthermore,
personal fitness assistants will be able to access training data from thousands of users and
transfer methods will use them to recover the best low-dimensional feature representation, which
will be constantly adapted to make learning the best fitness plan for a new user more accurate.
Finally, complex robotic tasks will be automatically decomposed in a hierarchy of subtasks
whose solutions will be transferred and reused from task to task.

The results of the project will be primarily disseminated within the machine learning community.
The theoretical, algorithmic, and empirical results will be published in major national and
international conferences and journals.