From Model-Based Testing to Cognitive Test Automation – PHILAE

The PHILAE project aims to automate the creation and maintenance of automated functional tests by using model inference and machine-learning techniques, leveraging existing execution traces and software development meta-data. The aim is to dramatically reduce the cost of regression testing in agile software projects (typically 25% of development costs). This will be achieved through a 4-step iterative and incremental process:
1. Execution traces coming from the system in operation but also from manual and automated test execution are used to select trace candidates as new regression tests. Search-based and algorithms and coverage metrics will be used to classify and select traces;
2. From selected traces and existing workflows, active model inference is used to infer updated workflow models, which align with the current state of the implementation;
3. Reduced regression test suites are generated from the updated workflows, and these are then executed on the current system implementation;
4. Based on the test execution results, the defects detected, and development meta-data (such as commits in the code repository), a smart analytics and fault reporting system provides information on the quality of the system.

The expected results of the PHILAE project are to produce an automated testing technology for web applications developed in an agile lifecycle development at technology maturity level TRL 4. This technology will be structured in four components (one for each objective) supporting a smooth test creation and maintenance process. The PHILAE technology will be developed and assessed on the basis of three use cases from the partners, providing data from real-size projects.

The novelty of the PHILAE approach lies in the generalization and association of machine learning, model inference and automated test generation techniques to automatically invent, update and evolve functional automated regression tests.
More precisely, the PHILAE technology is using low-level test execution traces, user execution traces, test meta-data and scripts and high-level workflow elements. Firstly, these artefacts will be connected together via multi-level learning models trained by selecting relevant features of the available data. Dedicated clustering methods will enable us to raise the abstraction level of tests by grouping together execution traces and to detect anomalies by automatic classification. Thirdly, bridging the abstraction gap in this way will allow us to automatically infer prototypes of the test adaptor code that makes high-level tests executable. Combined with passive extraction of models from artefacts, active inference based on derived tests will improve high-level test models. These techniques will be used iteratively in continuous integration processes, learning to update, select, prioritize and schedule the execution of test cases.

The PHILAE project consortium is composed of 6 partners – 4 labs – UFC/FEMTO-ST, UGA/LIG, USC and SRL/CERTUS, 1 large enterprise – Orange Labs Services, and 1 Innovative SME – Smartesting Solutions & Services, with strong scientific complementary expertise. The project is decomposed into six workpackages:
WP1- Select traces as new regression test candidates
WP2- Abstract workflows from traces
WP3- Generate updated executable test suites
WP4- Smart analytics and fault reporting
WP5- Case studies and evaluation
WP6- Project Management, Dissemination and exploitation
These workpackages are organized in four phases defining four milestones at dates T0+6, T0+18, T0+28 and T0+36 (End of the project).

In terms of dissemination and innovation transfer, the strategy of the PHILAE consortium is to provide the resulting tool-set as open-source software, freely and publicly available from the PHILAE source code and version control repository.