Resumen de Automated methods for the evaluation and analysis of training operations in an unmanned aircraft system
Víctor Rodríguez Fernández
In recent years, Unmanned Aircraft Systems (UASs) (or Remotely Piloted Aircraft Systems (RPASs)) have become a popular topic in many different research fields and industrial applications. These systems operate with one or multiple Unmanned Aerial Vehicles (UAVs), reducing both the human and economical risks of many sensitive tasks, such as infrastructure inspection, monitoring coastal zones, traffic and disaster management, agriculture or forestry, among many others.
Although modern UASs are designed to control the UAVs autonomously, the role of UAS operators is still a critical aspect that guarantee the mission success due to the high costs involved in any operation of this kind. For this reason, operators are trained in simulation environments, where they face different situations and alerts, take adequate decisions, and get prepared to solve them successfully in a real scenario.
Unfortunately, the increasing use of UASs has not been met with appropriate integration of training science. Most of the tasks of evaluation and analysis carried out by an instructor during the debriefing of a training session are still performed rudimentarily and individually for each operator, due to the current lack of methods and tools capable of doing it automatically on a large scale. Nowadays, an expert instructor evaluates the behaviour of a single operator in each session, creating a report (usually handwritten) with different aspects such as his/her responsiveness to alerts or the evolution of his/her performance. Thus, the introduction of intelligent and automatic methods in this regard would allow to to scale up the number of operators that take part of a training session. Furthermore, the instructor would be provided not only with an individual report, but also with a collective analysis of a group of operators, which is a potential mechanism for allowing operator selection, adaptive training, and behavioural pattern analysis.
This dissertation is focused on providing intelligent and automated methods to training operations in a UAS by supporting instructors in some debriefing tasks, such as: 1. the analysis of operator performance; 2. the extraction of behavioural patterns; 2. the procedure following evaluation. To achieve these main objectives we will on base techniques that rely, partially or exclusively, on the mission data logs produced during multiple training sessions. More specifically, we will study the applicability of time series clustering, markovian modelling and process mining.
Regarding the task of performance analysis, we describe a method to discover a set of representative operator profiles directly from the mission logs, where the evolution of the operator performance during a mission is the main unit of measure. The temporal profile of the operator performance is defined based on the combination of a set of numerical measures that quantify different facets of the operator response in a specific simulation environment. Then, time series clustering techniques are used to retrieve automatically the most discriminant profiles that describe the performance evolution of a group of trainees.
The use of performance measures is not easily scalable among different simulation environments, and thus, it is interesting to use directly raw operator interactions as the basis for the task of extracting behavioural patterns. In this regard, the current methods based on Hidden Markov Models (HMMs) are used to create predictive models of the operator's behaviour. These methods have been extended in two different ways: First, the use of Multichannel HMMs is proposed in order to enrich the meaningfulness of the model states with the usage of parallel sources of information from the mission logs; On the second point, the inner modelling limitations of HMMs are considered, and based on this, the applicability of a more flexible approach based on high order Double Chain Markov Models (DCMMs) is studied.
All the proposed methods for the analysis of performance and behavioural patterns are tested rely exclusively on the simulation logs produced during the experimentation, i.e., there is no prior knowledge about the nominal operator behaviour. However, for the task of procedure following evaluation, an instructor is in charge of controlling that the operator is correctly following the guidelines described in an operating procedure, or checklist. In order to automate this task, conformance checking techniques (a family of process mining techniques) have been adapted to the use of time-based data and time-aware processes, in order to solve some limitations of the classical methods.
In order to demonstrate the effectiveness of each of the proposed approaches, several experiments have been carried out in different simulation environments. On the one hand, the approaches for automating the tasks of performance analysis and behavioural pattern extraction have been tested in a lightweight and simple multi-UAV simulation environment, with inexperienced operators. On the other hand, for the task of procedure following evaluation, a case study in a realistic UAS has been provided. Additionally, in order to prove the generality of this last approach, another case study in a external domain (longwall mining) is also provided.
The automation of the instructor tasks mentioned above may lead to the development of an all-in-one training analysis tool, which is useful not only for carriyng out a deeper and more robust debriefing of the training sessions, but also to perform operator selection, to adapt and improve the transfer of training, and to predict abnormal behaviour in real operations.
