Investigating Ocular and EEG parameters for pilot’s cognitive load estimation using flight simulators

Abstract

Present day aircraft systems are becoming incredibly complicated with the advent of new technologies. As per International civil aviation organization’s safety report, failure of mechanical or any physical hardware has reduced over the years due to highly reliable systems. The main reasons of incidents or accidents are pilot related - either pilot error due to experiencing high workload, loss of situational awareness, delay in task execution, lack of crew coordination; to name a few. Further, the proposed next generation aircraft cockpit solutions focus more towards considering human autonomy. In general, they focus on concepts such as providing larger Multi-Function Displays (MFDs), integrated and interchangeable displays, Pilot Vehicle Interface (PVI) solutions such as touch based displays, 3-dimensional audio, gesture recognition, speech recognition, haptic flight controls, seamless cockpit displays and so on. Hence it is important that pilot’s capabilities and limitations are considered while designing these display interfaces. We cannot assume that pilots will capture all the available information at all times for decision making. In other words, human’s sensory modalities such as vision, auditory senses, verbal senses and sense of touch should be utilized optimally. Designers should consider the limitation that human errors are induced when the mental tasks demand more resources than operator’s capabilities. Hence, when these kinds of advanced concepts are being designed, designers should also parallelly develop technologies for validating these designs. Such evaluations should consider pilot to be in the loop. The scope of my research is to understand human factors in an aircraft cockpit and to identify methodologies to evaluate pilot aircraft interfaces. My dissertation initiates with identifying areas wherein eye gaze tracking can be used to evaluate cockpit interfaces. In the first user study, I set up a non-intrusive SmartEye based eye tracking system at CSIR-NAL’s flight training simulator and pilot in the loop simulations were conducted for predefined test scenarios. The study concentrates on specific applications of eye gaze tracking such as evaluating usefulness of display or symbology, monitoring pilot’s scan behavior, understanding pilot-display interactions and to estimate the variations of pilot’s cognitive load. Results from this study found correlation between the statistical inferences obtained from the ocular parameters with those obtained from the flight path deviations. Based on the understanding on the various applications of eye tracking, I develop a support system that can assist the designers or evaluators in real time. Real-time display of gaze fixation, scan path of gaze movement and scatterplot of the gaze location is provided as feedback on pilot’s attention allocation. Further, I discuss about a user study using NALSim flight simulator with emphasis on understanding the effect of secondary task on pilot’s cognitive load. Algorithms are developed to evaluate distribution of gaze fixations, to quantify variations in pupil dilation and to know the electroencephalogram (EEG) band power variations. Another aim of this study was to find relation between physiological measures such as eye gaze and EEG based measurement and flying performance-based measures. I analyzed participant’s ocular parameters, power levels of different EEG frequency bands, and flight parameters for estimating variations in cognitive workload. Results indicate that introduction of secondary task increases pilot’s cognitive workload significantly. This study also proposes a methodology for estimating pilot’s cognitive workload based on his/her physiological measures such as EEG, ocular parameters and pilot’s flying performance. My research also extends to design and develop a virtual reality-based flight simulator with option for cognitive load estimation. VR based flight simulator is proposed to be a low cost and modular option for initial phase of design wherein lot of design iterations are involved. I conducted a user study with Air force test pilots to understand pilot’s interaction on the aircraft in an AI enabled battlefield scenario. In this study, I considered an AI enabled target aircraft with one-on-one air combat scenarios and evaluated effect of pilot-aircraft interactions on pilot’s cognitive load. Results indicate that the developed algorithms estimate cognitive load accurately in the VR environment. Furthermore, I applied metrics derived from these parameters to different regression-based machine learning algorithms. Efficacy of different machine learning algorithms were evaluated to find that a neural network model with two branch parallel architecture provided optimal performance. This system is proposed to be used for real time prediction of pilot’s state. Such a system can be used for detecting user’s cognitive state as part of a user alert system.