Introduction
Optimal pilot-aircraft interaction has always been considered a cornerstone for achieving effective operational performance while maintaining a high level of safety during a task or mission. As more and more complex flight tasks are carried out, more and more information reach the crew members. There are new technological solutions on the market, and the performance during a mission is measurable. When considering human-machine interaction based on advancements in neuroscience, it is possible to measure and evaluate the effectiveness of any Human-Machine Interface (HMI). To support aircrew’s performance, the available innovations, such as data fusion or Artificial Intelligence (AI)-assisted decision-making and task management, must be leveraged for the successful conduct of military missions. AI and big data management are key factors when coupled with machine learning to improve and run modern operational scenarios. A network-centric, integrated weapon system provides flexibility to the Joint Force commander and contributes to the success of current and forthcoming joint missions.
In joint operations, when two or more nations use all available domains, it will be crucial to utilize all assets and capabilities as quickly and effectively as possible, in order to have the best overall picture of the battlespace. As such, it is important to address and validate the creation of the next generation of cockpits, optimized for the aircrew. Advanced command and control systems, which provide secure and interoperable support to conduct the mission, will ensure access to an integrated and synchronized system of systems and will enable information superiority on the battlefield. In the future, the way commanders at all levels visualize and understand the battlefield, utilizing certain aids to guide and direct their units, will be determinant for victory.
Operational Background
According to JAPCC’s Joint All-Domain Operations flyer released in 2021, all-domain operations include ‘the rapid processing of data and management of intelligence, as well as the technical ability and policies required to enable efficient operations inclusive of all contributed assets’.1 Other NATO publications use the term Multi-Domain Operations (MDO), which primarily describes the same challenges of the mission environment. Finding a coherent, common-use term is evolving, but it will not change the meaning behind the definition of HMI. Additionally, it is important to develop a connected, sophisticated interface that can assist commanders and their subordinate military personnel in sharing information simultaneously and without delay, as well as deciding and acting rapidly.
As Todd Prouty recognized in one of his articles ‘Joint All-Domain Command and Control (JADC2) is taking shape as a guiding concept to link operations’ and ‘will use AI and machine learning to connect the joint force by collecting, processing and computing huge amounts of data at machine speed’.2 As witnessed in this complicated scenario, the tactical and flight Situational Awareness (SA) of the flight crew is critical. Both types of SA are equally important as they might not only impact the successful completion of the mission, but even the strategic level’s intent. The easiest way to define SA is as a thorough knowledge of your surroundings. Tactical SA implies that the crew knows the scenario, its task and role in the mission, and all troops involved in the same area of operations. They know how to fly the mission, and also the aim and consequences of success or failure. The flight SA mainly focuses on the flight’s performance and parameters, the location in space and time, and the aircraft’s performance. The two SAs are distinct and need to be monitored constantly during flight. Usually, both require different levels of attention in different phases of the mission and, where capacity exists, can be shared among crew members. Some technical enhancements can improve just one SA, but it is desirable to increase both simultaneously to meet requirements and raise the overall SA. These developments must also support the strategic level’s intent and provide the SA it requires in the decision-making process.
Modern airframes and cockpits should support the on-board workload of the aircrew and combat pilots require that support to remain effective. This could be managed automatically through AI, enabling the aircrew to focus more effort on their tasks and missions. It could be argued that it is a basic necessity for airframes to be enhanced with algorithms to complement the crew’s ability to handle the increased information flow during flight.
During the conduct of operations it is anticipated that the situation may change rapidly and the commander must take immediate actions to re-task the forces. On the ground or in flight, the pilot may receive a new task on short notice. This new order should not be formatted as purely basic information; the support will be optimal when the whole update package can also be visualized. An example is a Digital Moving Map system depicting detailed information about friendly and adversary forces, including the coordination information. When the pilot changes the Flight Plan, the cockpit and all its settings will be updated automatically. As stated in the National Defense Magazine, ‘the ability to assemble, fuse and analyze data from limitless sources, transforming it into actionable intelligence delivered to the tactical edge, requires unprecedented processing power on the move’.3 To comply with these requirements, a push towards an integration of the next generation of HMI should be standardized in all modern cockpits.
HMI-Cockpit Evolution. © Left to Right: Ramon Berk, Comando Aviazione dell’Esercito, Leonardo
Optimizing the HMI of Civilian Aircraft
Notably, recent technological developments of aircraft cockpits have seen tremendous transformations. In only a few years, the cockpit has transitioned from the ‘classic flight deck’, with analogue quadrants, to the modern ‘glass cockpit’ in which classical instruments are presented via sophisticated multifunctional displays. Most information is interlinked between the instruments, the Flight Management System, and the autopilot functions. In modern cockpits the traditional ‘knobs and dials’ have been abandoned and replaced by electronic reconfigurable displays and multifunction reconfigurable controls, the so-called ‘soft keys’.4
Traditionally driven by safety and performance enhancements, developments in cockpit design and the way information is displayed seem much more driven by efficiency and competitiveness criteria.5 For example, in the All Condition Operations and Innovative Cockpit Infrastructure (ALICIA) project, 41 partners from 14 countries are cooperating in research and development activities intended to realize a cockpit system capable of delivering all-conditions operations. Considering the increasing number of commercial flights in the near future, the project aims to achieve higher levels of efficiency and competitiveness by using new operational concepts and cockpit designs.6
ALICIA promises new solutions capable of providing the crew with greater SA while reducing crew workload and improving the overall aircraft safety. This is a radical rethinking of the HMI concept that seeks a holistic integration of technologies. In the envisaged concept, ALICIA makes use of multimodal input/output devices to deliver an all-conditions operations application integrated within an enhanced crew interface.7
Optimizing the HMI of Military Aircraft
Improving the HMI of military aircraft is a much more complicated task. The situations to be analysed are numerous and more complex when compared to commercial flights. In the military cockpit, the tasks associated with the flight itself merge with those necessary to accomplish the combat mission, often while flying in dangerous areas and degraded environments. Additionally, the military aircraft are equipped with many more devices designed to deal with the integrated combat mission and for armament system management.
The typical tasks for military flights can be split in two categories:
- Piloting and navigation: performed during the whole flight;
- Combat tasks: only performed during some phases of the flight mission.8
When combat tasks occur they must be carried out simultaneously with piloting and navigation tasks, which is the main difference between military and commercial aviation. Based on own experience, the military pilots have to judge which one has the priority at any particular phase of flight. Therefore, they devote most of their resources to it, leaving what are often mistakenly considered less important tasks to be performed by on-board automatic systems or using the remnants of their attention.
Unfortunately, the complexity and unpredictability of military flights, in terms of tasks, risks, threats, duration, weather conditions, etc., often cause the crew to easily exceed their personal limits. When it happens, the risk is that the mission will not be accomplished or may even be abandoned. In the worst case scenario, the aircraft and crew could be lost or the crew could act without proper or optimal SA, leading to an increased risk of collateral damage.
Emerging and disruptive technologies can improve the HMI on future military aircraft. They can introduce new solutions based on AI, deep learning, or Real-Time Convolutional Neural Networks (RT/CNN) to integrate new capabilities, such as systems with cognitive solutions. As an example, the development and evolution of Cognitive Human-Machine Interfaces and Interactions (CHMI2), used to support adaptive automation in the One-To-Many (OTM) concept for multiple Unmanned Aerial Vehicles,9 could also be exploited to support adaptive automation in the accomplishment of ‘multiple tasks in the military cockpit’.
Similarly, it may be possible to investigate and develop CHMI2 to monitor the pilots’ cognitive workload and provide appropriate automation to support overloaded crews. These advanced systems should be able to read the orders arriving in the cockpit, analyse the related threats, and propose the most ‘suitable to task’ mission profiles and concepts of operation. At the same time, they should compute all mission required data, such as fuel consumption, time on target, ‘playtime’, routes, battle positions, disposition of enemy and friendly forces, selection of weapon systems and ammunitions, collateral damage estimates, and the appropriate rules of engagement, etc. The level of automation and HMI formats and functions will then be dynamically chosen considering the crew’s cognitive state.
In one of his studies from 2009, Cezary J. Szczepanski proposed a different approach to HMI optimization based on the fact that the critical factor for mission success is the workload of the aircraft operator. If the workload exceeds a specific limit, the mission cannot be successfully completed. Therefore, he proposed a way to objectively measure the crew’s workload during mission execution; specifically, the design of an HMI in such a way to ensure, even in the worst case scenario, that the workload could not exceed the limits of the human operator.10
Almost eleven years later, in 2020, the NATO Science and Technology Organization set up a research group to evaluate whether aircrew have the capability to perform their assigned tasks, with enough spare capacity to take on additional tasks, and further capacity to cope with emergencies. This group aims to identify and establish a real-time objective methodology, based on specific metrics, to evaluate HMI effectiveness.
The assessment of the cognitive state through real-time measurement of neurophysiological parameters promises to support the development of new forms of adaptive automation. This will implement an enhanced level of autonomy, similar to a virtual on-board pilot, which will assist crews in decision-making and relieve them of repetitive or mission-distracting tasks. Adaptive automation appears to be a crucial component in achieving optimal HMI. It promises to support high levels of autonomy to reduce human workload while maintaining sufficient levels of systems’ control. This may be particularly important when performing missions that require a sustained workload. This presupposes a comprehensive analysis of the ethical and moral implications associated with an autonomous decision-making machine. However, this is beyond the scope of this article.
Recommendations
The battles of the future will be increasingly fast-paced and dynamic. Emerging and disruptive technologies promise to revolutionize the way commanders at all levels will plan and conduct battlefield operations. AI, machine learning, enhanced command and control systems, and advanced big data management will significantly benefit commanders, improve SA, and dramatically accelerate the decision-making process. Modern militaries envisage future operations in a fully integrated, connected, and synchronized way, which spawned the MDO concept to refine commander’s ability to task/re-task all forces quickly and effectively across multiple domains.
This pronounced dynamism in the conceptual and planning phases must be reflected in the execution phase as well. Therefore, it must be assumed that while commanders will be able to reorganize and re-task forces with little or no prior notice, crews must also be able to process and execute those new orders quickly, effectively, and safely with little or no time to pre-plan or rehearse.
These new requirements will undoubtedly influence the design and development of the next generation of cockpits for military aircraft. There is a need to adopt a new way of conceiving the next generation of HMI that focuses more on the pilot’s true cognitive capabilities. Additionally, new solutions are needed to providing the crew with greater SA, while reducing their workload to the maximum acceptable level that they remain effective. They should incorporate task prioritization principles by judiciously considering what the aircrew can hand over to autonomous processes or systems.
This article has focused on air power and the pilot’s workload while on-board the aircraft. It is foreseen that in modern scenarios all platforms will face the same challenges. At every level of an operation, all military personnel should develop a new mindset that reflects the increased integration and usage of HMI. For this to happen, a renewed awareness of the importance of human factors is needed. Similar to civil aviation, NATO will need to develop and adopt new criteria to guide the design of tomorrow’s military aviation interfaces. HMI improvement must encompass all aviation tasks and focus on enabling real-time planning and execution. Without careful attention to the pressures military pilots are exposed to, HMI improvements will only make piloting safer without a similar increase in effectiveness during mission execution. Developing the means to assess the crew’s cognitive state through real-time measurement of neurophysiological parameters and the subsequent development of new forms of adaptive automation will be critical to achieving an HMI that meets the requirements posed by future battlefields.
