Advanced Automation Paradoxes and Commercial Aviation

Fabrizio Interlandi and Giacomo Belloni 11 min read

Introduction

Although dated, some scientific studies remain highly relevant today when intelligent technologies rapidly emerge. Revisiting these studies is often valuable, as it ensures that future technologies are designed with an operator-centric approach, thereby mitigating many of the errors made in the past.

Lisanne Bainbridge's "Ironies of Automation" (1983) profoundly explores the paradoxes and unintended consequences of increasing reliance on automation in complex systems. She contends that automation can exacerbate problems related to human operators rather than resolve them and that “designer errors can be a major source of operating problems” (Bainbridge, 1983, p. 775). She posits that designers may perceive the human operator as inherently unreliable and inefficient, leading to the belief that the operator should be removed from the system altogether (Bainbridge, 1983).

In a future dominated by intelligent machines, although technologies can improve performance in general, they cannot do so when unexpected, non-linear, and unprecedented situations arise. Advanced technologies excel at recognising patterns and making predictions based on historical data; however, they struggle with unprecedented events (Belloni, 2024). The operator will continue to play a central role in any complex environment. Therefore, the design of advanced cockpits must begin by carefully considering the intricate dynamics of human-machine interaction.

Although Bainbridge's work covers many areas, its relevance to modern commercial aviation is strong. Over time, the industry has increasingly moved toward advanced automation, with modern aircraft now depending heavily on automated systems for navigation, flight control, and even decision-making. However, this transformation has not been without its difficulties.

Bainbridge's insights into the pitfalls of automation are echoed in several aviation incidents in which pilots were unable to intervene efficaciously when automated systems failed or behaved unexpectedly.

As Sengupta et al. (2016) argue, recent statistics indicate a growing number of accidents attributable to human error, often stemming from an over-dependence on or inadequate understanding of automated systems. This trend is substantiated by accident investigation reports from notable air disasters, such as the Turkish Airlines B-737 crash during the approach to Schiphol Airport in 2009, the Air France A-330 crash near Rio de Janeiro in 2009, and the Asiana Airlines B-777 crash in San Francisco in 2013. These incidents prompt a critical evaluation of whether automation has genuinely fulfilled its intended purpose of reducing human error and enhancing safety or whether it has introduced new challenges in an attempt to address previous issues (Sengupta et al., 2016).

The Changing Role of Pilots

In the early days of aviation, pilots were required to manually control nearly every aspect of the flight, from takeoff to landing. As automation has advanced, the role of the pilot has evolved from being a hands-on controller to that of a system manager and monitor. Bainbridge's observation that human operators become passive monitors in highly automated systems is particularly relevant here. Modern pilots often rely on automated systems to handle most of the flight, taking the controls only when necessary.

However, this shift to monitoring has introduced new challenges. For instance, in highly automated environments, pilots may become less engaged, leading to decreased situational awareness and a reduced ability to respond effectively in emergency situations. This disengagement can be catastrophic when the automation fails or when the situation requires quick, decisive manual control.

The Paradox of Automation in Aviation

Bainbridge's concept of the "paradox of automation" is vividly illustrated in several high-profile aviation incidents.

One notable example is the crash of Air France Flight 447 in 2009. The Airbus A330 was on a routine flight from Rio de Janeiro to Paris when it encountered a high-altitude stall over the Atlantic Ocean. The stall was caused by the icing of the pitot tubes, which provided incorrect airspeed data to the aircraft's systems. The autopilot disengaged, handing control back to the pilots. However, the pilots, who were unprepared for this sudden reversion to manual control, responded inappropriately by pulling the aircraft's nose up, exacerbating the stall. Despite repeated stall warnings, the crew did not recognise the situation correctly, and the aircraft eventually crashed into the ocean, killing all 228 people on board.

This tragedy highlights the paradox of automation: systems designed to enhance safety and reduce workload can lead to situations in which pilots need more preparation to take over when things go wrong.

"After the autopilot disconnection, while the aeroplane was stable in cruise, several pilot inputs significantly degraded the aeroplane's kinetic energy. The rapid exit from the flight envelope was not anticipated by the pilots, nor as it understood" (BEA, 2012, p. 209).

In the case of Air France Flight 447, the pilots were not adequately trained or experienced in handling high-altitude stalls manually, as they typically relied on the aircraft's automated systems to manage such situations. "The stall warning and the buffeting were not identified either. This was probably due to a lack of specific training." (BEA, 2012, p. 204)

When these systems failed, the pilots were unable to intervene effectively, leading to a fatal outcome. They received many warnings from the onboard systems, but none explicitly informed them that the airspeed data displayed on their instruments was corrupted. The failure of the systems to communicate this critical information left the pilots unaware of the root cause of the situation, possibly contributing to cognitive overload and misunderstanding of the situation, which ultimately led to their inability to apply the required corrective actions.

When a system fails, it is crucial that the technology hands over control to the human operators in a manner that enables them to manage the situation effectively. This handover must prioritise delivering the essential information in the correct order, in other words, the information that pilots need to Fly, Navigate, and Communicate. When automation breaks down, providing this structured and clear information is critical for avoiding overwhelming the human operator and ensuring a successful transition from automated to manual control.

Overtrust and Complacency: The Case of Asiana Airlines Flight 214

Another critical point is the tendency of operators to overtrust automated systems, leading to complacency, as evident in the crash of Asiana Airlines Flight 214 in 2013. The Boeing 777 was on approach to San Francisco International Airport when it struck a seawall short of the runway, resulting in three fatalities and numerous injuries. Impact forces and a post-crash fire destroyed the aeroplane. (NTSB, 2014)

The investigation revealed that the pilots had relied too heavily on the autothrottle system to maintain the aircraft's speed during the final approach. However, the autothrottle was not engaged as expected, and the aircraft's speed dropped dangerously low. Despite multiple warnings and the aircraft's low-speed condition, the pilots failed to correct the impending stall.

Among the recommendations issued by the National Transportation Safety Board (NTSB) was to develop advanced training that improves "flight crew understanding of automatic throttle modes and automatic activation system logic [and] the functionality of automated systems for flightpath management." (NSTB, 2014, p. 130) Moreover, "develop guidance that will help manufacturers improve the intuitiveness of existing and future interfaces between flight crews and auto flight systems." (NSTB, 2014, p. 130)

This incident underscores the dangers of overtrusting automation and the potential for complacency when pilots assume that the automated systems will manage critical flight parameters without continuous oversight.

Human-Machine Interface Design: The Crash of Turkish Airlines Flight 1951

Bainbridge also critiques the design of human-machine interfaces (HMI) in automated systems, which can significantly impact an operator's ability to manage a system effectively. Poor interface design can lead to confusion and errors, especially in critical situations. It was a significant contributing factor in the crash of Turkish Airlines Flight 1951 in 2009.

The Boeing 737 was on approach to Amsterdam Schiphol Airport when it crashed short of the runway, killing nine people. The accident was caused by a malfunction in the aircraft's radio altimeter, which provided incorrect altitude data to the autothrottle system. The autothrottle, in response to the faulty data, reduced the engines to idle, causing the aircraft to lose speed and altitude. The pilots did not notice the autothrottle's behaviour until it was too late to recover.

The design of the aircraft's HMI played a decisive role in this incident. The interface did not clearly indicate the autothrottle's status or the cause of the power reduction, making it difficult for the pilots to understand what was happening and respond appropriately.

"The investigation revealed that the available indications and warnings in the cockpit were not sufficient to ensure that the cockpit crew recognised the too big a decrease in speed at an early stage." (Dutch Safety Board, 2010, p. 85)

Bainbridge's emphasis on intuitive and straightforward interfaces directly applies here. Suppose the HMI had better communicated the state of the autothrottle and the underlying issue with the radio altimeter. In that case, the pilots might have had a better chance to intervene and prevent the crash.

Training and Expertise: The Lion Air and Ethiopian Airlines 737 MAX Crashes

Bainbridge's concerns about the impact of automation on operator training and expertise are especially relevant in the context of the Boeing 737 MAX crashes involving Lion Air Flight 610 in 2018 and Ethiopian Airlines Flight 302 in 2019. Both crashes were caused by the malfunction of the Maneuvering Characteristics Augmentation System (MCAS), an automated system designed to prevent the aircraft from stalling.

As we already mentioned in a previous article (The limits of data-driven technologies in the world of complexity), in both cases, the pilots struggled to understand and counteract the MCAS's repeated nose-down commands. The unexpected behaviour of automation and insufficient training on the system left the pilots unable to regain control of the aircraft. These tragedies, which resulted in the loss of 346 lives, highlight the critical importance of comprehensive training for pilots, especially in understanding and managing automated systems.

Bainbridge's argument that operators need a deep understanding of the systems they oversee is starkly illustrated here. In both cases, the pilots lacked sufficient training on the MCAS, a relatively new system that operated differently from the automated systems they were used to. This lack of training, coupled with the system's poor design and inadequate information in the cockpit, led to catastrophic outcomes.

Conclusion: A Call for Balanced Automation in Aviation

The aviation incidents discussed here exemplify the "ironies of automation" that Bainbridge articulated over four decades ago, shedding light on the complexities and paradoxes inherent in automated systems. These events underscore a crucial paradox: as control systems become more advanced, the human operator's role, though reduced in routine operations, becomes even more critical in moments of crisis.

This paradox arises because automation, while designed to reduce the need for human intervention, tends to fail or behave unpredictably in the most complex and unexpected situations. When this happens, the human operator—now more of a monitor than an active participant—is suddenly required to step in. However, this intervention is often needed at the "worst possible time," when the situation has escalated, and the automated systems have reached the limits of their capabilities.

At this point, the human operator is expected to quickly assess and rectify the situation despite potentially having a limited or unclear understanding of what is going on. The complexity that automation is meant to manage now becomes a barrier, as the operator must swiftly make sense of a system they have not been actively engaged with. This can lead to delays in response, incorrect decisions, and in the worst cases, catastrophic outcomes, as seen in incidents like Air France Flight 447 and the 737 MAX crashes.

The challenge, then, is to design automated systems that acknowledge this paradox. Systems must be built to manage routine tasks and ensure the human operator is adequately prepared and informed to intervene when necessary. This requires automation that supports rather than replaces human decision-making, with interfaces that clearly convey critical information and training programs that ensure pilots understand the systems they oversee.

Bainbridge's insights remain profoundly relevant as the aviation industry continues to integrate increasingly sophisticated automated systems. The more advanced these systems become, the greater the need for a balanced approach that recognises the indispensable role of the human operator. Far from diminishing the operator's importance, advanced automation makes their contribution even more crucial, especially in those rare but critical moments when everything depends on human intervention. This understanding should guide the ongoing development of aviation technology, with the ultimate goal of creating safer and more resilient systems.

As the aviation industry continues to introduce more advanced technologies in the flight deck—technologies that may further reduce the role of human pilots—this concept becomes even more critical. The push toward increased automation, including potential future developments like fully autonomous aircraft, heightens the importance of ensuring that human pilots are not just passive monitors but remain essential contributors to flight safety. The more advanced these systems become, the greater the need for a balanced approach that recognises the indispensable role of the human operator. Suppose the design of automation acknowledges that automation can fail leaving the system potentially out of control. In that case, ensuring safety during those rare but critical moments when everything depends on human intervention becomes paramount. In such cases, it is essential that the system design not only integrates the human component but also fully acknowledges and supports its critical role. It involves creating systems that actively facilitate the human operator's ability to effectively take over control when automation can no longer manage the situation. The design must ensure that the operator is always prepared, well-informed, and capable of making quick, safe decisions, even in the most complex and high-pressure scenarios. This approach recognises that effective handover from automation to human control is not just a contingency plan but a fundamental aspect of system design, crucial for maintaining safety and preventing catastrophic failures when the limits of automation are reached.

References

Bainbridge, L. (1983). Ironies of Automation. Automatica, Vol. 19, No. 6. pp. 775 779.

Belloni, G. (2024). The limits of data-driven technologies in the world of complexity. https://www.linkedin.com/pulse/limits-data-driven-technologies-world-complexity-giacomo-belloni-khr4f/?trackingId=Ry0JsNtPZ75eOL98eREU9Q%3D%3D

BEA (2012). Final Report Air France flight AF 447. 1st June 2009. Bureau d’Enquêtes et d’Analyses.

Dutch Safety Board (2010). Crashed during approach, Boeing 737-800, near Amsterdam Schiphol Airport, 25 February 2009.

NTSB (2014). Accident Report. Asiana Airlines Flight 214, Boeing 777-200ER. 6 July, 2013. National Transportation Safety Board.

Sengupta, S., Donekal, A. K., Mathur, A. R. (2016). Automation in Modern Airplanes - A Safety and Human Factors Based Study. APCOSEC 2016 – 10th Asia Oceania Systems Engineering Conference. Bangalore, India, 9 Nov -11 Nov, 2016.