By: Badrinath Chindalur, Head of the Performance Centre of Excellence (COE), ITC Infotech
Mandar Taskar, Head of Strategic Client Engagement, ITC Infotech
In today’s highly interconnected and complex digital landscape the need to ensure business-critical systems are resilient and reliable has become more critical than ever before. Such systems can have a direct influence on end customer experience, an organisation’s brand image and customer loyalty, and regulatory implications. Traditional quality assurance approaches might often fall short in uncovering potential failures in live environments, especially under unpredictable scenarios. This is where Chaos Engineering steps in, a proactive approach to identifying and mitigating system vulnerabilities by intentionally inducing failures and observing system responses. Chaos Engineering involves controlled experimenting on a software system, often in a production or production-like environment, to gain confidence in the system’s ability to withstand turbulent and unexpected scenarios. By simulating failures, engineers can identify system weaknesses before they manifest in real-world situations.
One of the most significant tech incidents in recent memory has been the payment outage in the UK on July 12, 2024. A widespread outage blocked UK shoppers from making online and card payments through major payment providers. The disruption led to serious concerns about the reliability of cashless transactions and serves as a prime example of how Chaos Engineering could have helped mitigate the impact from such a large-scale failure. The outage affected customers of numerous retailers, fast-food chains, and supermarkets. Shoppers at large retail giants were left frustrated as were unable to purchase their groceries due to the breakdown. The issue stemmed from a technical failure within a third-party payment provider system, which cascaded into widespread service disruption.
By employing Chaos Engineering principles, there was a high probability of third-party payment provider being able to minimise or perhaps even avoid system outage. Chaos Engineering could have enabled the payment provider to proactively test their systems under scenarios that mimic real-world failures. For example, by simulating network failures through injection of latency or packet loss in a controlled environment to observe how systems rerouted traffic and maintained connectivity. This would have benefited in developing and testing failover mechanisms, ensuring that network traffic could seamlessly reroute in the event of an actual failure. Additionally, testing configuration changes before they reach production could have uncovered any relevant issues. By simulating these changes in a test environment that mirrors the production setup, the engineers could have monitored the effects on data center connectivity and network traffic. Robust rollback procedures could then have been designed and tested to ensure that any disruptions caused by such changes could be quickly and effectively reverted.
The underlying philosophy of Chaos Engineering is to encourage building systems that are resilient to failures. This means incorporating redundancy into system pathways, so that the failure of one path does not disrupt the entire service. Additionally, self-healing mechanisms can be developed such as automated systems that detect and respond to failures without the need for human intervention. These measures help ensure that systems can recover quickly from failures, reducing the likelihood of long-lasting disruptions.
To effectively implement Chaos Engineering and avoid incidents like the payments outage, organisations can start by formulating hypotheses about potential system weaknesses and failure points. They can then design chaos experiments that safely simulate these failures in controlled environments. Tools such as Chaos Monkey, Gremlin, or Litmus can automate the process of failure injection and monitoring, enabling engineers to observe system behaviour in response to simulated disruptions. By collecting and analysing data from these experiments, organisations can learn from the failures and use these insights to improve system resilience. This process should be iterative, and organisations should continuously run new experiments and refine their systems based on the results.
The payments outage in the UK highlights the importance of proactively identifying and addressing system vulnerabilities before they result in widespread disruption. Chaos Engineering provides a structured approach to uncovering hidden weaknesses in complex systems, enabling organisations to build more resilient and reliable services. By embracing Chaos Engineering, companies can avoid costly outages and ensure a seamless experience for their users, even when unexpected disruptions occur. A comprehensive performance and Chaos Engineering framework can not only ensure high-performing and scalable applications but also enhance system stability and reliability. Through proactive experimentation and continuous improvement, organisations can safeguard their operations and deliver consistent service, even in the face of adversity, ultimately delivering enriched customer experience.