5 Mega Software Disasters Of 2024

OpenAI faced a major outage on December 11, disrupting key services like ChatGPT, Sora, and its developer API for a few hours. 2024 has been the year of major software disasters. How can organisations like Microsoft, Google, and Oracle avoid these mistakes and stay safe in an increasingly digital world?

The outage occurred during the public launch of Sora, a highly anticipated tool, and the rollout of ChatGPT integration with Apple’s iOS 18.2. Both events brought a surge of new users, pushing OpenAI's servers to their limits. Apple Intelligence functionality also suffered disruptions due to its reliance on OpenAI's infrastructure.

Software misconfigurations are a leading cause of security breaches. According to the Cloud Security Report by Check Point, 82% of enterprises have faced security incidents caused by cloud misconfigurations, resulting from human error rather than software defects. These mistakes create serious risks, with 27% of businesses reporting security breaches in their software systems because of misconfigurations. 

OpenAI’s VP of Engineering, Srinivas Narayanan, wrote in a message to users:

“Between 3:16 pm and 7:38 pm PT today, the OpenAI API, ChatGPT, and Sora were unavailable… I’m very sorry for the trouble. In short, a configuration change was made that caused many of our servers to become unavailable.”’

This incident highlighted the often-overlooked danger of application misconfigurations —small but critical errors, such as a misplaced setting or overlooked access control, that can lead to widespread service failures, costly data breaches, and substantial financial losses.

The OpenAI outage

On Wednesday, December 11, 2024, OpenAI experienced one of its most prolonged outages, disrupting key services such as ChatGPT, its developer API, and the newly launched video generation tool, Sora. The outage, which began around 3 PM Pacific Time, lasted approximately three hours before services were fully restored. The incident affected millions of users, leaving businesses and developers reliant on OpenAI’s tools unable to operate.

What caused the outage?

OpenAI clarified that the outage was not linked to a security breach or the recent product launch. Instead, the root cause was a new telemetry service introduced to collect Kubernetes metrics. 

Kubernetes, an open-source system that manages containers for running software in isolated environments, is critical to OpenAI's infrastructure.

“Telemetry services have a very wide footprint, so this new service’s configuration unintentionally caused … resource-intensive Kubernetes API operations,” OpenAI wrote. 

This overwhelmed Kubernetes API servers, causing the control plane—the system that oversees and manages Kubernetes clusters—to fail in many of OpenAI’s larger clusters.

OpenAI’s reliance on DNS caching added to the problem. DNS (Domain Name System) converts IP addresses into readable domain names, like turning “142.250.191.78” into “Google.com”. The caching system delayed the visibility of the error, allowing the telemetry service rollout to continue before the full extent of the problem was noticed.

“This was a confluence of multiple systems and processes failing simultaneously and interacting in unexpected ways,” OpenAI admitted in its statement. “Our tests didn’t catch the impact the change was having on the Kubernetes control plane, [and] remediation was very slow because of the locked-out effect.”

Five major software disasters of 2024

The year 2024 saw major software failures that caused serious disruptions and highlighted the dangers of misconfigurations and system errors. These incidents affected industries worldwide, exposing sensitive data and causing significant financial losses. They showed how even small mistakes in software updates or system configurations could lead to large-scale problems, making it clear that strong testing and better management are essential for today’s interconnected digital systems.

Here are the five major software disasters that disrupted the tech world in 2024:

January: CrowdStrike and Microsoft

In January 2024, a flawed update in CrowdStrike’s Falcon platform led to one of the largest IT outages in history, causing severe global disruptions to Microsoft Windows systems. The outage affected key sectors such as healthcare, finance, and aviation, leaving millions of users facing the dreaded "blue screen of death" (BSOD). This event highlighted the risks posed by software misconfigurations and the interconnected nature of critical IT systems.

Root cause: The problem originated from a misconfigured logic update in CrowdStrike’s Falcon platform, specifically within a component known as channel file 291. This file contained an undetected logic error, which was released to customers during a routine update. CrowdStrike’s Falcon platform is deeply integrated into the Windows kernel, allowing it to monitor and protect the system at a very low level. However, this close integration also meant that any fault in the Falcon software had a direct and significant impact on Windows systems.

Once the update was applied, the error propagated across systems, causing them to crash and become inoperable. Validation processes within CrowdStrike failed to catch the error, and millions of devices running Windows were affected globally.

The Impact

The consequences of the outage were far-reaching:

• Aviation disruptions: Over 10,000 flights worldwide were grounded or delayed as airlines experienced system failures. Major carriers, including Delta and United Airlines, were forced to suspend operations temporarily, stranding thousands of passengers.
• Public transport: Major cities such as New York and Washington D.C. reported issues with their public transit systems, causing delays and commuter frustration.
• Healthcare systems: Hospitals and clinics faced chaos as appointment booking systems and emergency services were disrupted, potentially endangering lives.
• Financial services: Online banking platforms and payroll systems experienced outages, causing delays in financial transactions and widespread inconvenience for businesses and individuals.
• Economic costs: The total financial impact of the incident is estimated at $5.4 billion, with Fortune 500 companies bearing the brunt of the losses.

How CrowdStrike responded?

CrowdStrike quickly moved to address the crisis and implement measures to prevent future occurrences. The company re-evaluated its update processes and made several critical changes to enhance the safety of its software deployments. Updates are now treated with the same level of scrutiny as new software releases, ensuring careful testing before deployment. CrowdStrike introduced a phased rollout approach, allowing updates to be applied to smaller groups first, reducing the risk of widespread disruption if errors are found. Customers were also given more control over when updates were applied, enabling them to better prepare for any potential issues.

In the aftermath of the incident, CrowdStrike also conducted a comprehensive review of its validation processes. By improving its testing and error detection mechanisms, the company aimed to ensure that no similar logic flaws would escape notice in the future. These efforts were accompanied by transparent communication with affected clients and public acknowledgement of the error, demonstrating accountability and a commitment to learning from the incident.

February: Google Firebase misconfigurations

In February 2024, a major security lapse in Google’s Firebase databases exposed over 19.8 million plaintext credentials, affecting nearly 4,000 businesses globally. Sensitive user information such as email addresses, phone numbers, and payment details was left accessible due to widespread misconfigurations. This incident highlights the dangers of improper cloud database setups and the critical need for better security practices in modern cloud environments.

Root cause: The issue arose because many developers failed to configure Firebase databases securely. Default settings were often left unchanged, allowing unauthorised access to sensitive data. Firebase, a platform widely used for app development, relies on developers to set security rules, but many lack the necessary expertise or awareness to do so. The problem was compounded by the absence of encryption for some of the exposed data, with plaintext credentials stored directly in the databases.

Further investigation revealed that some databases had no security rules in place, making them completely open to public access. Others had weak or improperly configured rules that could be exploited by attackers. The use of plaintext passwords instead of encrypted ones made the data even more vulnerable, allowing attackers to misuse the credentials for phishing and other malicious activities.

The Impact

The consequences of the breach were severe:

• Over 125 million sensitive user records were exposed, including financial and personal data.
• The leaked information created significant risks of phishing attacks, identity theft, and credential stuffing, where stolen credentials are used to gain unauthorised access to other accounts.
• Businesses that relied on Firebase faced reputational damage, as customers lost trust in their ability to protect data.
• Regulatory scrutiny increased, with organisations facing potential fines for failing to safeguard user information.

How did Google respond?

Google took swift action to address the issue, urging developers to adopt multifactor authentication (MFA) to add an extra layer of security. The company emphasised the importance of encrypting sensitive data, even within trusted systems, and encouraged developers to follow Firebase’s security best practices. Google also provided updated tools and resources to help developers detect and fix misconfigurations, such as improved alerts and more accessible documentation.

This breach served as a stark reminder of the shared responsibility model in cloud computing, where cloud providers and developers must work together to ensure data security. It also highlighted the need for organisations to prioritise security training for their teams and regularly audit their cloud configurations.

April: The Open Web Application Security Project (OWASP)

In April 2024, the Open Web Application Security Project (OWASP), a globally respected leader in web security, suffered an unexpected data breach. The incident involved resumes submitted by members between 2006 and 2014, which were accidentally exposed online due to a misconfigured legacy wiki server. The breach raised concerns about OWASP’s security practices and served as a cautionary tale for organisations relying on outdated systems.

Root cause: The breach occurred because OWASP’s old wiki server had directory browsing enabled, allowing unauthorised users to view and download files stored on the server. This misconfiguration had gone unnoticed for over a decade, as the system had not undergone regular security audits. The affected resumes were part of OWASP’s early membership process, which required individuals to submit documents to demonstrate their professional credentials.

Unfortunately, when OWASP transitioned to newer membership systems, the old resumes were neither archived securely nor deleted, leaving them vulnerable to unauthorised access. The organisation’s reliance on this outdated infrastructure ultimately led to the exposure of sensitive data.

The Impact

The breach had several repercussions:

• Personal information, including names, addresses, phone numbers, and other professional details, was exposed, putting affected individuals at risk of identity theft.
• OWASP’s credibility as a leader in cybersecurity was damaged, as the incident called into question its ability to follow best practices.
• Many of the individuals affected could not be contacted due to outdated email addresses, leaving them unaware of the potential risks.

How OWASP responded?

Upon discovering the breach, OWASP moved quickly to secure its systems. The exposed data was removed from the server, and directory browsing was disabled to prevent further unauthorised access. Cached files were purged from web archives, ensuring they could not be retrieved online. OWASP issued a public apology and acknowledged the breach, outlining the steps it had taken to prevent similar incidents in the future.

The incident highlighted the dangers of relying on outdated systems and the importance of routine security checks. It also served as a reminder that even organisations focused on cybersecurity must continually evaluate their own practices to stay ahead of evolving threats. By addressing these vulnerabilities, OWASP aimed to rebuild trust and reinforce its reputation as a global leader in web security.

June: Exposed e-Commerce data by Oracle NetSuite

In June, researchers identified widespread misconfigurations in Oracle NetSuite’s SuiteCommerce platform. These vulnerabilities exposed sensitive customer data across thousands of e-commerce websites.

Root cause: The root cause of the issue was the improper configuration of Custom Record Types (CRTs) by website administrators. CRTs are used within SuiteCommerce to manage specific data, such as customer information and purchase records. However, many administrators failed to apply proper access controls, leaving CRTs exposed to unauthorised users.

Attackers exploited these misconfigurations by manipulating URLs and querying sensitive data through leaky APIs. Essentially, without proper authentication requirements, these APIs allowed attackers to access customer information such as names, addresses, phone numbers, and even order details. Further compounding the issue, many businesses using SuiteCommerce did not enable robust logging mechanisms. This made it difficult to detect whether sensitive data had been accessed or if breaches had already occurred, leaving companies blind to potential exploits.

The Impact:

The vulnerabilities caused widespread concern due to the sensitive nature of the data exposed:

• Customer data leaks: Personal information, including names, addresses, and phone numbers, was made accessible to unauthorised parties.
• Fraud and identity theft risks: The exposed data increased the likelihood of phishing, financial fraud, and identity theft.
• Limited detection capabilities: Due to insufficient logging, many businesses were unable to determine the full extent of the breach or whether their customer data had been compromised.
• Reputational damage: E-commerce businesses using SuiteCommerce faced loss of trust from customers who expected their personal information to be securely handled.

How Oracle responded

Oracle promptly issued detailed guidelines to help businesses secure their CRT configurations. These guidelines provided step-by-step instructions on how to restrict access to sensitive data by implementing proper authentication protocols and limiting API access. Oracle also recommended that administrators conduct regular audits of access controls to ensure they remain secure over time.

To further address the issue, Oracle enhanced its resources and tools for users, providing improved documentation and training on best practices for managing SuiteCommerce security. This included emphasising the importance of enabling robust logging mechanisms to help businesses monitor and detect unauthorised access to their systems.

November: Cloudflare’s lost logs and overloaded systems

In November 2024, a configuration error in Cloudflare’s log management system led to a cascading failure that caused the loss of 55% of customer logs over 3.5 hours. These logs, essential for compliance, auditing, and system monitoring, were irretrievably lost, leaving many organisations in a difficult position. The incident highlighted the challenges of managing distributed systems at scale and the critical importance of robust failover mechanisms.

Root Cause: The issue began when Cloudflare rolled out a misconfigured update to its log forwarding system, known as Logfwdr. This system is responsible for receiving event logs from Cloudflare’s global network and forwarding them to customer-specific destinations. However, the update mistakenly loaded a blank configuration, making it impossible for the system to determine which logs should be sent to which customer.

To prevent disruption, Logfwdr relied on a "fail open" mechanism, designed to send logs for all customers as a fallback. While this approach was meant to ensure continuity, the massive scale of Cloudflare’s operations—handling logs for millions of customers—caused the system to become overwhelmed. The log buffers, managed by a system called Buftee, were inundated with far more data than they could handle, leading to cascading failures across the entire log management infrastructure.

The overload forced the system into an unresponsive state. Even after the faulty configuration was corrected within minutes, the damage had already been done. Recovery required a complete reset of the affected systems, which took hours to complete.

The Impact

The cascading failure had significant repercussions for Cloudflare and its customers:

• Loss of critical logs: Over half of the event logs for affected customers were permanently lost. These logs were essential for compliance, security audits, and operational monitoring, leaving organisations exposed to potential regulatory and operational risks.
• Service downtime: Cloudflare’s log services were disrupted for 3.5 hours, delaying customers’ ability to monitor and respond to issues in their systems.
• Customer trust erosion: The incident damaged Cloudflare’s reputation as a reliable service provider, raising questions about the resilience of its infrastructure.

How Cloudflare responded 

In response to the incident, Cloudflare carried out a detailed investigation and made important changes to prevent similar problems in the future. They reviewed their system's failover mechanisms and fixed issues in the "fail open" process, which had worsened the situation. 

Cloudflare also introduced regular "overload tests" to check how their systems handle sudden increases in data. To avoid future errors, they improved how updates are tested, ensuring faulty configurations are caught early. Additionally, Cloudflare enhanced its monitoring and alert systems to quickly detect and fix misconfigurations before they lead to bigger issues.

Preventing future disasters – Cybersecurity tips

With software misconfigurations and security lapses becoming a common cause of data breaches and disruptions, taking proactive measures is essential. Here are some simple yet effective tips to improve cybersecurity and prevent future disasters:

• Set Up Proper Security Configurations
  Always check and configure security settings when setting up systems, databases, or cloud services. Avoid using default settings, as they can leave your systems vulnerable to attackers.
• Regularly Audit Systems
  Conduct regular security audits to identify and fix vulnerabilities. This includes reviewing access controls, checking for misconfigurations, and updating outdated systems.
• Use Multifactor Authentication (MFA)
  Enable MFA wherever possible. It adds an extra layer of security by requiring users to verify their identity with something they have (like a phone) and something they know (like a password).
• Encrypt Sensitive Data
  Ensure all sensitive data is encrypted, both when it is stored and when it is transmitted. Encryption makes it much harder for attackers to use the data if they gain access to it.
• Train Employees on Security Best Practices
  Provide regular training to employees to help them understand cybersecurity risks and how to avoid them. Simple steps like recognising phishing emails or using strong passwords can go a long way in protecting your organisation.
• Enable Robust Logging and Monitoring
  Set up logging and monitoring systems to detect suspicious activity. These systems can help you respond quickly to potential breaches and prevent further damage.
• Implement Configuration as Code (CaC)
  Treat system configurations as code, managing them through version control and automated testing. This ensures that changes are tracked and errors can be caught before deployment.
• Test Updates Before Deployment
  Always test software updates in a controlled environment before rolling them out to production systems. This helps catch issues early and avoids widespread disruptions.
• Perform Disaster Recovery Drills
  Prepare for the unexpected by conducting regular disaster recovery drills. This ensures your team knows how to respond during an incident and helps minimise downtime.
• Work with Trusted Vendors
  Choose reliable software and service providers with a strong focus on security. Verify their track record and make sure they offer support for best practices like secure configurations and regular updates.

The software misconfigurations of 2024 revealed the far-reaching consequences of seemingly minor errors. From OpenAI’s outage to CrowdStrike’s cascading failures, these incidents underscored the critical need for proactive measures in cybersecurity and configuration management. By adopting practices like Configuration as Code, rigorous testing, and routine audits, organisations can mitigate the risks of misconfigurations and build resilient systems for the future.

The lessons of 2024 are clear: the silent threat of misconfigurations must be addressed with urgency, innovation, and a commitment to continuous improvement. Only then can organisations safeguard their systems, data, and reputations in an increasingly interconnected digital world.