Cloudflare announced that it lost 55% of all logs forwarded to customers over a 3.5-hour due to an error in its log collection service on November 14, 2024.
See also: Cloudflare: Warns of hacking gang targeting Asia
Cloudflare offers an extensive logging service to customers, allowing them to monitor traffic to their website and filter that traffic based on certain criteria.
These Cloudflare logs allow customers to analyze traffic to their hosts to monitor and investigate security incidents, troubleshoot problems, DDoS attacks, traffic patterns, or to perform website optimizations.
For customers who wish to analyze these logs using external tools, Cloudflare offers a “logpush” service that collects logs from its various endpoints and pushes them to external storage services, such as Amazon S3, Elastic, Microsoft Azure, Splunk, Google Cloud Storage, etc.
These logs are generated on a massive scale, as Cloudflare processes over 50 trillion customer event logs every day, of which approximately 4.5 trillion logs are sent to customers.
Cloudflare says a bug in its logpush service caused customer logs to be lost for 3.5 hours on November 14. The incident was caused by a misconfiguration in Logfwdr, a key component in Cloudflare's log pipeline, which is responsible for pushing event logs from the company's network to downstream systems.
See also: Cloudflare blocked the largest DDoS attack ever recorded (3.8 Tbps)
Specifically, a configuration update introduced a bug that issued an “empty configuration,” incorrectly telling the system that there were no clients whose logs were configured for forwarding, and so the logs were discarded.
Logfwdr is designed with a failsafe that defaults to forwarding all logs in the event of “empty” or invalid configurations to prevent data loss. However, this failsafe caused a huge increase in the volume of logs being processed as it attempted to forward logs for all clients.
It overwhelmed Buftee, a distributed caching system that holds logs temporarily when downstream systems cannot process them in real time, which was called upon to handle 40 times more logs than its designed capacity.
Buftee has its own set of buffer overflow safeguards, such as resource caps and throttling, but these failed due to improper configuration and lack of prior testing.
As a result, within just five minutes of the misconfiguration in Logfwdr, Buftee crashed and required a full restart, further delaying recovery and resulting in the loss of even more Cloudflare logs.
See also: Cloudflare: New announcements on Hugging Face, Workers AI and new products
🔒 Protect your privacy with Proton VPN
Swiss VPN from the creators of Proton Mail — strict no-logs policy, strong encryption, and built-in NetShield that blocks ads, trackers, & malware.
- ✔ No-logs, based in Switzerland (except 14-Eyes)
- ✔ NetShield: blocks ads, trackers & malicious domains
- ✔ Covers all devices — free version available
The link is an affiliate link — SecNews may receive a commission at no additional cost to you. It does not affect the independence of our article writing.
Cloudflare is a company that provides internet security and website performance improvement services. Its services include DDoS protection, content delivery (CDN), load speed optimization, and security tools to protect data and users. Cloudflare helps strengthen the online presence of businesses by ensuring fast and secure access to websites and applications.
Source: bleepingcomputer
